Protocols / SEC EDGAR — Company Tickers

6 stepsduckdb >=1.2

The reproducible definition of how the source.edgar Dataset is ingested from the SEC's official company_tickers_exchange.json. Fetch (with the SEC-required User-Agent and a conditional-re-fetch freshness contract), flatten the columnar JSON to the four-field schema, materialise, export a reproducible Parquet, then describe and gate it. Clean information retrieval — the entity-resolution join lives in the separate crosswalk Protocol.

producessource.edgarSEC EDGAR — Company Tickers

Gated against a finetype schema contract

This Protocol’s output is checked against a self-derived finetype schema (schema.finetype.json) by its finetype_validate@1 step (validate). The gate runs in check-only mode: it types every column and verifies the observable envelope — type, length bounds, numeric ranges, closed enum domains and semantic labels. A pass-through on today’s run; a fail-closed drift tripwire for future ones. It writes nothing, so the built Parquet stays byte-identical.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_edgarop: http_fetch@1
Fetch the SEC's company_tickers_exchange.json. Carries the content-freshness contract (ETag / Last-Modified sidecar → an unchanged upstream is a 304, not a re-download).
producesbuild/edgar.json
with
```
url: https://www.sec.gov/files/company_tickers_exchange.json
out: build/edgar.json
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])
```
02loadsql: models/load.sql
Flatten the columnar {fields, data} JSON to the four-field schema (cik, ticker, exchange, company_name).
readsbuild/edgar.json
model
```
models/load.sql
```
03packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesedgar_out
model
```
models/package.sql
```
04export_edgarop: parquet_export@1
Write the terminal Parquet the publish seam consumes. The total order_by makes the bytes reproducible.
readsedgar_outproducesbuild/edgar.parquet
with
```
input: edgar_out
dest: build/edgar.parquet
compression: zstd
order_by: company_name, cik, ticker
```
05describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails if a curated key names a column absent from the Parquet.
readsbuild/edgar.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/edgar.parquet
overrides: descriptor.overrides.json
out: datapackage.json
```
06validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/edgar.parquetschema.finetype.json
with
```
parquet: build/edgar.parquet
schema: schema.finetype.json
```