Protocols / EDGAR ↔ GLEIF — Company-to-LEI Crosswalk

21 stepsduckdb >=1.2

The reproducible definition of how the crosswalk.edgar_gleif Dataset is made. It fetches the published EDGAR and GLEIF Datasets, builds the SEC entity universe, pulls two authoritative fund-LEI sources (GLEIF SEC registrations and four quarters of SEC Form N-CEN), name-resolves the operating-company tail with a frozen Splink model, tiers deterministic and probabilistic edges together, then exports, gates and describes one point-in-time crosswalk. Every step is a typed operator or a SQL model — no opaque steps.

producescrosswalk.edgar_gleifEDGAR ↔ GLEIF — Company-to-LEI Crosswalk

Gated against a finetype schema contract

This Protocol’s output is checked against a self-derived finetype schema (schema.finetype.json) by its finetype_validate@1 step (validate). The gate runs in check-only mode: it types every column and verifies the observable envelope — type, length bounds, numeric ranges, closed enum domains and semantic labels. A pass-through on today’s run; a fail-closed drift tripwire for future ones. It writes nothing, so the built Parquet stays byte-identical.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_edgarop: http_fetch@1
Fetch the published source.edgar Parquet (the crosswalk left side).
producesbuild/edgar.parquet
with
```
url: https://openlake.meridian.online/edgar.parquet
out: build/edgar.parquet
```
02fetch_gleifop: http_fetch@1
Fetch the published source.gleif Parquet.
producesbuild/gleif.parquet
with
```
url: https://openlake.meridian.online/gleif.parquet
out: build/gleif.parquet
```

03fetch_cik_lookupop: http_fetch@1

Fetch the raw SEC cik-lookup-data list (the SEC entity universe).

producesbuild/cik_lookup.txt

with

url: https://www.sec.gov/Archives/edgar/cik-lookup-data.txt
out: build/cik_lookup.txt
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])

04build_sec_entitiessql: models/sec_entities.sql
Parse the cik-lookup list and ticker-join it in DuckDB into the SEC entity table.
readsbuild/cik_lookup.txtbuild/edgar.parquet
model
```
models/sec_entities.sql
```
05export_sec_entitiesop: parquet_export@1
Export the typed SEC-entities Parquet that load + resolve read.
producesbuild/sec_entities.parquet
with
```
input: sec_entities
dest: build/sec_entities.parquet
```
06fetch_gleif_sec_registrationsop: gleif_ra_fetch@1
Fetch GLEIF entities registered at SEC EDGAR (RA000665) — the deterministic CIK/series↔LEI backbone for funds. Cursor-paged via dlt.
producesbuild/gleif_ra_sec.csv
with
```
ra: RA000665
out: build/gleif_ra_sec.csv
```

07fetch_ncen_2026q2op: http_fetch@1

Fetch SEC Form N-CEN 2026 Q2 (the annual fund filing — most direct fund-LEI authority).

producesbuild/ncen_zips/2026q2.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2026q2_ncen.zip
out: build/ncen_zips/2026q2.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])

08extract_ncen_2026q2op: archive_extract@1
Extract REGISTRANT.tsv (registrant CIK↔LEI) + FUND_REPORTED_INFO.tsv (series↔LEI) from the 2026 Q2 zip.
readsbuild/ncen_zips/2026q2.zipproducesbuild/ncen/2026q2/registrant.tsvbuild/ncen/2026q2/fund_reported_info.tsv
with
```
archive: build/ncen_zips/2026q2.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2026q2
```

09fetch_ncen_2026q1op: http_fetch@1

Fetch SEC Form N-CEN 2026 Q1 (quarters are pinned for reproducibility).

producesbuild/ncen_zips/2026q1.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2026q1_ncen.zip
out: build/ncen_zips/2026q1.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])

10extract_ncen_2026q1op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2026 Q1 zip.
readsbuild/ncen_zips/2026q1.zipproducesbuild/ncen/2026q1/registrant.tsvbuild/ncen/2026q1/fund_reported_info.tsv
with
```
archive: build/ncen_zips/2026q1.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2026q1
```

11fetch_ncen_2025q4op: http_fetch@1

Fetch SEC Form N-CEN 2025 Q4.

producesbuild/ncen_zips/2025q4.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2025q4_ncen.zip
out: build/ncen_zips/2025q4.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])

12extract_ncen_2025q4op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2025 Q4 zip.
readsbuild/ncen_zips/2025q4.zipproducesbuild/ncen/2025q4/registrant.tsvbuild/ncen/2025q4/fund_reported_info.tsv
with
```
archive: build/ncen_zips/2025q4.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2025q4
```

13fetch_ncen_2025q3op: http_fetch@1

Fetch SEC Form N-CEN 2025 Q3.

producesbuild/ncen_zips/2025q3.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2025q3_ncen.zip
out: build/ncen_zips/2025q3.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; [email protected])

14extract_ncen_2025q3op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2025 Q3 zip.
readsbuild/ncen_zips/2025q3.zipproducesbuild/ncen/2025q3/registrant.tsvbuild/ncen/2025q3/fund_reported_info.tsv
with
```
archive: build/ncen_zips/2025q3.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2025q3
```
15loadsql: models/load.sql
Load + normalise the fetched sources (SEC entities, GLEIF, RA registrations, four N-CEN quarters) into typed tables.
readsbuild/sec_entities.parquetbuild/gleif.parquetbuild/gleif_ra_sec.csvbuild/ncen/*/registrant.tsvbuild/ncen/*/fund_reported_info.tsv
model
```
models/load.sql
```
16resolveop: splink_resolve@1
Probabilistic Fellegi-Sunter / Splink name match for the operating-company tail (no official CIK↔LEI). Frozen model, SEED=42, precision-first.
readsbuild/sec_entities.parquetbuild/gleif.parquetproducesbuild/resolved.parquet
with
```
edgar: build/sec_entities.parquet
gleif: build/gleif.parquet
out: build/resolved.parquet
```
retry capped at 1 attempt; timeout 14400s (~4h full-corpus job).
17tiersql: models/tier.sql
Union the deterministic (authoritative) edges with the Splink (confirmed / candidate) edges into one tiered edge table.
readsbuild/resolved.parquetproducescrosswalk_edges
model
```
models/tier.sql
```
18packagesql: models/package.sql
Materialise the terminal edge table (adds the as_of stamp + search corpus).
readscrosswalk_edgesproducesedgar_gleif_out
model
```
models/package.sql
```
19export_edgar_gleifop: parquet_export@1
Write the terminal Parquet the publish seam consumes. The total order_by makes the bytes reproducible.
readsedgar_gleif_outproducesbuild/edgar_gleif.parquet
with
```
input: edgar_gleif_out
dest: build/edgar_gleif.parquet
compression: zstd
row_group_size: 50000
order_by: company_name, key_type, key, lei
```
20validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/edgar_gleif.parquetschema.finetype.json
with
```
parquet: build/edgar_gleif.parquet
schema: schema.finetype.json
```
21describeop: datapackage_describe@1
Emit datapackage.json from the built Parquet — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, per-field prose, primaryKey and foreignKeys. Hard-fails on curated-vs-Parquet drift.
readsbuild/edgar_gleif.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/edgar_gleif.parquet
overrides: descriptor.overrides.json
out: datapackage.json
```