| 1 | # Docket - Data Source Contracts |
| 2 | |
| 3 | All primary sources are free + official. State (OpenStates) is the one paid tier. No source offers webhooks → **everything is polling + change-detection.** |
| 4 | |
| 5 | ## Primary (the backbone) |
| 6 | |
| 7 | ### Congress.gov API - federal bills, votes, members · PRIMARY |
| 8 | - Base: `https://api.congress.gov/v3/` |
| 9 | - Auth: free api.data.gov key, `?api_key=` · **5,000 req/hr** · 250 records/call max |
| 10 | - Freshness: same-day for actions/status |
| 11 | - Change detection: `GET /bill?sort=updateDate&fromDateTime={ISO}` every 15-30 min; store last `updateDate` per bill, diff. |
| 12 | - Covers: bills (101st+), amendments, summaries, members, committees, nominations; **House roll-call vote endpoints (beta, 118th+)**. |
| 13 | - ⚠️ **Reliability risk:** multi-day silent outage Aug 2025. **Fallback:** `unitedstates/congress` scraper (GovInfo bulk XML, LoC-independent). Ingest must tolerate 24-72h Congress.gov downtime. |
| 14 | |
| 15 | ### GovInfo API - full bill text, Congressional Record, public laws · PRIMARY |
| 16 | - Base: `https://api.govinfo.gov/` · same api.data.gov key · **36,000 req/hr** (most generous) |
| 17 | - Delta poll: `GET /published/{start}/{end}` hourly. Collections: `BILLS`, `CREC`, `PLAW`, appropriations, `CHRG`, `FR`, `CFR`. |
| 18 | - Use for: full text, public-law text, appropriations docs. 1-2 business-day lag. |
| 19 | |
| 20 | ### USAspending.gov API - federal spending (awards, outlays) · PRIMARY |
| 21 | - Base: `https://api.usaspending.gov/api/v2/` · **no auth** |
| 22 | - Awards: `POST /search/spending_by_award/` (10k-result cap/call → use `/bulk_download/` for completeness). Agency: `/agency/`, accounts: `/federal_accounts/`. |
| 23 | - Change detection: filter by `action_date`, poll daily (agencies have 2-business-day reporting SLA). |
| 24 | |
| 25 | ### Treasury Fiscal Data API - debt, deficit, cash flows · PRIMARY |
| 26 | - Base: `https://api.fiscaldata.treasury.gov/services/api/fiscal_service/` · **no auth** |
| 27 | - Datasets: Debt to the Penny (daily), Daily Treasury Statement (T+1), Monthly Treasury Statement (outlays by agency, ~3wk lag), Historical Debt. |
| 28 | - Poll daily ~6pm ET. |
| 29 | |
| 30 | ## Vote-speed supplements |
| 31 | - **House Clerk XML** `https://clerk.house.gov/evs/` - votes within hours, no key. Poll 10-30 min during session. |
| 32 | - **Senate roll-call XML** `https://www.senate.gov/legislative/LIS/roll_call_lists/` - no key. |
| 33 | - **Voteview** `https://voteview.com/data` - bulk CSV + ideology scores. **Internal QA only** (never user-facing labels). |
| 34 | |
| 35 | ## State · IN MVP (paid) |
| 36 | ### OpenStates API v3 - 50 states + DC/PR |
| 37 | - Base: `https://v3.openstates.org/` · **API key required** |
| 38 | - Tiers: Free 500/day·10rps (too tight) → **Bronze 5k/day / Silver 50k/day (paid, contact for pricing)**. Production needs paid. |
| 39 | - JSON, standardized schema. Freshness varies by state. |
| 40 | - **Ship an honest per-state coverage map** - surface which states lag so the UI never implies completeness it doesn't have. |
| 41 | |
| 42 | ## CBO (cost estimates) · USEFUL, no API |
| 43 | - Scrape `cbo.gov/cost-estimates` by bill for per-bill scores; macro CSVs at `github.com/us-cbo`. Dollar figures only, no characterization. |
| 44 | |
| 45 | --- |
| 46 | |
| 47 | # FULL-SPECTRUM SOURCES (added 2026-06-26 - the real Docket scope) |
| 48 | |
| 49 | Spending was only one lens. These power the bills/laws/amendments/riders/closed-door pillars (PLAN.md §1b). |
| 50 | |
| 51 | ## Amendments & "what got slipped in" · SIGNATURE FEATURE |
| 52 | |
| 53 | ### Amendments data - Congress.gov API |
| 54 | - `GET /v3/amendment/{congress}/{type}/{number}` (+ `/actions`, `/text`, `/amendments`, `/cosponsors`). Types: `HAMDT`, `SAMDT`. Each carries `amendedBill`. Parent bill: `/bill/{c}/{type}/{num}/amendments`. |
| 55 | - ⚠️ Coverage: searchable Senate amendment **text only 117th Congress (2021)+**; House + older link to Congressional Record, not machine-readable. Manager's/en-bloc packages are **not** discrete objects anywhere - only inferable from sequence + CREC narrative. |
| 56 | |
| 57 | ### Bill-version text-diff (the engine of "what changed") |
| 58 | - **GovInfo BILLS** XML, 113th Congress (2013)+. Version chain: `IH`→`RH`→`EH`→`EAS`→`ENR` (House; `IS/RS/ES` Senate). PackageId `BILLS-{congress}{chamber}{number}{version}` (e.g. `BILLS-117hr3076ih`). |
| 59 | - List versions: `GET /related/BILLS-{id}` · fetch text: `GET /packages/{packageId}/xml`. |
| 60 | - **Diff recipe:** fetch consecutive versions → parse `<section>`/`<subsection>` w/ identifiers → align by id (+ fuzzy heading match, the hard part) → `difflib` within aligned sections → classify **added / removed / amended**. New section w/ no prior counterpart = inserted provision. |
| 61 | - **Rider detection** (R Street keyword signals on appropriations bills, regex over added sections): `Notwithstanding | Is Amended | Are Authorized | Provided, | Provided Further, | In General | Unless | Until | Determines`. Flags candidates for surfacing, not a verdict. |
| 62 | - **Conference reports** (GovInfo `CRPT`, 104th+): the biggest slipped-in vector. No structured diff trail - must diff the conference report text against the last engrossed version ourselves. Explanatory statements (modern omnibus) live in CREC, carry legal force. |
| 63 | |
| 64 | ### Honest gaps (state these in-product, never hide) |
| 65 | - Voice-vote / unanimous-consent additions: **no individual record exists** (~94% of Senate measures; most manager's amendments). Surface "passed by voice vote - no individual record." That absence is itself information. |
| 66 | - Conference negotiation records, pre-117th Senate amendment text, "who wrote this rider": not public. |
| 67 | |
| 68 | ## Closed-door / procedural · GovInfo CREC |
| 69 | - `CREC` collection (api.govinfo.gov, free key), sections HOUSE/SENATE/DIGEST/EXTENSIONS. Daily, ~1-day lag, bulk XML 1994+. The single best source to extract voice votes, UC agreements, cloture, suspension, motions-to-recommit, discharge-petition signatories - all via **text parsing** (no structured endpoint). |
| 70 | - Caveat: "Extensions of Remarks" = statements never spoken on the floor; speeches can be revised after delivery. Not a verbatim transcript - don't inflate "floor activity." |
| 71 | - Committee markups/hearings: Congress.gov schedule (House 2017+, Senate 2019+) via docs.house.gov; transcripts discretionary + months-late (GovInfo `CHRG`). |
| 72 | |
| 73 | ## Lobbying · LDA.gov · EASY WIN |
| 74 | - ⚠️ `lda.senate.gov` retires **2026-06-30** → use **`lda.gov/api`** (REST, free key, JSON). LD-1/LD-2 (covered bills + spend) / LD-203 (contributions). Quarterly. Answers "who lobbied this bill." |
| 75 | |
| 76 | ## Executive branch (where policy is made between laws) |
| 77 | - **Federal Register API** `federalregister.gov/api/v1` · **no key**, JSON/CSV, same-day. Executive Orders, proposed/final rules (NPRM), notices. EASY WIN. |
| 78 | - **regulations.gov API** (GSA key) - public comment dockets on rules (citizen-engagement layer). |
| 79 | - **OpenOMB** `openomb.org` (JSON, daily) - OMB **apportionments** (executive withholding/release of appropriated funds), FY2022+, mirrors statutory `apportionment-public.max.gov`. Live 2025-26 controversy. → **link source to the official `.max.gov` record, not openomb.org** (see third-party rule below). |
| 80 | |
| 81 | ## State granular · OpenStates + LegiScan |
| 82 | - **LegiScan API** - 50 states + Congress, free tier 30k q/mo, better bill-text + amendment completeness than OpenStates. Use as completeness supplement. |
| 83 | - **OpenStates v3** - richer standardized vote/people schema, fully open. Both lag 24-48h; worst states historically MA/NE/KY/AL; ballot measures + interim committee work poorly covered by both. |
| 84 | - **Per-state data-confidence score** (required): flag stale scrapes + known-missing data types (votes, committee actions, amendment text). Honesty about gaps is a differentiator. |
| 85 | |
| 86 | ## Member trades · STOCK Act · HARD |
| 87 | - No official API. Official PDFs at `efdsearch.senate.gov` + `disclosures-clerk.house.gov`. Third-party normalized JSON via Lambda Finance (free) - but **display the official disclosure as source**. Amounts are ranges; the trade→disclosure gap is the metric. |
| 88 | |
| 89 | --- |
| 90 | |
| 91 | ## Source allowlist (neutrality rule #1) |
| 92 | `*.gov` (covers congress/govinfo/usaspending/fiscaldata.treasury/cbo/clerk.house/senate/federalregister/lda/regulations/*.max.gov/efdsearch.senate/disclosures-clerk.house/docs.house), **plus a curated `STATE_LEG_HOSTS` registry** for state legislatures that don't use `.gov` (e.g. `legis.state.pa.us`, `leginfo.legislature.ca.gov`, `legislature.mi.gov`, `*.state.*.us`). |
| 93 | |
| 94 | **Third-party-normalizer rule (neutrality):** when data comes from an aggregator (OpenStates, LegiScan, OpenOMB, Lambda Finance), the user-facing `source_url` MUST point to the **underlying official government record**, never the aggregator. Aggregators are pipes, not sources. |
| 95 | |
| 96 | ## Ingest principles |
| 97 | - One poller per source, each writes normalized `actions` rows + raw payload to `raw_events`. |
| 98 | - Idempotent: dedupe on `(source, externalId, updateDate)`. |
| 99 | - A poll never blocks the feed; change-detection table is the single source of "what's new." |
| 100 | - Rate-budget aware; exponential backoff; Congress.gov fallback wired from day one. |