docs/INGESTION_STRATEGY.md

39 lines · markdown

# Docket -  Ingestion Strategy (efficiency & automation)
 
Roadmap for ingest v2. Today's pollers are naive (full pull each run). This is how we make 50-state + federal ingestion fast, cheap, automated, and self-healing. Ranked by impact-to-effort.
 
## Top 10 (impact-to-effort)
1. **Watermark change-detection** -  GovInfo `collections/{col}/{lastModStart}/{lastModEnd}?offsetMark=*`; Congress.gov `fromDateTime` + **`updateDateIncludingText`** (catches text-only edits `updateDate` misses). Store `(source, last_run_at)` in `poller_watermarks`; update atomically only on successful upsert batch. ~10 lines, kills re-pulling.
2. **LegiScan weekly ZIP bulk** -  `getDatasetList`/`getDataset`: one download = all 50 states + Congress. Public API caps at 30k/mo (~one busy state). Bulk for backfill + weekly; API only for intra-week deltas.
3. **OpenStates / Plural Postgres dump** (`open.pluralpolicy.com/downloads/`) -  monthly all-50-state bootstrap, zero scraping; v3 API for incrementals.
4. **SHA-256 content-hash gate** -  hash raw payload before parse; skip parse/upsert if unchanged. `INSERT … ON CONFLICT(source,external_id) DO UPDATE … WHERE hash != EXCLUDED.hash`. Applies everywhere.
5. **RSS/Atom feed registry** -  ~20 states publish official feeds (KS, WA, HI, TX, WI, WV…). `feedparser` + last-seen GUID = sub-minute, zero-budget change detection.
6. **Session-calendar priority polling** -  `sessions` table (active/interim/dormant); poll active hourly, recess daily, dormant weekly. ~70% volume cut off-season.
7. **ARQ + aiometer async workers** -  Redis-backed asyncio queue (better fit than Celery here); `aiometer` for per-domain concurrency + rps caps; `tenacity` backoff on 429. Distribute across worker processes via hash-ring.
8. **LLM selector self-healing** -  on field-extraction failure, queue raw HTML + schema → a local LLM proposes CSS/XPath → validate against live HTML → write to `scraper_selectors` table (no redeploy). ~98% accuracy across layout changes (2025 research).
9. **Embedding action-vocab normalization** -  50 states use different action strings → embed with a local embedding model → cluster (HDBSCAN) → map to one canonical taxonomy (INTRODUCED/REFERRED/PASSED/SIGNED/VETOED…). One-time batch, re-run on new strings.
10. **Near-duplicate bill clustering** -  embed bill summaries → store vectors in the existing **Qdrant** → cosine >0.92 flags model legislation (ALEC etc.) spreading across states. Unique Docket feature; zero marginal API cost.
 
## Recommended architecture
- **Source registry YAML** -  each source: `type` (api|bulk|rss|scrape), `change_detection` (watermark|rss|hash|head), `poll_interval_active/dormant`, `rate_limit_rps`, `js_required`, `priority`. Scheduler is data-driven; new/changed sources need no code.
- **Three-tier ingest:** (1) bulk bootstrap once per session/congress; (2) delta API polling via watermarks (hourly active / daily dormant); (3) persistent RSS watchers emitting change events immediately.
- **ARQ worker pool**, partitioned by source hash-ring, Redis broker; workers write raw payloads → `raw_ingest` staging.
- **Hash-gated parse stage** -  second worker computes `content_hash`, skips unchanged, else runs source parser → `*_staging` → idempotent upsert + emit `bill_versions` diff on text change.
- **Dead-letter + alerting** -  failures → `scrape_failures` (raw error + HTML snapshot + retry count); Discord webhook after 3; 6-hour retry worker.
- **LLM repair sidecar** -  separate process (local LLM), consumes a Redis `selector_repair_queue`; main scrapers read selectors from DB each run.
 
## 50-state scraping
- **Fork `openstates/openstates-scrapers`** (~200 scrapers on `scrapelib` + `spatula` + `pupa`) -  battle-tested session mgmt/robots/retry. Don't start from scratch.
- JS-rendered states (~15: FL, IL…): `scrapy-playwright`. But **probe first** -  ~40% of "JS" state sites call an undocumented JSON/GraphQL endpoint that's cleaner + more stable than the DOM. Catalog these as a priority tier.
- **LLM-as-parser** only for the messy tail (PDFs, unstructured HTML) with `Outlines` constraint-decoded JSON + Pydantic validation. `lxml`/`trafilatura` beats LLM on structured HTML by 1000× -  never put the LLM in the hot path.
 
## Creative / wildcards
- **Wayback CDX** (`web.archive.org/cdx/search/cdx`) for historical backfill of states that purge old sessions.
- **GovTrack/`unitedstates/congress` git mirrors** -  bill versions as commits; `git log --diff-filter=M` gives changed-bills-since-last-sync without API.
- **Undocumented state JSON endpoints** -  check the network tab before writing any scraper.
 
## Local LLM: value vs trap
- **Use:** selector self-healing, action-vocab embedding/normalization, PDF/messy-HTML extraction (long tail), near-dup clustering. All batch/exception-path.
- **Trap:** LLM as primary parser for structured HTML; LLM in the real-time hot path (creates permanent backlog at 100k+ bills/session); trusting LLM output without schema validation; re-parsing the Congressional Record (use `unitedstates/congressional-record`).
 
*Adopt incrementally: watermarks + hash-gate + bulk-first first (items 1-4) -  biggest win, least effort -  then the registry + workers, then the LLM sidecar.*

1	# Docket - Ingestion Strategy (efficiency & automation)
2
3	Roadmap for ingest v2. Today's pollers are naive (full pull each run). This is how we make 50-state + federal ingestion fast, cheap, automated, and self-healing. Ranked by impact-to-effort.
4
5	## Top 10 (impact-to-effort)
6	1. Watermark change-detection - GovInfo `collections/{col}/{lastModStart}/{lastModEnd}?offsetMark=`; Congress.gov `fromDateTime` + `updateDateIncludingText`* (catches text-only edits `updateDate` misses). Store `(source, last_run_at)` in `poller_watermarks`; update atomically only on successful upsert batch. ~10 lines, kills re-pulling.
7	2. LegiScan weekly ZIP bulk - `getDatasetList`/`getDataset`: one download = all 50 states + Congress. Public API caps at 30k/mo (~one busy state). Bulk for backfill + weekly; API only for intra-week deltas.
8	3. OpenStates / Plural Postgres dump (`open.pluralpolicy.com/downloads/`) - monthly all-50-state bootstrap, zero scraping; v3 API for incrementals.
9	4. SHA-256 content-hash gate - hash raw payload before parse; skip parse/upsert if unchanged. `INSERT … ON CONFLICT(source,external_id) DO UPDATE … WHERE hash != EXCLUDED.hash`. Applies everywhere.
10	5. RSS/Atom feed registry - ~20 states publish official feeds (KS, WA, HI, TX, WI, WV…). `feedparser` + last-seen GUID = sub-minute, zero-budget change detection.
11	6. Session-calendar priority polling - `sessions` table (active/interim/dormant); poll active hourly, recess daily, dormant weekly. ~70% volume cut off-season.
12	7. ARQ + aiometer async workers - Redis-backed asyncio queue (better fit than Celery here); `aiometer` for per-domain concurrency + rps caps; `tenacity` backoff on 429. Distribute across worker processes via hash-ring.
13	8. LLM selector self-healing - on field-extraction failure, queue raw HTML + schema → a local LLM proposes CSS/XPath → validate against live HTML → write to `scraper_selectors` table (no redeploy). ~98% accuracy across layout changes (2025 research).
14	9. Embedding action-vocab normalization - 50 states use different action strings → embed with a local embedding model → cluster (HDBSCAN) → map to one canonical taxonomy (INTRODUCED/REFERRED/PASSED/SIGNED/VETOED…). One-time batch, re-run on new strings.
15	10. Near-duplicate bill clustering - embed bill summaries → store vectors in the existing Qdrant → cosine >0.92 flags model legislation (ALEC etc.) spreading across states. Unique Docket feature; zero marginal API cost.
16
17	## Recommended architecture
18	- Source registry YAML - each source: `type` (api\|bulk\|rss\|scrape), `change_detection` (watermark\|rss\|hash\|head), `poll_interval_active/dormant`, `rate_limit_rps`, `js_required`, `priority`. Scheduler is data-driven; new/changed sources need no code.
19	- Three-tier ingest: (1) bulk bootstrap once per session/congress; (2) delta API polling via watermarks (hourly active / daily dormant); (3) persistent RSS watchers emitting change events immediately.
20	- ARQ worker pool, partitioned by source hash-ring, Redis broker; workers write raw payloads → `raw_ingest` staging.
21	- Hash-gated parse stage - second worker computes `content_hash`, skips unchanged, else runs source parser → `*_staging` → idempotent upsert + emit `bill_versions` diff on text change.
22	- Dead-letter + alerting - failures → `scrape_failures` (raw error + HTML snapshot + retry count); Discord webhook after 3; 6-hour retry worker.
23	- LLM repair sidecar - separate process (local LLM), consumes a Redis `selector_repair_queue`; main scrapers read selectors from DB each run.
24
25	## 50-state scraping
26	- Fork `openstates/openstates-scrapers` (~200 scrapers on `scrapelib` + `spatula` + `pupa`) - battle-tested session mgmt/robots/retry. Don't start from scratch.
27	- JS-rendered states (~15: FL, IL…): `scrapy-playwright`. But probe first - ~40% of "JS" state sites call an undocumented JSON/GraphQL endpoint that's cleaner + more stable than the DOM. Catalog these as a priority tier.
28	- LLM-as-parser only for the messy tail (PDFs, unstructured HTML) with `Outlines` constraint-decoded JSON + Pydantic validation. `lxml`/`trafilatura` beats LLM on structured HTML by 1000× - never put the LLM in the hot path.
29
30	## Creative / wildcards
31	- Wayback CDX (`web.archive.org/cdx/search/cdx`) for historical backfill of states that purge old sessions.
32	- GovTrack/`unitedstates/congress` git mirrors - bill versions as commits; `git log --diff-filter=M` gives changed-bills-since-last-sync without API.
33	- Undocumented state JSON endpoints - check the network tab before writing any scraper.
34
35	## Local LLM: value vs trap
36	- Use: selector self-healing, action-vocab embedding/normalization, PDF/messy-HTML extraction (long tail), near-dup clustering. All batch/exception-path.
37	- Trap: LLM as primary parser for structured HTML; LLM in the real-time hot path (creates permanent backlog at 100k+ bills/session); trusting LLM output without schema validation; re-parsing the Congressional Record (use `unitedstates/congressional-record`).
38
39	Adopt incrementally: watermarks + hash-gate + bulk-first first (items 1-4) - biggest win, least effort - then the registry + workers, then the LLM sidecar.