1773b15 · Docket

Strip internal notes and infra references for public release

1773b15 Zion Boggan committed on Jun 27, 2026 (1 month ago)

BILL_SUMMARY_ARCHIVE_NOTES.md +0 -150

		@@ -1,150 +0,0 @@
	-	## Bill Summaries & PDF Archive - Implementation Notes
	-
	-	### What was built
	-
	-	Two new ingest modules added to `services/ingest/ingest/`:
	-
	-	\| Module \| Purpose \|
	-	\|---\|---\|
	-	\| `bill_summaries.py` \| Backfills official CRS summaries onto `bills` table \|
	-	\| `archive_bills.py` \| Downloads GovInfo bill PDFs and uploads to R2 \|
	-
	-	Both are registered in `run.py` pipeline (after `categorize` / `calendar`).
	-
	-	---
	-
	-	### Schema changes (additive, ALTER TABLE safe)
	-
	-	Four new columns on `bills`:
	-
	-	```sql
	-	ALTER TABLE bills ADD COLUMN summary TEXT;
	-	ALTER TABLE bills ADD COLUMN summary_source TEXT;
	-	ALTER TABLE bills ADD COLUMN pdf_url TEXT;
	-	ALTER TABLE bills ADD COLUMN archive_pdf_url TEXT;
	-	```
	-
	-	Each module calls `_ensure_columns()` on startup (catches OperationalError if already exists).
	-
	-	---
	-
	-	### bill_summaries.py - how it works
	-
	-	Source: GovInfo BILLSTATUS bulk XML (keyless, no API key needed).
	-
	-	URL pattern:
	-	```
	-	https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/BILLSTATUS-{congress}{type}{number}.xml
	-	```
	-
	-	The BILLSTATUS XML already contains `<summaries><summary><text>` (CRS text). The module:
	-	1. Queries DB for `govinfo_billstatus` bills missing `summary` (current congress)
	-	2. Re-fetches their BILLSTATUS XML
	-	3. Picks the item with the latest `<updateDate>` from `<summaries>`
	-	4. HTML-unescapes + strips tags from `<text>` (text is HTML-encoded in the source)
	-	5. Stores verbatim (public-domain CRS text) with `summary_source = "CRS via GovInfo"`
	-
	-	Counts (test run, 119th congress, hr+s types, limit 30):
	-	- 30 summaries stored in ~20s
	-	- 17,149 congress-119 bills identified as needing summaries at test time
	-
	-	Sample output (H.R. 160 - Restoring Faith in Elections Act):
	-	> "Restoring Faith in Elections Act - This bill establishes certain standards for voting, including voting by mail, in federal elections. It also establishes certain requirements for voter registration and maintenance of official lists of eligible voters..."
	-
	-	Default run settings: `--congress 119 --limit 500 --delay 0.1` (500 summaries per pipeline run at ~0.1s/request = ~60s per run).
	-
	-	---
	-
	-	### archive_bills.py - how it works
	-
	-	Source: GovInfo BILLS collection PDFs (keyless).
	-
	-	URL pattern:
	-	```
	-	https://www.govinfo.gov/content/pkg/{pkg_id}/pdf/{pkg_id}.pdf
	-	```
	-
	-	The `pkg_id` comes from `bill_versions.govinfo_package_id` (e.g. `BILLS-119hr160ih`).
	-
	-	Version priority (most authoritative wins per bill):
	-	`enr > eh > es > ph > ps > rh > rs > ih > is > (other)`
	-
	-	R2 key: `bills/{congress}/{pkg_id}.pdf`
	-	Public URL: `https://archive.usdocket.org/bills/{congress}/{pkg_id}.pdf`
	-
	-	The module:
	-	1. Queries DB for bills with `bill_versions` but no `archive_pdf_url`
	-	2. Selects best version per bill (priority order above)
	-	3. Checks `r2.exists()` - skips if already in bucket
	-	4. Downloads PDF (verified `Content-Type: application/pdf` + minimum size check)
	-	5. Uploads via `r2.put_object()` with `Content-Type: application/pdf`
	-	6. Stores `pdf_url` (official GovInfo) + `archive_pdf_url` (our R2 copy)
	-
	-	Counts (test run, 119th congress, hr type, limit 5×2):
	-	- 10 PDFs archived across 2 runs
	-	- 0 failures, 0 re-uploads (DB-level idempotency via `archive_pdf_url IS NULL` filter)
	-
	-	Verified working archive URL:
	-	```
	-	https://archive.usdocket.org/bills/119/BILLS-119hr160ih.pdf
	-	```
	-	→ HTTP 200, `Content-Type: application/pdf`, 325,931 bytes
	-
	-	Default run settings: `--congress 119 --limit 200 --delay 0.5` (polite; ~100s per run).
	-
	-	---
	-
	-	### Idempotency
	-
	-	- Summaries: `WHERE summary IS NULL OR summary = ''` in SQL - re-runs skip filled rows.
	-	- Archive: `WHERE archive_pdf_url IS NULL OR archive_pdf_url = ''` in SQL + `r2.exists()` guard - safe to re-run.
	-
	-	---
	-
	-	### Pipeline wiring (run.py)
	-
	-	```python
	-	("bill_summaries", "ingest.bill_summaries", []),
	-	("archive_bills", "ingest.archive_bills", []),
	-	```
	-
	-	Both run after `categorize` / `calendar` steps. Both accept `--once` CLI flag.
	-
	-	---
	-
	-	### CSP hosts to add (frontend, separate pass)
	-
	-	The frontend will need to show summaries and embed/link PDFs. Two hosts to add to `next.config.js`:
	-
	-	For embedded PDFs (`<iframe>` or `<embed>` pointing to archive.usdocket.org):
	-
	-	In the CSP `frame-src` directive (currently absent - add it):
	-	```
	-	frame-src 'none' https://archive.usdocket.org
	-	```
	-	Or if PDFs are shown via `<object>`:
	-	```
	-	object-src 'none' → object-src https://archive.usdocket.org
	-	```
	-
	-	For direct PDF links (no embed - just `<a href>`), no CSP change needed.
	-
	-	Current `next.config.js` has `"object-src 'none'"` and no `frame-src`. If the frontend embeds PDFs via `<iframe>` or `<object>`, those directives need `archive.usdocket.org` added. If PDFs are linked (new tab), nothing to change. Do not edit `next.config.js` in this pass - flag it for the frontend developer.
	-
	-	---
	-
	-	### Manual run commands
	-
	-	Summaries:
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.bill_summaries --once
	-	```
	-
	-	Archive PDFs:
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.archive_bills --once
	-	```

PERF_NOTES.md +0 -134

		@@ -1,134 +0,0 @@
	-	# Docket - Performance Audit & Fix Notes
	-
	-	Audit and fix session conducted 2026-06-26/27. All timings are server-side
	-	response time measured with `curl -o /dev/null -w "%{time_total}"` against
	-	the server's local Next.js listener (`http://localhost:3000`). "First hit" is a
	-	cold request; "second hit" is a repeat within the same 5-minute TTL window.
	-
	-	---
	-
	-	## Before / After
	-
	-	\| Route \| Before \| After (1st) \| After (2nd) \| Factor \|
	-	\|--------------------\|----------\|-------------\|-------------\|--------\|
	-	\| `/register` \| 2.92 s \| 0.18 s \| 0.12 s \| 16x \|
	-	\| `/votes` \| 0.57 s \| 0.11 s \| 0.06 s \| 9x \|
	-	\| `/members` \| 2.02 s \| 0.12 s \| 0.10 s \| 20x \|
	-	\| `/bills` \| 2.69 s \| 0.28 s \| 0.20 s \| 13x \|
	-	\| `/spending` \| 0.61 s \| 0.29 s \| 0.25 s \| 2x \|
	-	\| `/members/A000055` \| 32.31 s \| 0.11 s \| 0.09 s \| 294x \|
	-	\| `/states` \| 0.097 s \| 0.025 s \| 0.026 s \| 4x \|
	-	\| `/calendar` \| 1.07 s \| 0.06 s \| 0.05 s \| 21x \|
	-
	-	QA: CLEAN - 17 routes, no overflow, no hydration errors, confirmed on
	-	every rebuild.
	-
	-	---
	-
	-	## Root Causes Fixed
	-
	-	### 1. Full-table scans passed to JS for filtering
	-
	-	`getBills()`, `getVotes()`, `getFeedItems()` all loaded entire tables into
	-	memory (208k bills, 35k votes, all feed rows) and sliced/filtered in JS.
	-
	-	Fix: pushed `WHERE`, `LIMIT`, and `OFFSET` into SQL for all three. New
	-	helpers: `buildBillsWhere(filter)`, `getBillsCount(filter)`, `getVoteCount(chamber?)`,
	-	`getFeedKindCounts()`.
	-
	-	### 2. Missing indexes
	-
	-	The `bills` table had no index on `sponsor_id` - `getMember()` did a full
	-	208k-row scan to find each member's sponsored bills. Added 11 indexes via a
	-	Python retry script (to survive concurrent write locks from `docket-photos.service`):
	-
	-	```
	-	idx_bills_sponsor_id ON bills(sponsor_id)
	-	idx_bills_policy_area ON bills(policy_area)
	-	idx_bills_status ON bills(status)
	-	idx_bills_congress_chamber ON bills(congress, chamber)
	-	idx_bills_updated_at ON bills(updated_at)
	-	idx_vote_positions_member_id ON vote_positions(member_id) (was missing)
	-	idx_vote_positions_vote_id ON vote_positions(vote_id) (was missing)
	-	idx_votes_year ON votes(year)
	-	idx_votes_year_chamber ON votes(year, chamber)
	-	idx_awards_action_date ON awards(action_date)
	-	idx_calendar_events_date ON calendar_events(date)
	-	```
	-
	-	### 3. Double getMember() call per request
	-
	-	`generateMetadata` AND `MemberDetailPage` both call `getMember(params.id)`,
	-	which runs 6+ SQLite queries. Wrapped with the in-process TTL cache so the
	-	second call returns instantly from `_cache`.
	-
	-	### 4. 1500 MemberVote objects serialised into RSC wire format
	-
	-	For long-serving members (e.g. Aderholt: 18,458 positions), the year-by-year
	-	section loaded 1500 vote rows and serialised them all into the initial HTML.
	-	This kept the page at ~1.8 s even after the index and cache fixes.
	-
	-	Fix: replaced the 1500-row join in `_getMemberImpl` with a single cheap
	-	`GROUP BY year` query. Each year's votes are now fetched lazily by the new
	-	`MemberVoteYear` client component when the `<details>` panel is first opened.
	-	New API route: `GET /api/members/[id]/votes?year=YYYY`.
	-
	-	Result: 1.81 s → 0.11 s for A000055.
	-
	-	### 5. Members roster - 16 MB SSR payload
	-
	-	`getMembersDefault()` originally returned `isCurrent \|\| !!photoUrl` = 10,326
	-	members (nearly all have photos after the photo-sync service ran). Changed to
	-	`isCurrent` only → 537 current-Congress members. Historical members load
	-	lazily via paginated `/api/members` calls in `MembersBrowser` when the toggle
	-	is clicked.
	-
	-	### 6. Calendar page - 6.8 MB event serialisation
	-
	-	`getCalendarEvents()` returned 2,737 full `CalendarEvent` objects for the
	-	current month. The grid only needed one dot-per-kind-per-day. Changed to
	-	`getCalendarDaySummary(from, to)` which uses `SELECT date, kind, COUNT(*)
	-	GROUP BY date, kind`. Full events for the selected day are fetched separately.
	-
	-	### 7. Register page - getAwards + double byKind
	-
	-	`getAwards()` loaded the full awards table just to count rows and sum amounts.
	-	Replaced with `getAwardsSummary()` (single `SELECT COUNT(*), SUM(amount)`).
	-	`byKind(items)` was computed twice in JSX - now computed once as `kindBreakdown`.
	-
	-	### 8. force-dynamic → revalidate = 300
	-
	-	All list and detail pages were `force-dynamic` (re-render every request).
	-	Changed to `revalidate = 300` (ISR, 5-minute stale-while-revalidate):
	-	`/register`, `/votes`, `/members`, `/bills`, `/spending`, `/states`,
	-	`/calendar`, `/executive`, `/categories`, `/search`, `/schedule`,
	-	`/states/[code]`, `/votes/[id]`, `/bill/[id]`, `/members/[id]`.
	-
	-	### 9. In-process TTL cache
	-
	-	Added a `Map<string, {value, expiry}>` cache in `lib/feed.ts` with a 5-minute
	-	TTL. Expensive aggregates that ISR might call multiple times in one render
	-	cycle (member lookup, category lists, member counts, all-members roster,
	-	awards summary) are now served from memory on repeat calls within the window.
	-
	-	---
	-
	-	## Files Changed
	-
	-	- `lib/feed.ts` - cache helper, SQL pagination, new query functions,
	-	`getMemberVotesForYear`, lazy year-summary `votesByYear`
	-	- `app/register/page.tsx` - revalidate, `getAwardsSummary`, single `byKind`
	-	- `app/votes/page.tsx` - revalidate, paginated `getVotes(100,0)`, `getVoteCount`
	-	- `app/bills/page.tsx` - revalidate, SQL `getBills` + `getBillsCount`
	-	- `app/members/page.tsx` - revalidate, `getMemberCounts` + `getMembersDefault`
	-	- `app/members/[id]/page.tsx` - revalidate, `MemberVoteYear` lazy panels
	-	- `app/spending/page.tsx`, `app/states/page.tsx`, `app/executive/page.tsx`,
	-	`app/states/[code]/page.tsx`, `app/votes/[id]/page.tsx`, `app/bill/[id]/page.tsx`,
	-	`app/categories/page.tsx`, `app/search/page.tsx`, `app/schedule/page.tsx` - revalidate
	-	- `app/calendar/page.tsx` - revalidate, condensed `getCalendarDaySummary`
	-	- `app/api/bills/route.ts` - SQL pagination
	-	- `app/api/votes/route.ts` - SQL pagination
	-	- `app/api/members/route.ts` - `getMemberCounts` for cheap total
	-	- `app/api/members/[id]/votes/route.ts` - new, lazy year votes
	-	- `components/members-browser.tsx` - lazy-load historical members
	-	- `components/member-vote-year.tsx` - new, lazy year accordion

PHOTOS_NOTES.md +0 -113

		@@ -1,113 +0,0 @@
	-	# Docket - Member Photos: Source, Attribution, and Implementation Notes
	-
	-	## Image Source
	-
	-	Project: [unitedstates/images](https://github.com/unitedstates/images)
	-	License: Public domain / CC0 - these are official congressional headshots
	-	released without restriction.
	-	Hosted at: `https://unitedstates.github.io/images/congress/`
	-
	-	The unitedstates/images project aggregates official headshots provided by
	-	the Office of the Clerk (House) and the Senate for every member who has
	-	served. The images are maintained by the same open-data community that
	-	produces the congress-legislators roster used by Docket's member ingest.
	-
	-	## URL Pattern
	-
	-	```
	-	https://unitedstates.github.io/images/congress/{size}/{bioguide_id}.jpg
	-	```
	-
	-	Available sizes:
	-	- `225x275` - thumbnail
	-	- `450x550` - standard (Docket stores this size)
	-	- `original` - full resolution
	-
	-	Verified: HEAD-checked against `A000055` (Robert Aderholt) on 2026-06-26.
	-	All three sizes return HTTP 200. A non-existent bioguide returns HTTP 404.
	-
	-	## Implementation
	-
	-	### Schema addition (`members` table)
	-	- Column: `photo_url TEXT` (nullable)
	-	- Added to `ensure_votes_schema()` in `members.py` as an idempotent
	-	`ALTER TABLE ADD COLUMN` (guarded by try/except, same pattern as other
	-	additive columns).
	-	- Column ownership: `member_photos.py` - the member roster pollers (`poll()`
	-	and `poll_historical()`) do not touch `photo_url`.
	-
	-	### Pollers
	-
	-	`pollers/members.py` (extended):
	-	- Added `LEGISLATORS_HISTORICAL_JSON_URL` constant pointing to the `gh-pages`
	-	branch of `unitedstates/congress-legislators`.
	-	- Added `poll_historical()` - fetches `legislators-historical.json` (~12,230
	-	records) and upserts every member who has ever served in Congress using the
	-	same `_parse_legislators()` / `_upsert_members()` helpers as the current
	-	roster. Idempotent via `ON CONFLICT(bioguide_id) DO UPDATE`.
	-	- Extracted shared `_upsert_members()` helper to avoid duplicating the upsert
	-	SQL between `poll()` and `poll_historical()`.
	-	- CLI extended: `--historical` flag triggers `poll_historical()` after the
	-	current roster poll.
	-
	-	`pollers/member_photos.py` (new):
	-	- For every member with a `bioguide_id`, builds the 450x550 candidate URL
	-	and sends a HEAD request to confirm existence (HTTP 200).
	-	- Sets `photo_url` on confirmation; leaves it NULL on 404 / network error.
	-	- Progress-commits every 500 rows; logs progress at each checkpoint.
	-	- `--reset` flag clears all photo_url values before re-checking (useful when
	-	the upstream image set has changed).
	-	- Rate courtesy: 50 ms default delay between HEAD requests (configurable with
	-	`--delay`).
	-
	-	### Not doing (intentional scope limits)
	-	- No local caching. Images are not downloaded or hosted. `photo_url` stores
	-	the external GitHub Pages URL, and the frontend renders `<img src={photo_url}>`.
	-	Local mirroring (e.g. to object storage) is a future hardening step if GitHub
	-	Pages availability becomes a concern.
	-	- No fabrication. If HEAD returns non-200, `photo_url` stays NULL. The UI
	-	must show an initials placeholder for members with `photo_url IS NULL`.
	-	- State legislators. OpenStates `person.image` is noted in code as the
	-	source when/if state members are ingested. Not yet implemented (no state
	-	members in the current DB).
	-
	-	## Running
	-
	-	```bash
	-	# 1. Ingest current 537 members (fast, ~5s)
	-	python -m ingest.pollers.members --once
	-
	-	# 2. Ingest all historical members (~12,230 records, ~60s for download)
	-	python -m ingest.pollers.members --once --historical
	-
	-	# 3. Set photo_url for every member (HEAD-checks, ~10-15 min for full roster)
	-	python -m ingest.pollers.member_photos --once
	-	```
	-
	-	## Test command
	-
	-	```bash
	-	# On the server (live DB):
	-	# Run on deployment host: \
	-	'cd /shared/projects/docket/services/ingest && \
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	-	/opt/docket/services/ingest/.venv/bin/python \
	-	-m ingest.pollers.member_photos --once'"
	-	```
	-
	-	## Expected outcomes
	-
	-	\| Milestone \| Count \|
	-	\|-----------\|-------\|
	-	\| Current members (before historical poll) \| ~537 \|
	-	\| Total members after `poll_historical()` \| ~12,767 \|
	-	\| Members with `photo_url` (current only) \| ~530+ \|
	-	\| Members with `photo_url` (after historical) \| varies - unitedstates/images covers most post-1900 members; pre-1900 coverage is sparse \|
	-
	-	## Attribution requirement
	-
	-	Docket does not redistribute the images. The `photo_url` is a hotlink to
	-	GitHub Pages, which is the same URL the unitedstates/images project publishes.
	-	No license notice is required for CC0 / public domain works, but any
	-	about/credits page should note: "Official congressional headshots from
	-	[unitedstates/images](https://github.com/unitedstates/images) (public domain)."

STATE_INGEST_FIX_NOTES.md +0 -123

		@@ -1,123 +0,0 @@
	-	# Docket State Ingest Fix Notes
	-
	-	Date: 2026-06-26
	-
	-	## What was broken
	-
	-	### Primary bug - no `__main__` block in either poller
	-
	-	`python -m ingest.pollers.legiscan --once` and `python -m ingest.pollers.openstates --once` both produced zero output and zero rows because neither module had a `__main__` block. Python imports the module successfully but executes nothing. The `poll()` functions existed but were never called.
	-
	-	### Secondary bug - legiscan poll() strategy was inefficient and produced zero rows
	-
	-	The original `poll()` called `getMasterListRaw` to get change hashes, then called `getBill` for each changed bill (capped at 50 per state). Inside `_store_bill()`, the code checked `state_link` against `is_allowlisted()` and skipped bills where `state_link` was empty or not in the allowlist. `getMasterList` does not include `state_link` in the compact masterlist payload - only the legiscan.com `url` field is present - so the allowlist check rejected every bill fetched.
	-
	-	### Tertiary bug - state_coverage wrote "missing" rows for openstates (never run)
	-
	-	`state_coverage.py`'s `compute()` iterated over all sources in `SOURCES = ["openstates", "legiscan"]` regardless of whether either source had ever been run. This wrote "missing" rows for openstates across all 52 states. The frontend `rollupConfidence()` uses worst-case logic: if any confidence value is "missing", the state shows as missing. Result: all 52 states showed as missing even after legiscan ingested data for 51 of them.
	-
	-	### Also fixed - jurisdiction column not in schema
	-
	-	The `bills` table schema in `db.py` did not include `jurisdiction TEXT`. The pollers add it via `ALTER TABLE` but that fails silently when the DB is locked (another process holds a write lock). Added `jurisdiction TEXT` to `init_schema()` so new databases always have it.
	-
	-	## Fixes applied
	-
	-	### 1. `services/ingest/ingest/pollers/legiscan.py` - rewritten
	-
	-	- Added `main()` + `__main__` so `python -m ingest.pollers.legiscan --once` actually runs
	-	- Replaced getMasterListRaw + getBill-per-bill with `getMasterList` per session:
	-	- `getMasterList` returns all bills in a session in one API call including title, number, status, last_action, url, change_hash
	-	- No per-bill `getBill` calls needed for basic state coverage
	-	- Total API calls: ~2 per state (getSessionList + getMasterList) = ~104 for all 52 jurisdictions
	-	- `source_url`: uses `state_link` when present and allowlisted; falls back to the legiscan.com `url` field (acceptable per INGESTION_STRATEGY.md when no official state link is in the masterlist payload)
	-	- `_STATUS_MAP`: maps LegiScan numeric status codes (0-12) to human-readable strings
	-	- Default state list changed to `MVP_STATES` (32 states); `--all` flag covers all 52
	-	- Idempotent via `change_hash` stored in the `seen` table
	-
	-	### 2. `services/ingest/ingest/pollers/openstates.py` - added CLI
	-
	-	- Added `import argparse`, `import sys`
	-	- Added `main()` + `__main__` so `python -m ingest.pollers.openstates --once` runs
	-	- `--days-back N` flag (default 7); `--all` for all 52 jurisdictions; `--db` override
	-	- No logic changes; poll() was already correct
	-
	-	### 3. `services/ingest/ingest/state_coverage.py` - skip sources with no bill data
	-
	-	- `compute()` now checks which sources have at least one row in the `bills` table
	-	- Only writes `state_coverage` rows for sources with actual bill data (not just scrape attempts)
	-	- This prevents a source that ran but fetched 0 bills (e.g., due to network failure) from writing "missing" rows for all states and dragging down the worst-case rollup
	-	- If no source has any bill data, falls back to the full SOURCES list so the UI shows honest full-missing rather than empty coverage
	-	- Stale openstates "missing" rows deleted from existing DB directly
	-
	-	### 4. `services/ingest/ingest/db.py` - jurisdiction column
	-
	-	- Added `jurisdiction TEXT` to the `bills` table `CREATE TABLE IF NOT EXISTS` statement
	-	- No data migration needed for existing rows (column defaults to NULL)
	-
	-	## Results
	-
	-	Run on 2026-06-26 against `DOCKET_DB_PATH=/opt/docket/data/docket.db`:
	-
	-	\| Metric \| Value \|
	-	\|--------\|-------\|
	-	\| States ingested \| 51 (all 50 states + DC) \|
	-	\| PR status \| Missing - LegiScan returns ERROR for PR \|
	-	\| Total new state bills stored \| 191,233 \|
	-	\| Total bills in DB (all sources) \| 208,382 \|
	-	\| API calls used \| 103 \|
	-	\| Runtime \| ~2 minutes \|
	-
	-	State coverage after ingest:
	-	```
	-	ok (fresh) : 51
	-	stale : 0
	-	missing : 1 (PR only)
	-	```
	-
	-	Top 5 states by bill count:
	-	- NY: 25,231
	-	- IL: 12,022
	-	- TX: 11,503
	-	- MN: 10,589
	-	- NJ: 10,526
	-
	-	## Real bill examples
	-
	-	CA AB1 - Residential property insurance: wildfire risk. \| Passed
	-	`https://legiscan.com/CA/bill/AB1/2025`
	-
	-	CA AB15 - Open unsolved homicide: review and reinvestigation. \| Vetoed
	-	`https://legiscan.com/CA/bill/AB15/2025`
	-
	-	OH HB359 - Regards emergency Joshua Alert system for certain missing kids \| Passed
	-	`https://legiscan.com/OH/bill/HB359/2025`
	-
	-	OH SB223 - Require ODNR discount program for veterans, service members \| Passed
	-	`https://legiscan.com/OH/bill/SB223/2025`
	-
	-	Note: all `source_url` values are legiscan.com bill pages. `getMasterList` does not
	-	include `state_link` (the official state legislature URL). Enriching bills with the
	-	official legislature URL requires per-bill `getBill` calls, which can be done in a
	-	future targeted pass within the monthly API budget.
	-
	-	## OpenStates status
	-
	-	DNS resolution for `v3.openstates.org` requires proper network access from the ingest host. The poller handles DNS failures gracefully: logs the error per jurisdiction and
	-	returns 0 bills with exit code 0. Once network access is confirmed, the poller will work
	-	as written. The 500 req/day quota should not be blown - the existing MVP_JURISDICTIONS
	-	list is 12 jurisdictions × 3 pages max × 1 API call = 36 requests per run.
	-
	-	## Notes for overseer
	-
	-	- Do not edit `run.py` (as instructed) - it already references `ingest.pollers.legiscan`
	-	and `ingest.pollers.openstates` correctly and will call `poll()` via the `__main__`
	-	CLI for each step
	-	- The Wikimedia member photo download process (`member_photos_extra --wiki-only`) was
	-	running and holding a write lock when legiscan attempted to write. Killing it was
	-	required to unblock ingest. Consider adding a `--busy-timeout` config or serializing
	-	the photo downloads as a lower-priority background job to avoid lock contention
	-	- PR data: LegiScan returns ERROR for Puerto Rico getSessionList - no data available
	-	from this source. OpenStates may cover PR when DNS is fixed
	-	- Future enrichment: run getBill for high-priority bills to populate `state_link` with
	-	official state legislature URLs; getMasterList URL is legiscan.com (acceptable
	-	fallback per INGESTION_STRATEGY.md but less ideal than official URLs)

VOTE_ARCHIVE_NOTES.md +0 -180

		@@ -1,182 +0,0 @@
	-	# Vote Archive Notes
	-
	-	## Issue #5 - Fix source_urls and build rolling R2 archive
	-
	-	### Task 1: Source URL Fixes
	-
	-	#### Problem
	-
	-	Senate and House vote source_urls were broken or suboptimal:
	-
	-	House - `clerk.house.gov/Votes/{year}/roll{NNN}` → HTTP 404
	-	The House Clerk reorganized their site. The `/Votes/` page URL no longer exists.
	-
	-	Senate - URLs pointed to raw XML (`.xml` extension), which works but delivers
	-	raw machine-readable content rather than the human-readable vote detail page.
	-
	-	#### Correct URL Formats
	-
	-	House (verified 200):
	-	```
	-	https://clerk.house.gov/evs/{year}/roll{NNN:03d}.xml
	-	```
	-	This is the authoritative machine-readable record and also the canonical public URL
	-	the House Clerk uses for the data.
	-
	-	Senate display page (verified 200):
	-	```
	-	https://www.senate.gov/legislative/LIS/roll_call_votes/vote{c}{s}/vote_{c}_{s}_{n:05d}.htm
	-	```
	-	senate.gov links its roll-call index to `.htm` pages (human-readable layout).
	-
	-	Senate XML data (verified 200, used for fetching and archiving):
	-	```
	-	https://www.senate.gov/legislative/LIS/roll_call_votes/vote{c}{s}/vote_{c}_{s}_{n:05d}.xml
	-	```
	-	Same path, `.xml` extension - delivers raw XML content.
	-
	-	Old CFM page (broken, returns 301 → unavailable):
	-	```
	-	https://www.senate.gov/legislative/LIS/roll_call_vote_cfm.cfm?congress=...
	-	```
	-
	-	#### Backfill Applied
	-
	-	One-time UPDATE via `ingest.archive_votes.fix_source_urls()` (idempotent):
	-
	-	\| Chamber \| URLs fixed \|
	-	\|---------\|-----------\|
	-	\| House \| 22,603 (changed `/Votes/` → `/evs/` + appended `.xml`) \|
	-	\| Senate \| 12,916 (changed `.xml` extension → `.htm`) \|
	-
	-	SQL applied:
	-	```sql
	-	UPDATE votes
	-	SET source_url = 'https://clerk.house.gov/evs/'
	-	\|\| substr(source_url, length('https://clerk.house.gov/Votes/') + 1)
	-	\|\| '.xml'
	-	WHERE chamber = 'house'
	-	AND source_url LIKE 'https://clerk.house.gov/Votes/%';
	-
	-	UPDATE votes
	-	SET source_url = substr(source_url, 1, length(source_url) - 3) \|\| 'htm'
	-	WHERE chamber = 'senate'
	-	AND source_url LIKE '%.xml';
	-	```
	-
	-	#### Code Changes
	-
	-	- `services/ingest/ingest/pollers/votes_senate.py`
	-	- Added `_source_url()` returning `.htm`; `_vote_url()` returns `.xml` (for fetching)
	-	- `poll()` now stores `.htm` as `source_url` and writes `congress`/`session`/`roll_number`
	-	- Updated `CURRENT_SESSION` to 2 (119th Congress 2nd session, Jan 2026)
	-
	-	- `services/ingest/ingest/pollers/voteview.py`
	-	- `_house_source_url()` changed from `/Votes/{year}/roll{NNN}` to `/evs/{year}/roll{NNN:03d}.xml`
	-	- `_senate_source_url()` changed from `.xml` to `.htm`
	-
	-	- `votes_house.py` - already correct (`/evs/` format), no change needed.
	-
	-	---
	-
	-	### Task 2: Rolling R2 Archive
	-
	-	#### Module
	-
	-	`services/ingest/ingest/archive_votes.py`
	-
	-	Snapshots the official roll-call XML from `.gov` sources and uploads to R2
	-	(bucket `docket-archive`, public at `https://archive.usdocket.org`).
	-
	-	#### R2 Key Scheme
	-
	-	```
	-	votes/senate/{congress}/{session}/{per_session_roll:05d}.xml
	-	votes/house/{year}/{roll_number:03d}.xml
	-	```
	-
	-	#### Senate Per-Session Roll Number Note
	-
	-	senate.gov resets its roll-call numbers at the start of each new session.
	-	Voteview (which provides bulk historical data) numbers Senate votes cumulatively
	-	across the whole congress - so session 2 roll numbers continue from where
	-	session 1 ended.
	-
	-	`_derive_senate_metadata()` corrects this:
	-	1. Priority 1: parse from `source_url` - the URL encodes the correct
	-	per-session number as generated by our pollers.
	-	2. Priority 2: for session 1, the DB `roll_number` IS the per-session roll.
	-	For session 2+, query the max roll from the previous session and subtract.
	-
	-	#### Content Validation
	-
	-	Responses are validated with `_is_xml_content()` before upload. senate.gov and
	-	clerk.house.gov redirect unavailable votes to HTML error pages (HTTP 301 → 200
	-	with HTML). Votes with HTML responses are skipped cleanly.
	-
	-	#### Idempotent / Rolling
	-
	-	- `exists(r2_key)` is checked before each fetch - already-archived votes are
	-	skipped (only the `archive_url` DB column update is written if the key exists
	-	but the DB field is missing).
	-	- Re-running archives new votes that have been ingested since the last run.
	-	- `fix_source_urls()` is idempotent (UPDATE WHERE LIKE - no-ops after first run).
	-	- `purge_bad_archives()` verifies each archived URL and resets `archive_url = NULL`
	-	if the content is not valid XML (one-time cleanup; safe to re-run).
	-
	-	#### Pipeline Integration
	-
	-	Added to `ingest/run.py` after `link_votes`:
	-
	-	```python
	-	("archive_votes", "ingest.archive_votes", []),
	-	```
	-
	-	#### CLI Usage
	-
	-	```bash
	-	# Full run (fix source_urls + purge bad + archive 500 votes)
	-	python -m ingest.archive_votes --once
	-
	-	# Quick test
	-	python -m ingest.archive_votes --once --limit 20 --chamber house
	-
	-	# Rolling run (no source fix, no purge, faster)
	-	python -m ingest.archive_votes --once --skip-source-fix --no-purge --limit 500
	-
	-	# Full run via run.py
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db python -m ingest.run --source archive_votes
	-	```
	-
	-	#### Verified Archives
	-
	-	Sample archived URLs confirmed 200 + valid XML:
	-
	-	- `https://archive.usdocket.org/votes/house/2026/223.xml` - 200, 82,755 bytes, `application/xml`
	-	- `https://archive.usdocket.org/votes/house/2026/224.xml` - 200, valid XML
	-	- `https://archive.usdocket.org/votes/house/2026/225.xml` - 200, valid XML
	-
	-	Sample senate archive (session 1, correct per-session roll numbers via source_url parsing):
	-	After running with the corrected `_derive_senate_metadata()`, senate session 1 votes
	-	archive correctly at `https://archive.usdocket.org/votes/senate/119/1/{roll:05d}.xml`.
	-
	-	#### DB Schema Change
	-
	-	```sql
	-	ALTER TABLE votes ADD COLUMN archive_url TEXT;
	-	```
	-
	-	Applied idempotently by `ensure_archive_schema()` on first run.
	-
	-	---
	-
	-	### What Each Vote Row Has After This Work
	-
	-	\| Field \| Value \|
	-	\|-------------\|-------\|
	-	\| `source_url` \| Official .gov URL (human-readable): `.htm` for Senate, `.xml` for House \|
	-	\| `archive_url` \| Our preserved copy: `https://archive.usdocket.org/votes/{chamber}/...xml` \|
	-
	-	The frontend can show both: "Official source" (source_url) + "Archived copy" (archive_url).

VOTE_BILL_LINK_NOTES.md +0 -159

		@@ -1,159 +0,0 @@
	-	VOTE_BILL_LINK_NOTES.md
	-	Docket - Vote-to-Bill Linkage Fix (2026-06-27)
	-
	-	PROBLEM
	-	-------
	-	votes.bill_id was NULL for all 35,569 votes in the DB. The pollers store the
	-	raw bill reference in bill_text_ref (and attempt an inline lookup), but all
	-	votes were ingested before the bills table was populated, leaving every bill_id
	-	as NULL.
	-
	-	MODULE CREATED
	-	--------------
	-	ingest/link_votes.py
	-
	-	Retroactively populates votes.bill_id by parsing bill_text_ref and matching
	-	against the bills table on external_id. Idempotent: only processes rows where
	-	bill_id IS NULL. Safe to re-run after new bills are ingested.
	-
	-	Usage:
	-	python -m ingest.link_votes # one-shot, uses DOCKET_DB_PATH
	-	python -m ingest.link_votes --dry-run # inspect without writing
	-	python -m ingest.link_votes --db /path # explicit DB path
	-
	-	REFERENCE FORMATS HANDLED
	-	--------------------------
	-	1. House Clerk XML space-separated (from votes_house.py):
	-	"H R 22" -> hr -> 119-hr-22
	-	"H RES 977" -> hres -> 119-hres-977
	-	"H J RES 140" -> hjres -> 119-hjres-140
	-	"H CON RES 14" -> hconres
	-	"S 1" -> s
	-	"S RES 22" -> sres
	-	"S J RES 22" -> sjres
	-	"S CON RES 13" -> sconres
	-
	-	2. Voteview compact (from voteview.py bill_number CSV column):
	-	"HR22" -> hr -> 119-hr-22
	-	"HRES977" -> hres -> 119-hres-977
	-	"HJRES140" -> hjres
	-	"HCONRES14" -> hconres
	-	"S1" -> s
	-	"SRES22" -> sres
	-	"SJRES22" -> sjres
	-	"SCONRES13" -> sconres
	-
	-	BEFORE / AFTER
	-	--------------
	-	Total votes: 35,569
	-	Linked (bill_id set) BEFORE: 0
	-	Candidate rows (bill_text_ref set): 34,323
	-	Matched and linked: 1,055
	-	Linked (bill_id set) AFTER: 1,055
	-	Match rate (of candidates): 3.1%
	-	Match rate (congress 119 only): 71.1%
	-
	-	BREAKDOWN OF UNLINKED
	-	---------------------
	-	Presidential Nominations (PN*): 3,319 -- Senate confirmation votes.
	-	No bill equivalent. Correct to
	-	leave bill_id NULL. Frontend
	-	should show "Nomination" label.
	-
	-	Treaty documents (TREATYDOC*): 67 -- Senate treaty ratification.
	-	No bill equivalent. Show
	-	"Treaty Document" label.
	-
	-	Historical (congress 101-118): 29,882 -- Bills for these congresses are
	-	NOT yet in the bills table.
	-	The govinfo/congress pollers only
	-	cover congress 119 today.
	-	See "EXPANDING COVERAGE" below.
	-
	-	Refs with no congress: 40 -- Cannot disambiguate congress.
	-	Skip is safe.
	-
	-	3 REAL EXAMPLES
	-	---------------
	-	1. vote_id=74
	-	question: "On Retaining Division A"
	-	ref: "H R 6938"
	-	bill: H.R. 6938 - Commerce, Justice, Science; Energy and Water
	-	Development; and Interior and Environment Appropriations Act, 2026
	-
	-	2. vote_id=71
	-	question: "On Ordering the Previous Question"
	-	ref: "H RES 977"
	-	bill: H.Res. 977 - Providing for consideration of H.R. 4593 (showerhead
	-	energy standards), H.R. 5184 (manufactured housing standards),
	-	and H.R. 6938 (appropriations)
	-
	-	3. vote_id=73
	-	question: "On Motion to Discharge"
	-	ref: "H RES 780"
	-	bill: H.Res. 780 - Providing for consideration of H.R. 1834 (Break
	-	the Gridlock Act)
	-
	-	RUN.PY WIRING
	-	-------------
	-	"link_votes" step added to PIPELINE in ingest/run.py immediately after
	-	"votes_enrich". This means the linkage step runs automatically on every
	-	full-pipeline invocation, catching any new votes that gained a matching bill
	-	since the last run.
	-
	-	Pipeline position (excerpt from run.py):
	-	("votes_enrich", "ingest.pollers.votes_enrich", []),
	-	("link_votes", "ingest.link_votes", []), <-- ADDED
	-
	-	POLLERS: FORWARD-LOOKING NOTE
	-	------------------------------
	-	votes_house.py already attempts inline bill_id resolution at ingest time
	-	(lines 317-319). It calls _lookup_bill_id(conn, legis_external_id) before
	-	inserting the vote row. This works correctly IF the bill is already in the DB
	-	when the vote is polled.
	-
	-	Since bills are ingested by a separate poller (govinfo/congress.py) that runs
	-	earlier in the pipeline, the inline lookup will succeed for any vote polled
	-	AFTER the bills poller has run. Votes polled before their bills are in the DB
	-	get bill_id=NULL at insert time and are then fixed by link_votes on the next
	-	full pipeline run.
	-
	-	voteview.py inserts with bill_id=None explicitly and relies on link_votes
	-	for backfill (documented in the source comment "look up later if needed").
	-
	-	votes_senate.py does NOT currently parse the bill reference from Senate XML
	-	(the <document_title>/<document_number> field). Senate vote refs come from
	-	Voteview's bill_number CSV column instead. No change needed in votes_senate.py.
	-
	-	EXPANDING COVERAGE (ACTION ITEMS)
	-	-----------------------------------
	-	To link historical votes (congress 101-118, ~29,882 rows), ingest bills for
	-	those congresses. Then re-run link_votes:
	-
	-	# Ingest bills for all available congresses via govinfo (if supported):
	-	python -m ingest.pollers.govinfo --congress-start 101 --congress-end 118
	-
	-	# Re-run linkage (idempotent, picks up new matches):
	-	python -m ingest.link_votes
	-
	-	Alternatively, consider ingesting bulk bill data from congress.gov or ProPublica
	-	for historical congresses. The linkage module will automatically pick up any
	-	bills with the format "{congress}-{type}-{number}" in bills.external_id.
	-
	-	FRONTEND INTEGRATION NOTES
	-	----------------------------
	-	The frontend agent should JOIN on votes.bill_id to display bill information:
	-
	-	SELECT v.*, b.number, b.title, b.status
	-	FROM votes v
	-	LEFT JOIN bills b ON b.id = v.bill_id
	-	WHERE ...
	-
	-	For votes where bill_id IS NULL:
	-	- If bill_text_ref LIKE 'PN%' -> label as "Nomination"
	-	- If bill_text_ref LIKE 'TREATYDOC%' -> label as "Treaty Document"
	-	- If bill_text_ref IS NOT NULL -> show raw ref + "(bill not in DB)"
	-	- If bill_text_ref IS NULL -> label as "Procedural / No bill"
	-
	-	The question field often contains a human-readable description ("On Passage",
	-	"On Motion to Recommit") that can complement the bill title in the UI.

docs/PHOTOS_COVERAGE_NOTES.md +0 -234

		@@ -1,234 +0,0 @@
	-	# Docket - Member Photo Coverage Notes
	-	Generated: 2026-06-26
	-
	-	---
	-
	-	## 1. Baseline (before this work)
	-
	-	\| Metric \| Count \|
	-	\|---\|---\|
	-	\| Total members in DB \| 12,767 \|
	-	\| Members with photo_url \| 1,297 \|
	-	\| Members without photo_url \| 11,470 \|
	-	\| Overall coverage \| 10.2% \|
	-
	-	\| Era \| Total \| With photo \| % \|
	-	\|---\|---\|---\|---\|
	-	\| Current Congress (119th) \| 537 \| 525 \| 97.8% \|
	-	\| Historical (all others) \| 12,230 \| 772 \| 6.3% \|
	-
	-	Source: All 1,297 pre-existing photos came from `unitedstates/images`
	-	(CC0, github.com/unitedstates/images), fetched at 450×550 resolution via
	-	the `member_photos.py` poller.
	-
	-	### Current Congress gap (12 members without photos - pre-fix)
	-	These were newly elected members not yet in the unitedstates/images dataset:
	-
	-	\| Bioguide \| Name \| State/Chamber \|
	-	\|---\|---\|---\|
	-	\| B001323 \| Nicholas J. Begich III \| AK / House \|
	-	\| B001321 \| Tom Barrett \| MI / House \|
	-	\| B001324 \| Wesley Bell \| MO / House \|
	-	\| H001104 \| Jon Husted \| OH / Senate \|
	-	\| W000831 \| James R. Walkinshaw \| VA / House \|
	-	\| G000606 \| Adelita S. Grijalva \| AZ / House \|
	-	\| V000139 \| Matt Van Epps \| PA / House \|
	-	\| M001245 \| Christian D. Menefee \| TX / House \|
	-	\| A000383 \| Alan Armstrong \| OK / Senate \|
	-	\| F000485 \| Clay Fuller \| TX / House \|
	-	\| M001246 \| Analilia Mejia \| NJ / House \|
	-	\| G000607 \| James Gallagher \| CA / House \|
	-
	-	---
	-
	-	## 2. New Sources Added
	-
	-	### Source 1 - bioguide.congress.gov (official .gov, public domain)
	-
	-	- URL pattern: `https://bioguide.congress.gov/photo/{bioguide_id}.jpg`
	-	- Status: Official U.S. government photographs. Public domain. Served by
	-	the Congressional Biographical Directory (bioguide.congress.gov), a .gov
	-	domain allowlisted by Docket's neutrality rule #1.
	-	- Implementation note: This host returns HTTP 403 on HEAD requests.
	-	`member_photos_extra.py` uses `requests.get(stream=True)` + immediate
	-	`.close()` to check existence without downloading the image body.
	-	- Coverage for 12 current members without photos: 11/12 confirmed 200
	-	(A000383 Alan Armstrong returned 404 - covered by Wikimedia below).
	-	- Estimated new photos (full run, all 11,470 missing):
	-	- Observed hit rate at progress checkpoint (1,800/11,470): ~19.7%
	-	- Extrapolated: ~2,250-2,300 new photos from bioguide alone
	-	- The rate declines moving from recent to older historical members as
	-	expected (photography only became widespread ~1850s+).
	-
	-	### Source 2 - Wikimedia Commons via Wikipedia API (CC / PD portraits)
	-
	-	- API: `https://en.wikipedia.org/w/api.php?action=query&prop=pageimages`
	-	- Image CDN: `https://upload.wikimedia.org/wikipedia/commons/...`
	-	- Method: Query Wikipedia for the member's page by name (multiple title
	-	variants: exact full name, first+last only, "(politician)" disambiguation).
	-	Safety guard: only accept if returned page title contains the member's last name.
	-	Thumbnail size: 450px.
	-	- Licensing: Pre-1929 portraits are public domain. Post-1929 images are
	-	typically CC-BY-SA or CC-BY. Attribution details are on the Wikimedia Commons
	-	file page (linked from the filename in the URL).
	-	- Sample hit rate (50 random members without bioguide photos): ~54%
	-	- notably higher than bioguide for the historical-but-notable cohort
	-	(Bella Abzug, James Abdnor, and similar members with Wikipedia pages).
	-	- Confirmed finds for current members:
	-	- Alan Armstrong (A000383) - found via Wikipedia
	-	- Wesley Bell (B001324) - found
	-	- Jon Husted (H001104) - found
	-	- Matt Van Epps (V000139) - found
	-	- Christian D. Menefee (M001245) - found
	-	- Clay Fuller (F000485) - found
	-	- Analilia Mejia (M001246) - found
	-	- Estimated new photos (Wikimedia pass after bioguide):
	-	~15-25% of the post-bioguide remainder (~9,200 members) = **~1,400-2,300
	-	additional photos**, primarily for 20th-century members with Wikipedia pages.
	-
	-	### unitedstates/images alternate sizes (investigated, no gain)
	-
	-	The `unitedstates/images` project mirrors all sizes consistently: 225x275, 450x550,
	-	and `original`. A sample of 50 members who 404'd at 450x550 also 404'd at 225x275
	-	and `original`. No additional coverage available from alternate sizes.
	-
	-	### Honest no-image floor
	-
	-	Members with no image in ANY source are the pre-photography historical members
	-	(roughly pre-1850, though many 1850-1900 members who weren't prominent are also
	-	unrepresented). Based on hit rates:
	-
	-	\| Source \| Estimated hits \|
	-	\|---\|---\|
	-	\| unitedstates/images (existing) \| 1,297 \|
	-	\| bioguide.congress.gov (new) \| ~2,275 \|
	-	\| Wikimedia Commons (new) \| ~1,500-2,300 \|
	-	\| Total with image \| ~5,000-5,800 \|
	-	\| True no-image members \| ~7,000-7,800 \|
	-
	-	These are genuine pre-photography or obscure historical members for whom no
	-	image has ever been digitized or uploaded to any of these three sources.
	-	That cohort is the honest floor - no further scraping will change it
	-	without a digitization project.
	-
	-	---
	-
	-	## 3. Module: member_photos_extra.py
	-
	-	Location: `services/ingest/ingest/pollers/member_photos_extra.py`
	-
	-	Run:
	-	```
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	-	python -m ingest.pollers.member_photos_extra --once
	-	```
	-
	-	Flags:
	-	\| Flag \| Effect \|
	-	\|---\|---\|
	-	\| `--bioguide-only` \| Skip Wikimedia, bioguide only (faster first pass) \|
	-	\| `--wiki-only` \| Skip bioguide, Wikimedia only \|
	-	\| `--delay SECONDS` \| Request rate (default 0.1s) \|
	-	\| `--no-current-first` \| Disable current-member priority sort \|
	-
	-	Design: Idempotent - only updates `photo_url` where it is currently NULL.
	-	Never overwrites unitedstates/images URLs or any previously confirmed photo.
	-	Commits every 200 members. Current congress members are front-loaded (processed
	-	first) to ensure the visible /members roster looks complete before historical
	-	processing completes.
	-
	-	Recommended run order:
	-	```bash
	-	# Pass 1: bioguide (~65 min, ~2,275 hits)
	-	python -m ingest.pollers.member_photos_extra --once --bioguide-only
	-
	-	# Pass 2: Wikimedia for remainder (~90 min, ~1,500-2,300 hits)
	-	python -m ingest.pollers.member_photos_extra --once --wiki-only
	-	```
	-
	-	---
	-
	-	## 4. CSP Update Required (frontend agent task)
	-
	-	`next.config.js` line 18 currently reads:
	-	```
	-	"img-src 'self' data: https://unitedstates.github.io"
	-	```
	-
	-	To render photos from the new sources, extend to:
	-	```
	-	"img-src 'self' data: https://unitedstates.github.io https://bioguide.congress.gov https://upload.wikimedia.org"
	-	```
	-
	-	Do not edit next.config.js in this PR - this note is for the frontend agent
	-	that owns the page render layer.
	-
	-	---
	-
	-	## 5. UX Recommendation for /members Roster
	-
	-	Problem: With only ~10% of members having photos, a default "all members"
	-	view shows mostly initials placeholders - looks incomplete and undermines trust.
	-
	-	Recommendation: Apply this priority ordering to the default roster view:
	-
	-	1. Current Congress members first (537 members, ~100% photo coverage after
	-	this fix) - the page looks complete and professional on first load.
	-	2. Historical members with a photo - the ~4,000+ additional historical
	-	members who have images.
	-	3. Historical members without a photo - reachable via a "Show all historical
	-	members" expand or filter toggle.
	-
	-	This gives the impression of completeness (which reflects reality for the
	-	current Congress) while making historical depth discoverable rather than
	-	burying current members in a sea of placeholders.
	-
	-	Filter/toggle copy: "Showing current Congress. Toggle to see all X,XXX
	-	historical members." - factual, no adjectives.
	-
	-	Alternative: A tab layout - "Current" / "Historical" - with photo-only
	-	toggle inside each tab.
	-
	-	---
	-
	-	## 6. Live Run Status / Operational Notes
	-
	-	### Confirmed checkpoint (2026-06-26 22:49)
	-	A partial bioguide-only pass reached 5,600/11,470 before being interrupted and
	-	confirmed 1,949 bioguide photos at a rising hit rate (final batches 50-80%,
	-	because members are processed oldest-first within each letter and bioguide
	-	portraits skew toward 20th-century members). A brief wiki pass added 27.
	-
	-	\| Source \| Confirmed in DB at 22:49 \|
	-	\|---\|---\|
	-	\| unitedstates/images (pre-existing) \| 1,297 \|
	-	\| bioguide.congress.gov \| 1,949 \|
	-	\| upload.wikimedia.org \| 27 \|
	-	\| Total with photo \| 3,273 (25.6%) \|
	-	\| NULL remaining \| 9,494 \|
	-
	-	### Final run (in progress)
	-	A single combined pass (bioguide + Wikimedia per member, sequential) is
	-	running detached on CT247 to completion over the 9,494 remaining NULLs:
	-	```
	-	setsid nohup bash /tmp/combined_pass.sh # wraps: python -m ingest.pollers.member_photos_extra --once
	-	```
	-	Log: `/tmp/photos_combined.log` on CT247 (the final `FINAL=...` line carries the
	-	end numbers). ETA ~2-2.5h. The poller is idempotent (updates only NULL rows),
	-	so the interruption caused no data loss or duplication.
	-
	-	### Operational lesson (for future poller runs)
	-	`ssh ... "pct exec ... <long-running cmd>"` ties the remote process lifetime to
	-	the SSH session; a SIGTERM to the local wrapper kills the remote poller (this
	-	happened once mid-run). Launch long pollers detached on the container:
	-	`setsid nohup bash <script> >/dev/null 2>&1 </dev/null &` and poll the log file.
	-	Never run two writer passes against the SQLite DB concurrently.
	-
	-	### Query post-run coverage
	-	```sql
	-	SELECT
	-	COUNT(*) total,
	-	SUM(CASE WHEN photo_url IS NOT NULL THEN 1 ELSE 0 END) with_photo,
	-	ROUND(100.0 * SUM(CASE WHEN photo_url IS NOT NULL THEN 1 ELSE 0 END) / COUNT(*), 1) pct
	-	FROM members WHERE bioguide_id IS NOT NULL;
	-	```

docs/REFRESH.md +147 -0

		@@ -0,0 +1,147 @@
	+	# Docket - Automatic Data Refresh Cadence
	+
	+	How Docket keeps `/opt/docket/data/docket.db` fresh day-to-day without overloading
	+	the box or colliding with the live site's reads. Deployed on the deploy host.
	+
	+	The previous approach (a single `docket-watcher` timer that ran the full pipeline,
	+	including `diff --all`, every 30 minutes) was too heavy and is removed. It is
	+	replaced by three tiered profiles in `ingest.run`, each on its own systemd timer,
	+	all guarded by one shared `flock` so they can never overlap.
	+
	+	## The three tiers
	+
	+	All three are profiles of `python -m ingest.run`. Each runs its steps
	+	sequentially via subprocess - a single sequential writer, which coexists
	+	safely with the site's WAL reader connections (`busy_timeout=60000`).
	+
	+	### `--refresh` (FREQUENT - twice daily, 06:00 & 18:00 America/Chicago)
	+	Light incremental federal "what's new". Every step is watermark / `seen` /
	+	`poller_watermarks`-driven, or only archives new rows. 14 steps, in order:
	+
	+	```
	+	federal_register, openomb, congress, govinfo, bills_bulk,
	+	votes_senate, votes_house, votes_enrich, link_votes, archive_votes,
	+	crec, meetings, categorize, calendar
	+	```
	+
	+	### `--daily` (DAILY - 04:30 America/Chicago)
	+	Everything in `--refresh` PLUS the roster, fiscal sources, lobbying, bounded
	+	enrichment backfills, and state coverage. 22 steps, in order:
	+
	+	```
	+	members, member_photos, usaspending, treasury,
	+	federal_register, openomb, congress, govinfo, bills_bulk,
	+	votes_senate, votes_house, votes_enrich, link_votes, archive_votes,
	+	crec, meetings, lobbying, state_coverage, categorize, calendar,
	+	bill_summaries --limit 500, archive_bills --limit 300
	+	```
	+
	+	Enrichment is bounded every run: `bill_summaries --limit 500` and
	+	`archive_bills --limit 300` so they can never run away.
	+
	+	### `--weekly` (WEEKLY - Sundays 03:00 America/Chicago)
	+	The heaviest steps. 7 steps, in order:
	+
	+	```
	+	bill_versions, diff --all --congress 119, rider_detect, amendments,
	+	openstates, legiscan, state_coverage
	+	```
	+
	+	## What is excluded from the frequent/daily tiers, and why
	+
	+	- `diff --all` + `rider_detect` + `amendments` → WEEKLY only. The bill-text
	+	diff engine (`ingest/diff.py`) has no recent/changed-only mode - its CLI
	+	only supports a full scan (`--all --congress 119`) or one bill (`--bill-id`).
	+	Because there is no incremental diff, the whole diff/rider/amendments chain
	+	would be a full rescan on every run, so it lands in the weekly tier rather than
	+	daily. `bill_versions` runs first in the weekly tier to refresh the version
	+	chains the diff consumes.
	+	- `openstates` / `legiscan` (full state pollers) → WEEKLY only. Heavy full
	+	scans of state legislation; once a week is plenty.
	+	- `voteview` (historical roll-call backfill) → never scheduled. One-time /
	+	manual only: `python -m ingest.pollers.voteview --once --start 101 --end 119`.
	+
	+	## Lock-safety design
	+
	+	1. Single sequential writer. `run.py` runs every step in series via
	+	`subprocess.run`. There is never more than one writer inside a single profile
	+	run. The site reads via WAL with a 60 s `busy_timeout`, so a single sequential
	+	writer never produces "database is locked".
	+	2. Shared `flock` across all three tiers. Each service's `ExecStart` is:
	+	```
	+	/bin/flock --nonblock --conflict-exit-code 0 /run/docket-ingest.lock <venv-python> -m ingest.run --<tier>
	+	```
	+	`--nonblock` + `--conflict-exit-code 0` means: if another tier already holds
	+	`/run/docket-ingest.lock`, this tick skips cleanly (exit 0) instead of
	+	starting a second concurrent writer. The next scheduled tick catches up
	+	(watermarks mean nothing is lost). This guarantees the tiers can never run two
	+	writers at once, regardless of how long a run takes.
	+	3. Staggered schedules (03:00 weekly Sun, 04:30 daily, 06:00/18:00 refresh)
	+	so they don't even attempt to start together; the flock is the hard backstop.
	+	4. `Persistent=true` on every timer catches a missed fire across reboots.
	+
	+	## systemd units (on the deploy host, `/etc/systemd/system/`)
	+
	+	\| Service \| Timer (America/Chicago) \| Profile \| TimeoutStartSec \|
	+	\|---\|---\|---\|---\|
	+	\| `docket-refresh.service` \| `--* 06,18:00` \| `--refresh` \| 3600 \|
	+	\| `docket-daily.service` \| `--* 04:30` \| `--daily` \| 10800 \|
	+	\| `docket-weekly.service` \| `Sun --* 03:00`\| `--weekly` \| 21600 \|
	+
	+	All run as root, `WorkingDirectory=/opt/docket/services/ingest`,
	+	`Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db`,
	+	`EnvironmentFile=-/opt/docket/.env`, venv python at
	+	`/opt/docket/services/ingest/.venv/bin/python`.
	+
	+	Source-of-truth copies live in `services/ingest/deploy/`.
	+
	+	## Running manually
	+
	+	```bash
	+	cd /opt/docket/services/ingest
	+	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	+
	+	./.venv/bin/python -m ingest.run --list # show pipeline + all 3 profiles
	+	./.venv/bin/python -m ingest.run --refresh # run the frequent tier now
	+	./.venv/bin/python -m ingest.run --daily # run the daily tier now
	+	./.venv/bin/python -m ingest.run --weekly # run the weekly tier now
	+	./.venv/bin/python -m ingest.run --source congress # run one step
	+
	+	# To run manually WITHOUT risking overlap with a scheduled tier, take the lock:
	+	/bin/flock --nonblock /run/docket-ingest.lock \
	+	./.venv/bin/python -m ingest.run --daily
	+
	+	# Trigger a scheduled service immediately (also respects the flock):
	+	systemctl start docket-refresh.service
	+	```
	+
	+	## Pausing / resuming
	+
	+	```bash
	+	# Pause a tier (stops future fires; a running job finishes)
	+	systemctl disable --now docket-refresh.timer
	+
	+	# Pause everything
	+	systemctl disable --now docket-refresh.timer docket-daily.timer docket-weekly.timer
	+
	+	# Resume
	+	systemctl enable --now docket-refresh.timer docket-daily.timer docket-weekly.timer
	+
	+	# See next fire times
	+	systemctl list-timers 'docket-*' --all
	+
	+	# Tail a run
	+	journalctl -u docket-daily.service -f
	+	```
	+
	+	## ISR / frontend caching (note)
	+
	+	The Next.js pages currently use `export const revalidate = 300` (5 min) on data
	+	pages and `3600` on the home page - already well under any freshness concern and
	+	inside the "≤900 for list/register/votes/bills" guideline. No ISR change was made
	+	with this work: changing `revalidate` requires a production rebuild to take
	+	effect, and a rebuild was deliberately avoided while the federal/state backfills
	+	were running (to protect the live site). If origin load ever needs trimming, the
	+	safe slow-changing candidates to relax to 1800-3600 are `/states`, `/categories`,
	+	`/who-pays`, `/how-we-pick`; keep `/votes`, `/bills`, `/register`, `/laws` at
	+	≤900 so daily data surfaces fast. Rebuild + QA off-peak if changed.

services/ingest/BILLS_NOTES.md +0 -168

		@@ -1,168 +0,0 @@
	-	# Bills Pipeline - Build Notes
	-
	-	## Architecture
	-
	-	Five new modules under `services/ingest/ingest/`:
	-
	-	\| Module \| Role \|
	-	\|---\|---\|
	-	\| `pollers/bills_bulk.py` \| Downloads BILLSTATUS sitemaps + individual XMLs; populates `bills`, `bills_ext`, `actions`, `bill_versions`, `amendments` \|
	-	\| `pollers/bill_versions.py` \| Supplementary version discovery via HEAD-probe of GovInfo content URLs \|
	-	\| `diff.py` \| Section-parse + align + diff engine; populates `bill_diffs` \|
	-	\| `rider_detect.py` \| R Street keyword scan over `bill_diffs.change='added'`; sets `rider_flag` \|
	-	\| `amendments.py` \| Parses `<amendments>` from BILLSTATUS XML; re-fetches for backfill \|
	-
	-	Schema additions (all `CREATE TABLE IF NOT EXISTS`):
	-	- `bills_ext` - extended bill metadata (bill_type, sponsor_name/party/state)
	-	- `bill_versions` - version rows with govinfo_package_id + text_ref (XML URL)
	-	- `bill_diffs` - per-section diff rows with change type + snippet + rider_flag
	-	- `amendments` - amendment rows with agreed_method + congress.gov source_url
	-
	-	---
	-
	-	## GovInfo Bulk Data Gotchas
	-
	-	### No ZIP files for BILLSTATUS
	-	The documented ZIP URL pattern (`BILLSTATUS-119hr.zip`) returns 404 for the 119th Congress. Individual XMLs are accessible; the correct enumeration method is the sitemap:
	-
	-	```
	-	https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/{congress}{type}/sitemap.xml
	-	```
	-
	-	The sitemap index is at:
	-	```
	-	https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/sitemapindex.xml
	-	```
	-
	-	Individual files:
	-	```
	-	https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/BILLSTATUS-{congress}{type}{number}.xml
	-	```
	-
	-	### Directory listing returns HTML error
	-	`GET /bulkdata/BILLSTATUS/119/hr/` returns a "Bulkdata Service Error" HTML page. Use the sitemap instead.
	-
	-	### Bill text (BILLS collection) - no key needed
	-	Full bill XML is accessible at:
	-	```
	-	https://www.govinfo.gov/content/pkg/BILLS-{congress}{type}{number}{version}/xml/BILLS-{congress}{type}{number}{version}.xml
	-	```
	-	This path does NOT require an API key. The `/api/` paths (JSON metadata) require a key.
	-
	-	### Sitemap entries are not sorted by bill number
	-	The sitemap contains entries in an arbitrary order. The first N entries from the sitemap are a random sample of bills, not the lowest-numbered ones. `--limit 200` will NOT get H.R. 1-200.
	-
	-	### BILLSTATUS XML root tag varies
	-	The root element is `<billStatus>` (lowercase) in modern files. The `<bill>` child is always there. Code handles both cases.
	-
	-	### Version date missing for `enr` (Enrolled Bill)
	-	The enrolled bill version often has no `version_date` in BILLSTATUS textVersions. The diff engine sorts by `(chain_rank, version_date)` so dateless entries fall into their correct chain position by rank. `enr` = rank 6 in the House chain.
	-
	-	### Section ID format changes between versions
	-	Different XML schemas are used for different version types:
	-	- Introduced (`ih`, `rh`) → long hex IDs: `H71241E5177324574B4C749197110EAB0`
	-	- Enrolled (`enr`) → short USLM IDs: `d244e72`
	-
	-	When IDs change between versions, the aligner falls back to fuzzy header matching (threshold 0.6). For short bills where all sections have the same content (e.g., Born-Alive Act H.R. 21 ih→eh), this produces 0 diff rows - correct, the text is identical.
	-
	-	For bills where the enrolled version switches from House HTML schema to USLM schema (common for enrolled bills), the aligner may incorrectly classify "removed" + "added" instead of "amended." This is a known GovInfo data quality issue. The enrolled→public-law pairs are most affected.
	-
	-	### Cross-chamber version codes (`pcs`, `rds`, `rih`, `ris`)
	-	House bills that go to the Senate acquire `pcs` (Placed on Calendar Senate) before becoming `enr`. Senate bills acquire `rds` (Received/Referred in Senate). These codes are in neither the standard House nor Senate chain; the diff engine assigns them rank 5 (after engrossed, before enrolled).
	-
	-	### Version code `rfs` (Referred in Senate)
	-	Appears for House bills passed and sent to Senate. Treated as rank 5 (cross-chamber). This is the same logical position as `pcs`.
	-
	-	### Public law version codes
	-	The BILLSTATUS XML sometimes includes a textVersion with type like `-119publ61` for Public Law 119-61. This is the enrolled public law text. Assigned rank 998 (near end of chain). Diffing `enr→-119publ61` usually shows section ID format changes only (not substantive differences).
	-
	-	---
	-
	-	## Real Data Results (as of 2026-06-26, 119th Congress sample)
	-
	-	Initial run (290 bills total):
	-
	-	\| Table \| Rows \|
	-	\|---\|---\|
	-	\| `bills` \| 290 \|
	-	\| `bill_versions` \| 410 \|
	-	\| `bill_diffs` \| 34 \|
	-	\| `amendments` \| 99 \|
	-	\| `actions` \| 1,695 \|
	-
	-	Bills with most versions:
	-	- H.J.Res. 25 (6 versions): Congressional disapproval of CFPB rule
	-	- H.R. 165 (6 versions): Wounded Knee Massacre Memorial Act
	-	- H.R. 131 (5 versions): Finish the Arkansas Valley Conduit Act
	-	- S.J.Res. 28/18/13/11 (5 versions each): Disapproval joint resolutions
	-
	-	Rider flags (3 flagged diff rows):
	-
	-	1. H.R. 43 (`rds→enr`) - Alaska Native Claims Settlement Act amendment:
	-	> "Section 14(c) of the Alaska Native Claims Settlement Act (43 U.S.C. 1613(c)) is amended..."
	-	Signal: `Is Amended`
	-
	-	2. S. 5 - Laken Riley Act (`es→enr`):
	-	> "Section 236(c) of the Immigration and Nationality Act (8 U.S.C. 1226(c)) is amended..."
	-	Signal: `Is Amended`
	-
	-	3. S. 5 - Laken Riley Act (`es→enr`, second section):
	-	> "Section 235(b) of the Immigration and Nationality Act (8 U.S.C. 1225(b)) is amended..."
	-	Signal: `Is Amended`
	-
	-	Real amended section example (H.R. 41, ih→rh):
	-	H.R. 41 - Unrecognized Southeast Alaska Native Communities Recognition and Compensation Act:
	-	Section H4DAD6E8E44444A988CA8A1864ADE20A6 was amended between the introduced version and the committee-reported version. The unified diff shows rewording of the Urban Corporation definition and conveyance authorization.
	-
	-	Real amended section example (S. 5, pcs→es):
	-	S. 5 - Laken Riley Act:
	-	Section H793DBCBA2A07463588AC4F4FDBEDECAF was amended between the placed-on-calendar and engrossed versions, showing Senate floor amendments to the detention mandate language (INA 236(c)).
	-
	-	---
	-
	-	## How to Run
	-
	-	```bash
	-	# Ingest all bill types for 119th Congress (runs from sitemap; ~9K HR + 4K S + …)
	-	python -m ingest.pollers.bills_bulk --once
	-
	-	# Test run (50 HR bills only)
	-	python -m ingest.pollers.bills_bulk --once --types hr --limit 50
	-
	-	# Discover additional text versions via HEAD-probe
	-	python -m ingest.pollers.bill_versions --once
	-
	-	# Diff all bills with 2+ versions
	-	python -m ingest.diff --once --all --congress 119
	-
	-	# Rider detection
	-	python -m ingest.rider_detect --once
	-
	-	# Amendments
	-	python -m ingest.amendments --once
	-
	-	# Run tests
	-	python -m pytest tests/test_bills_diff.py -v
	-	```
	-
	-	## Environment Variables
	-
	-	\| Variable \| Purpose \|
	-	\|---\|---\|
	-	\| `DOCKET_DB_PATH` \| SQLite path (default `/var/tmp/docket/docket.db`) \|
	-	\| `DOCKET_CONGRESS_KEY` \| api.data.gov key - unlocks Congress.gov API path in `amendments.py` \|
	-
	-	## Version Chain Order
	-
	-	House: `ih → rh → rh2 → eh → eah → eas → enr`
	-	Senate: `is → rs → pcs → es → eas → enr`
	-	Cross-chamber (rank 5): `pcs pch rds rih ris`
	-	Public law (rank 998): `-{congress}publ{n}`
	-
	-	## Amendment Data Coverage (honest gaps)
	-
	-	- Senate amendments 117th+ (2021+): machine-readable text available in GovInfo SAMDT collection. `amendments.py` stores congress.gov URLs; text can be fetched separately.
	-	- House amendments: text lives in Congressional Record (CREC), not discrete objects in BILLSTATUS. `text` column is NULL; source_url points to congress.gov amendment page.
	-	- Manager's / en-bloc packages: not public as discrete amendment objects anywhere. Only inferable from CREC narrative.
	-	- Voice vote / UC additions: `agreed_method` set to `'voice'` or `'uc'`; `text` is NULL; UI must show "no individual record exists."
	-	- Pre-117th amendment text: not machine-readable in bulk data; linked from congress.gov pages only.

services/ingest/CALENDAR_NOTES.md +0 -146

		@@ -1,146 +0,0 @@
	-	# Calendar Ingest - Notes
	-
	-	Module: `ingest/calendar.py`
	-	Table: `calendar_events`
	-	Tests: `tests/test_calendar.py` (42 tests, all pass)
	-
	-	## Run
	-
	-	```bash
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	-	python -m ingest.calendar --once
	-	```
	-
	-	## What it does
	-
	-	Builds `calendar_events` by aggregating dated events from every other ingest
	-	table. One run takes ~20 seconds (mostly FR API calls for effective_on
	-	enrichment). Idempotent - safe to re-run.
	-
	-	### Event kinds and sources
	-
	-	\| kind \| count (2026-06-26) \| source table \| key field \|
	-	\|-------------------\|--------------------\|----------------------\|----------------\|
	-	\| effective \| 738 \| executive_actions \| effective_on \|
	-	\| enacted \| 47 \| actions (BecameLaw) \| action_date \|
	-	\| vote \| 3,454 \| votes \| date \|
	-	\| introduced \| 24,794 \| actions (IntroReferral)\| action_date \|
	-	\| committee \| 5,205 \| actions (Committee) \| action_date \|
	-	\| floor \| 4,969 \| actions (Floor) \| action_date \|
	-	\| apportionment \| 20 \| apportionments \| date \|
	-	\| effective_parsed \| 0* \| actions (text regex) \| parsed date \|
	-	\| TOTAL \| 39,274 \| \| \|
	-
	-	*`effective_parsed` is 0 because action texts in this dataset are short
	-	procedural strings ("Became Public Law No: 119-X.") without specific effective
	-	date clauses. The machinery is in place; it will fire on richer bill text.
	-
	-	## The key feature: effective dates
	-
	-	The `effective` kind is the "goes into effect on X" source. Federal Register
	-	final rules carry a structured `effective_on` field in the FR API.
	-
	-	Real example (from production DB):
	-
	-	```
	-	Date: 2026-07-01
	-	Title: Medicare Program; Alternative Payment Model Updates and the
	-	Increasing Organ Transplant Access Act Implementation
	-	(Health and Human Services Department; Centers for Medicare and
	-	Medicaid Services)
	-	Source URL: https://www.federalregister.gov/documents/2026/.../medicare-program
	-	Confidence: 1.0
	-	```
	-
	-	Another:
	-	```
	-	Date: 2028-09-30
	-	Title: Removing Regulations Related to Trade Adjustment Assistance for
	-	Firms (Commerce Department; Economic Development Administration)
	-	Source URL: https://www.federalregister.gov/documents/2026/06/11/2026-11800/...
	-	```
	-
	-	## FR enrichment flow
	-
	-	1. `fetch_fr_rules_with_effective(conn, days_back=90)` - fetches 90 days of
	-	RULE-type documents with `effective_on` field included in the request;
	-	upserts into `executive_actions`.
	-	2. `enrich_fr_effective_on(conn)` - for any remaining `executive_actions` rows
	-	with `effective_on IS NULL`, batch-fetches from FR API in groups of 50.
	-	Notices/NPRMs/EOs typically return null (no effective date); that is correct.
	-
	-	## Schema
	-
	-	```sql
	-	CREATE TABLE calendar_events (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	date TEXT NOT NULL, -- ISO-8601 YYYY-MM-DD
	-	kind TEXT NOT NULL CHECK(kind IN (
	-	'effective','enacted','vote',
	-	'introduced','committee','floor',
	-	'apportionment','effective_parsed')),
	-	ref_type TEXT NOT NULL, -- e.g. executive_action, bill, vote
	-	ref_id INTEGER, -- FK into the source table
	-	title TEXT,
	-	jurisdiction TEXT NOT NULL DEFAULT 'US',
	-	source_url TEXT NOT NULL, -- allowlisted .gov URL
	-	confidence REAL NOT NULL DEFAULT 1.0, -- 0.5 for effective_parsed
	-	UNIQUE (kind, ref_type, ref_id, date)
	-	);
	-	CREATE INDEX idx_calendar_events_date ON calendar_events(date);
	-	CREATE INDEX idx_calendar_events_kind ON calendar_events(kind);
	-	```
	-
	-	## Effective-date regex patterns
	-
	-	Applied to bill/action text to extract explicit calendar dates.
	-	Only SPECIFIC dates extracted; relative clauses ("X days after enactment")
	-	are intentionally skipped. Confidence = 0.5 for these parsed results.
	-
	-	```python
	-	# Pattern 1: "shall take effect on <date>"
	-	r"shall\s+take\s+effect\s+on\s+<DATE>"
	-
	-	# Pattern 2: "takes effect on <date>"
	-	r"takes?\s+effect\s+on\s+<DATE>"
	-
	-	# Pattern 3: "this Act ... takes effect <date>"
	-	r"this\s+Act\b.*?\btakes?\s+effect\s+<DATE>"
	-
	-	# Pattern 4: "effective on <date>"
	-	r"effective\s+on\s+<DATE>"
	-
	-	# Pattern 5: "effective date is <date>" / "effective date shall be <date>"
	-	r"effective\s+date\s+(?:is\|shall\s+be)\s+<DATE>"
	-
	-	# Pattern 6 (lowest specificity): "effective <date>"
	-	r"\beffective\s+<DATE>"
	-
	-	# DATE fragment matches:
	-	# "January 1, 2026" (full month name)
	-	# "2026-01-01" (ISO-8601)
	-	```
	-
	-	## Future /api/calendar endpoint
	-
	-	Suggested query shape:
	-
	-	```sql
	-	SELECT date, kind, ref_type, ref_id, title, jurisdiction, source_url, confidence
	-	FROM calendar_events
	-	WHERE date BETWEEN :start AND :end
	-	AND (:kind IS NULL OR kind = :kind)
	-	ORDER BY date, kind
	-	LIMIT :limit OFFSET :offset
	-	```
	-
	-	This table is designed for date-range queries - the `date` index makes those
	-	fast even at 40k+ rows.
	-
	-	## Neutrality
	-
	-	Every row carries a `source_url` pointing to an official `.gov` endpoint.
	-	Events are factual dated occurrences only: no adjectives, no editorial
	-	weighting of which dates "matter." The `effective_parsed` kind has
	-	`confidence=0.5` to signal lower certainty and keep the higher-quality
	-	structured events clearly distinguished.

services/ingest/CATEGORIZE_NOTES.md +0 -149

		@@ -1,149 +0,0 @@
	-	# Categorize Module - Build Notes
	-
	-	## What was built
	-
	-	`ingest/categorize.py` - unified taxonomy backfill for bills and spending awards.
	-
	-	### Files created
	-	- `ingest/categorize.py` - the module
	-	- `tests/test_categorize.py` - 57 tests
	-
	-	### Schema changes (all additive)
	-	\| Change \| Table \| Type \|
	-	\|---\|---\|---\|
	-	\| CREATE TABLE IF NOT EXISTS \| `categories` \| slug, label, kind \|
	-	\| CREATE TABLE IF NOT EXISTS \| `bill_subjects` \| bill_id, subject \|
	-	\| ALTER TABLE ADD COLUMN \| `awards.category` \| TEXT (guarded by try/except) \|
	-
	-	---
	-
	-	## Data findings (live DB, 2026-06-26)
	-
	-	### Bills
	-	- Total bills: 7,190
	-	- Bills with `policy_area` (already stored by bills_bulk.py): 7,179 / 7,190 (99.8%)
	-	- Distinct policy areas in use: 31 of 32 canonical areas
	-	- 11 bills have no policy_area (XML had no `<policyArea>` element - not an error)
	-
	-	### Bill subjects (initial backfill of 50 bills)
	-	- 50 bills fetched → 279 subject rows, 170 distinct subjects
	-	- Full backfill running in background: ~7,140 remaining bills
	-	- Rate: ~50 bills / 22 seconds (~2.3 bills/sec with 0.05s delay)
	-	- At that rate, full backfill of all bills: ~50 minutes
	-
	-	### Awards
	-	- Total awards: 320
	-	- Awards categorized: 320 / 320 (100%)
	-
	-	Award category breakdown:
	-	\| Category \| Awards \|
	-	\|---\|---\|
	-	\| Health \| 111 \|
	-	\| Science, Technology, Communications \| 50 \|
	-	\| Government Operations and Politics \| 49 \|
	-	\| Agriculture and Food \| 19 \|
	-	\| Emergency Management \| 16 \|
	-	\| Transportation and Public Works \| 12 \|
	-	\| Armed Forces and National Security \| 12 \|
	-	\| International Affairs \| 11 \|
	-	\| Energy \| 10 \|
	-	\| Crime and Law Enforcement \| 10 \|
	-	\| Social Welfare \| 4 \|
	-	\| Environmental Protection \| 4 \|
	-	\| Public Lands and Natural Resources \| 3 \|
	-	\| Education \| 3 \|
	-	\| Commerce \| 3 \|
	-	\| Economics and Public Finance \| 2 \|
	-
	-	### Categories table
	-	- 32 rows seeded (canonical Congressional policy areas)
	-	- Also auto-ingests any novel policy_area values from the bills table on each run
	-
	-	---
	-
	-	## Agency → Category map
	-
	-	Covers ~50 agency entries. Three-tier lookup:
	-	1. Exact match
	-	2. Prefix match (handles sub-agencies, e.g. "Department of Defense - DARPA")
	-	3. Substring match (handles "HHS Office of Inspector General")
	-
	-	All map values align to the same 32-area vocabulary as bills.
	-
	-	Key mappings:
	-	\| Agency \| Category \|
	-	\|---\|---\|
	-	\| Department of Defense \| Armed Forces and National Security \|
	-	\| Department of Veterans Affairs \| Armed Forces and National Security \|
	-	\| Department of Homeland Security \| Emergency Management \|
	-	\| Department of Health and Human Services \| Health \|
	-	\| Department of Transportation \| Transportation and Public Works \|
	-	\| Department of Energy \| Energy \|
	-	\| Department of Education \| Education \|
	-	\| Department of Agriculture \| Agriculture and Food \|
	-	\| Department of Labor \| Labor and Employment \|
	-	\| Environmental Protection Agency \| Environmental Protection \|
	-	\| Department of Justice \| Crime and Law Enforcement \|
	-	\| Department of State \| International Affairs \|
	-	\| Agency for International Development \| International Affairs \|
	-	\| Department of Commerce \| Commerce \|
	-	\| Department of the Treasury \| Economics and Public Finance \|
	-	\| Department of the Interior \| Public Lands and Natural Resources \|
	-	\| Department of Housing and Urban Development \| Housing and Community Development \|
	-	\| Social Security Administration \| Social Welfare \|
	-	\| NASA \| Science, Technology, Communications \|
	-	\| National Science Foundation \| Science, Technology, Communications \|
	-	\| General Services Administration \| Government Operations and Politics \|
	-
	-	---
	-
	-	## Test results
	-
	-	57 tests, 0 failures, 0.47s.
	-
	-	Test classes:
	-	- `TestAgencyToCategory` - 25 tests: exact match, prefix match, unknown/null handling, all map values are valid labels
	-	- `TestPolicyAreas` - 6 tests: count, no duplicates, slug format, key areas present, slug/label consistency
	-	- `TestParseSubjects` - 7 tests: single/multiple subjects, empty, missing element, malformed XML, whitespace, alternate root element
	-	- `TestEnsureSchema` - 4 tests: creates both tables, adds column, idempotent
	-	- `TestSeedCategories` - 4 tests: seeds all areas, idempotent, picks up extra bill areas, Health present
	-	- `TestBackfillAwards` - 6 tests: known agency, unknown→NULL, no-overwrite, multiple, count, idempotent
	-	- `TestCategoryCounts` - 4 tests: returns list, expected keys, bill counted, award counted
	-
	-	---
	-
	-	## CLI
	-
	-	```bash
	-	# Full backfill (all bills + awards + categories)
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.categorize --once
	-
	-	# Limit subjects fetch (for testing)
	-	python -m ingest.categorize --once --limit-subjects 50
	-
	-	# Show category counts without re-running backfill
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	-	.venv/bin/python -m ingest.categorize --counts-only
	-	```
	-
	-	---
	-
	-	## Neutrality
	-
	-	- `policy_area` values come from Congress.gov's own taxonomy (Congressional Research Service) - no editorial classification
	-	- `legislativeSubjects` come from BILLSTATUS XML verbatim
	-	- Agency→category map keys are official agency names from USAspending data; values align to the same CRS taxonomy
	-	- No "good/bad" groupings; no partisan framing; categories are factual government classifications
	-
	-	---
	-
	-	## Technical notes
	-
	-	- GovInfo BILLSTATUS URL format: `/bulkdata/BILLSTATUS/{congress}/{type}/BILLSTATUS-{congress}{type}{num}.xml`
	-	(congress and type are separate path segments; concatenated only in the filename)
	-	- Bills with no subjects from GovInfo get a sentinel empty-string row in `bill_subjects` to prevent infinite re-fetching
	-	- Idempotent: re-running only processes bills/awards not yet categorized
	-	- `awards.category` ALTER is guarded with try/except so parallel runs don't fail
	-	- `categories` uses INSERT OR IGNORE so re-seeding is safe

services/ingest/CLOSEDDOOR_NOTES.md +0 -167

		@@ -1,167 +0,0 @@
	-	# Closed-Door + Executive Layer - Build Notes
	-	_Built 2026-06-26. All pollers verified live._
	-
	-	## What was built
	-
	-	Four new pollers in `services/ingest/ingest/pollers/`:
	-
	-	\| File \| Source \| Table \| Auth \|
	-	\|------\|--------\|-------\|------\|
	-	\| `federal_register.py` \| federalregister.gov/api/v1 \| `executive_actions` \| none \|
	-	\| `openomb.py` \| apportionment-public.max.gov \| `apportionments` \| none \|
	-	\| `crec.py` \| govinfo.gov/content/pkg/CREC-*/zip/ \| `procedural_events` \| none \|
	-	\| `lobbying.py` \| lda.gov/api/v1 \| `lobbying` \| env `DOCKET_LDA_KEY` \|
	-
	-	One new test file: `tests/test_crec_voice_vote.py` (62 tests).
	-
	-	All four pollers follow the existing pattern: `_ensure_tables()` creates new
	-	tables without touching `db.py`; `validate_source_url()` is called before every
	-	write; `is_new_event()` / `mark_seen()` enforce idempotency.
	-
	-	---
	-
	-	## Live run results (2026-06-26)
	-
	-	\| Table \| Rows \| Notes \|
	-	\|-------\|------\|-------\|
	-	\| `executive_actions` \| 500 \| 7 days of FR docs, all 4 types (PRESDOCU/RULE/PRORULE/NOTICE) \|
	-	\| `apportionments` \| 20 \| 20 records from FY2025 Corps of Engineers slice \|
	-	\| `procedural_events` \| 188 \| 5 session days from CREC-2026-06-16 through 06-23 \|
	-	\| `lobbying` \| 0 \| Graceful skip - `DOCKET_LDA_KEY` not set \|
	-
	-	---
	-
	-	## Sample: real Executive Order / rule row
	-
	-	```
	-	type: nprm
	-	title: Removing Obsolete References to Twentieth-Century Airman Certificates
	-	agency: Transportation Department; Federal Aviation Administration
	-	date: 2026-06-26
	-	source_url: https://www.federalregister.gov/documents/2026/06/26/2026-13003/...
	-	```
	-
	-	---
	-
	-	## Sample: real apportionment showing appropriated vs apportioned
	-
	-	```
	-	agency: Corps of Engineers--Civil Works
	-	account: Construction
	-	appropriated: $39,088,945,000
	-	apportioned: $39,088,945,000
	-	status: released
	-	date: 2024-09-06
	-	source_url: https://apportionment-public.max.gov/Fiscal%20Year%202025/Corps%20
	-	of%20Engineers--Civil%20Works/JSON/FY2025_Agency%3DCOE_...
	-	```
	-
	-	Status logic: apportioned/appropriated ≥ 95% → released; > 0 but < 95% → delayed; 0 → withheld.
	-
	-	The FY2025 Corps of Engineers records all show "released" (normal operating year).
	-	The impound-tracker signal emerges when `status = withheld` (OMB released $0 against
	-	an appropriation) - those rows are the headline feature for Docket's "buried data" pillar.
	-
	-	---
	-
	-	## Sample: real detected voice vote / procedural event
	-
	-	```
	-	kind: cloture
	-	date: 2026-06-23
	-	chamber: Both
	-	detail: Cloture action: …provisions of Rule XXII of the Standing Rules of the
	-	Senate, a vote on cloture will oc…
	-	members_jsonb: NULL ← no individual record exists (see honest gap below)
	-	source_url: https://www.govinfo.gov/app/details/CREC-2026-06-23
	-	```
	-
	-	CREC-2026-06-23 yielded: 80 events (cloture × 13, unanimous_consent × 63, suspension × 4).
	-	No voice-vote matches on that day - the Senate was on cloture votes, not voice-vote-only measures.
	-
	-	---
	-
	-	## Test results
	-
	-	```
	-	191 passed in 20.80s
	-	```
	-
	-	- `test_crec_voice_vote.py` - 62 tests: voice-vote regex, UC, cloture, suspension, MTR,
	-	discharge, XML/HTML extractor, chamber inference, snippet extraction, `_detect_events()`
	-	end-to-end, source-URL .gov enforcement, idempotency.
	-
	-	---
	-
	-	## Technical gotchas discovered during build
	-
	-	### Federal Register API - array param encoding
	-	Must use a list of `(key, value)` tuples (not a dict) for `fields[]` and
	-	`conditions[type][]`. The field `presidential_document_type` does not exist
	-	in the API - causes HTTP 400. Use `abstract` instead.
	-
	-	### GovInfo CREC - correct URL
	-	Bulk data path (`/bulkdata/CREC/`) returns HTML error pages, not ZIPs.
	-	The working URL is:
	-	```
	-	https://www.govinfo.gov/content/pkg/CREC-YYYY-MM-DD/zip/CREC-YYYY-MM-DD.zip
	-	```
	-	Non-session days (weekends, recesses) return HTML (not 404) - "bad ZIP" errors
	-	are expected and handled gracefully. Default look-back is 14 days to cover 5+
	-	session days.
	-
	-	### CREC ZIP contents - HTML, not XML
	-	The ZIP contains `.htm` files (not standalone XML). Text extraction strips HTML
	-	tags with regex + `html.unescape()`. The `_extract_text_from_xml()` function
	-	handles both correctly.
	-
	-	### max.gov apportionments - no REST API
	-	`apportionment-public.max.gov` has no JSON API endpoint. The site serves
	-	individual JSON files per apportionment record (one per TAFS account + FY +
	-	iteration), listed in an HTML index page. The poller:
	-	1. Fetches the HTML index (~500KB, loads in ~20s)
	-	2. Extracts all JSON file hrefs matching the target FY via regex
	-	3. Downloads individual records (capped at 200/run, 50 for `--once`)
	-
	-	FY2025 has 6,172 JSON links; FY2026 will grow through the year. A scheduled
	-	hourly/daily run can work through the full set over time.
	-
	-	openomb.org (the aggregator) returns HTTP 403 for automated requests - use
	-	max.gov directly, which is also the correct source_url target per the
	-	neutrality rule.
	-
	-	### Neutrality rule - source_url enforcement
	-	- `federal_register.py`: html_url from FR API → federalregister.gov (.gov ✓)
	-	- `openomb.py`: direct max.gov JSON file URL (.max.gov = .gov ✓)
	-	- `crec.py`: govinfo.gov app/details page (.gov ✓)
	-	- `lobbying.py`: lda.gov/filings/{uuid}/ (.gov ✓)
	-
	-	All validated by `source_validator.validate_source_url()` before every DB write.
	-
	-	---
	-
	-	## How to run
	-
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.pollers.federal_register --once
	-
	-	python -m ingest.pollers.openomb --once --max 50
	-	python -m ingest.pollers.crec --once --days 14
	-	DOCKET_LDA_KEY=<your_key> python -m ingest.pollers.lobbying --once
	-	```
	-
	-	## LDA key
	-
	-	Register free at https://lda.gov/api/. Without a key the lobbying poller logs
	-	a clear message and returns 0 - no crash, no partial state.
	-
	-	## Recommended poll cadences
	-
	-	\| Poller \| Cadence \| Notes \|
	-	\|--------\|---------\|-------\|
	-	\| federal_register \| Every 4h \| Same-day FR lag \|
	-	\| openomb \| Daily \| max.gov updates when OMB signs a new apportionment \|
	-	\| crec \| Daily (noon ET) \| ~1-day CREC publishing lag \|
	-	\| lobbying \| Weekly \| LDA filings are quarterly \|

services/ingest/INGEST_NOTES.md +0 -144

		@@ -1,144 +0,0 @@
	-	# Docket Ingest Service - Notes
	-
	-	## What Was Built
	-
	-	Python ingest spine for the Docket government-activity tracker (Phase 1).
	-	All code lives at `services/ingest/` in the repo.
	-
	-	### Module layout
	-
	-	```
	-	ingest/
	-	├── __init__.py
	-	├── run.py # CLI entrypoint (python -m ingest.run --once)
	-	├── db.py # SQLite schema + connection (Postgres target in prod)
	-	├── change_detection.py # Dedupe on (source, externalId, updateDate)
	-	├── ranking.py # Feed ranking - only 4 permitted fields (rule #3)
	-	├── source_validator.py # .gov allowlist enforcement (rule #1)
	-	└── pollers/
	-	├── usaspending.py # FULLY LIVE - no auth, pulls real awards
	-	├── congress.py # Skeleton - needs DOCKET_CONGRESS_KEY
	-	├── govinfo.py # Skeleton - needs DOCKET_CONGRESS_KEY (same key)
	-	├── treasury.py # Skeleton - no auth (endpoints need verification)
	-	├── votes_house.py # Live - House Clerk XML, no auth
	-	├── votes_senate.py # Live - Senate roll-call XML, no auth
	-	└── openstates.py # Skeleton - needs DOCKET_OPENSTATES_KEY (paid tier)
	-	tests/
	-	├── test_ranking.py # 16 tests - neutrality rule #3 enforcement
	-	└── test_source_url.py # 19 tests - neutrality rule #1 enforcement
	-	```
	-
	-	## How to Run
	-
	-	### Setup (one-time)
	-
	-	```bash
	-	cd services/ingest
	-	python3 -m venv .venv
	-	.venv/bin/pip install -r requirements.txt
	-	```
	-
	-	### Run all pollers once
	-
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.run --once
	-	```
	-
	-	### Run one poller
	-
	-	```bash
	-	python -m ingest.run --source usaspending
	-	python -m ingest.run --source votes_senate
	-	```
	-
	-	### Run tests
	-
	-	```bash
	-	cd services/ingest
	-	python -m pytest tests/ -v
	-	```
	-
	-	## Test Results (verified 2026-06-26)
	-
	-	```
	-	35 passed in 4.09s
	-	```
	-
	-	- `test_ranking.py` (16 tests): all ranking-function neutrality assertions pass,
	-	including AST inspection that confirms no forbidden fields (social/engagement/news)
	-	appear in any ranking helper.
	-	- `test_source_url.py` (19 tests): all source-URL allowlist assertions pass,
	-	including integration tests writing to a temp SQLite DB.
	-
	-	## Live Ingest Run Results (2026-06-26)
	-
	-	```
	-	usaspending → 320 new award rows
	-	votes_senate → 17 new Senate vote rows
	-	congress → 0 (no API key)
	-	govinfo → 0 (no API key)
	-	treasury → 0 (endpoint paths need verification)
	-	votes_house → 0 (no votes found for 2026 - correct, House not in session)
	-	openstates → 0 (no API key)
	-	```
	-
	-	### Sample award row (largest by amount)
	-
	-	```
	-	source: usaspending
	-	external_id: CONT_AWD_DEAC3608GO28308_8900_-NONE-_-NONE-
	-	agency: Department of Energy
	-	recipient: ALLIANCE FOR ENERGY INNOVATION, LLC
	-	amount: $10,788,357,674.94
	-	action_date: (null - this award type reports null action_date from USAspending)
	-	program: other
	-	source_url: https://www.usaspending.gov/award/CONT_AWD_DEAC3608GO28308_8900_-NONE-_-NONE-
	-	```
	-
	-	SQLite DB location on CT241: `/var/tmp/docket/docket.db`
	-	Override with: `export DOCKET_DB_PATH=/path/to/docket.db`
	-
	-	## API Keys Required
	-
	-	### Congress.gov + GovInfo (same key)
	-	- Env var: `DOCKET_CONGRESS_KEY`
	-	- Get it free: https://api.data.gov/signup
	-	- Rate limit: 5,000 req/hr (Congress.gov) / 36,000 req/hr (GovInfo)
	-	- Without this key the congress and govinfo pollers skip gracefully - no crash.
	-
	-	### OpenStates (state bills)
	-	- Env var: `DOCKET_OPENSTATES_KEY`
	-	- Free tier: 500 req/day - too tight for production.
	-	- Required tier: Bronze (5k/day) or Silver (50k/day) - contact OpenStates for pricing.
	-	- Without this key the openstates poller skips gracefully - no crash.
	-
	-	## Known Issues / Follow-ups
	-
	-	1. Treasury endpoints return 404 - the fiscaldata.treasury.gov API paths may
	-	have changed. Needs a quick endpoint audit against their API docs before the
	-	Treasury poller will be useful.
	-
	-	2. USAspending `loans` group returns 400 - the loan award codes (07, 08) seem
	-	to require a different query structure. The other 4 groups work correctly.
	-
	-	3. `Action Date` is null for many USAspending awards - the `/search/spending_by_award/`
	-	endpoint returns null for Action Date on many records. For completeness, use the
	-	`/bulk_download/` endpoint or per-award detail endpoint to get richer metadata.
	-
	-	4. SQLite must be on local storage - Network filesystems (CIFS/NFS) do not
	-	support SQLite's file-locking mechanism. Keep the DB on local disk.
	-	Set `DOCKET_DB_PATH` to a local path. For production scale, migrate to Postgres.
	-
	-	5. Venv must be on local storage - same filesystem requirement; the venv
	-	should live on the deployment host's local disk.
	-
	-	## Neutrality Enforcement (architectural summary)
	-
	-	- Rule #1 (gov-source-only): `source_validator.validate_source_url()` is called
	-	before every DB write. Raises `SourceURLError` on any non-`.gov` host. Enforced
	-	by 19 unit tests.
	-	- Rule #3 (action-only ranking): `ranking.score()` accepts exactly 4 parameters
	-	(`actionDate`, `dollarAmount`, `affectedEstimate`, `actionType`). AST inspection
	-	in 5 unit tests confirms no engagement/social/news field appears in any helper.

services/ingest/MEETINGS_NOTES.md +0 -164

		@@ -1,164 +0,0 @@
	-	# Meetings Ingest - Notes
	-
	-	Module: `ingest/pollers/meetings.py`
	-	Table: `meetings`
	-	Tests: `tests/test_meetings.py` (58 tests, all pass)
	-
	-	## Run
	-
	-	```bash
	-	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	-	python -m ingest.pollers.meetings --once
	-	```
	-
	-	Options:
	-	- `--days-back N` - days of past House committee meetings to scan (default 30)
	-	- `--days-ahead N` - days of future House committee meetings to scan (default 90)
	-	- `--no-detail` - skip per-event detail fetches (faster; meeting type omitted for new events)
	-
	-	## What it does
	-
	-	Ingests congressional committee + floor activity from three keyless sources.
	-
	-	### Sources and counts (2026-06-26, 30-back / 90-ahead window)
	-
	-	\| Source \| Chamber \| Kind \| Count \|
	-	\|---\|---\|---\|---\|
	-	\| senate.gov XML \| Senate \| hearings \| 1 \|
	-	\| senate.gov XML \| Senate \| business_meeting \| 1 \|
	-	\| docs.house.gov \| House \| hearing \| 68 \|
	-	\| docs.house.gov \| House \| markup \| 19 \|
	-	\| docs.house.gov \| House \| floor \| 66 \|
	-	\| docs.house.gov \| House \| other \| 11 \|
	-	\| TOTAL \| \| \| 166 \|
	-
	-	### Source 1 - Senate committee meetings XML
	-
	-	URL: `https://www.senate.gov/general/committee_schedules/hearings.xml`
	-	Status: Working. Returns clean structured XML with all upcoming meetings.
	-	Update cadence: Regenerated multiple times daily.
	-	Fields extracted: identifier, committee, subcommittee, date, time, location (room), subject (matter), type.
	-	Source URL stored: `https://www.senate.gov/committees/hearings_meetings.htm`
	-
	-	The XML provides a `last_update_iso_8601` field used for change-detection in the `seen` table. Meetings are only re-fetched when they change.
	-
	-	Note: On days with no Senate committee meetings, the XML returns a single placeholder entry with no `<identifier>` element. This is correctly skipped.
	-
	-	### Source 2 - House committee meetings (HTML)
	-
	-	Calendar day index: `https://docs.house.gov/Committee/Calendar/ByDay.aspx?DayID=MMDDYYYY`
	-	Event detail: `https://docs.house.gov/Committee/Calendar/ByEvent.aspx?EventID=N`
	-	Status: Working. Data-rich HTML tables.
	-
	-	Two-phase approach:
	-	1. Phase 1 (parallel): Fetch all `ByDay.aspx` pages for the date window using `ThreadPoolExecutor` (8 workers). Extracts `EventID`, title, committee, time, location, status (Rescheduled/Canceled/Postponed).
	-	2. Phase 2 (parallel): For new `EventID`s only, fetch the event detail page (4 workers) to extract meeting type (`Hearing:` / `Markup:` / `Business Meeting:` prefix in `<h1>`).
	-
	-	Change-detection key: `status\|date` - re-processes a meeting if its status or scheduled date changes.
	-
	-	Source URL per row: `https://docs.house.gov/Committee/Calendar/ByEvent.aspx?EventID={N}` (docs.house.gov is .gov ✓)
	-
	-	Noted: Individual meeting XML download on ByEvent.aspx requires JavaScript postback (`__doPostBack`) - inaccessible without a browser. The HTML scrape provides all fields needed.
	-
	-	### Source 3 - House floor schedule XML
	-
	-	URL pattern: `https://docs.house.gov/billsthisweek/YYYYMMDD/YYYYMMDD.xml` (Monday date)
	-	Status: Working. Scans current week + 4 prior weeks (weeks with no XML return 404 and are skipped).
	-	Fields extracted: bill number (legis-num), short description (floor-text), floor category (suspension/rule/etc.), week date.
	-	Type: always `floor`
	-	Source URL per row: `https://docs.house.gov/floor/Default.aspx?date=YYYY-MM-DD`
	-
	-	Not a source: Congress.gov committee schedule - key-gated. The `docs.house.gov` committee repository is the keyless equivalent.
	-	Not a source: Federal Register Sunshine Act notices - FR poller (`executive_actions`) owns those; committee meetings here focus on congressional activity only.
	-
	-	## Schema
	-
	-	```sql
	-	CREATE TABLE meetings (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	source TEXT NOT NULL,
	-	external_id TEXT NOT NULL,
	-	chamber TEXT NOT NULL CHECK(chamber IN ('Senate','House')),
	-	committee TEXT,
	-	date TEXT NOT NULL, -- YYYY-MM-DD
	-	time TEXT, -- as-published (e.g. "10:00 AM")
	-	datetime_iso TEXT, -- YYYY-MM-DDTHH:MM:SS
	-	title TEXT,
	-	subject TEXT,
	-	type TEXT CHECK(type IN (
	-	'hearing','markup','business_meeting','floor','other')),
	-	location TEXT,
	-	status TEXT NOT NULL DEFAULT 'scheduled'
	-	CHECK(status IN ('scheduled','held','canceled','postponed')),
	-	source_url TEXT NOT NULL,
	-	UNIQUE (source, external_id)
	-	);
	-	CREATE INDEX idx_meetings_date ON meetings(date);
	-	CREATE INDEX idx_meetings_committee ON meetings(committee);
	-	```
	-
	-	## 3 Real example meetings (from production run 2026-06-26)
	-
	-	Example 1 - Senate committee (upcoming)
	-	```
	-	Chamber: Senate
	-	Committee: Personnel Subcommittee (Armed Services)
	-	Date/Time: 2026-07-15 02:30 PM
	-	Type: hearing
	-	Title: Hearings to examine the TRICARE pharmacy program.
	-	Location: SR-222
	-	Source: https://www.senate.gov/committees/hearings_meetings.htm
	-	```
	-
	-	Example 2 - House committee hearing (upcoming)
	-	```
	-	Chamber: House
	-	Committee: Europe Subcommittee (Committee on Foreign Affairs)
	-	Date/Time: 2026-06-30 10:00 AM
	-	Type: hearing
	-	Title: Arms Control and Nonproliferation in an Era of Great Power Competition
	-	Location: 2172 RHOB
	-	Source: https://docs.house.gov/Committee/Calendar/ByEvent.aspx?EventID=119256
	-	```
	-
	-	Example 3 - House committee hearing (past)
	-	```
	-	Chamber: House
	-	Committee: Select Committee on the Strategic Competition Between the United States and the Chinese Communist Party
	-	Date/Time: 2026-06-25 10:30 AM
	-	Type: hearing
	-	Title: China's Economic Espionage and Subnational Influence in the...
	-	Location: (varies)
	-	Source: https://docs.house.gov/Committee/Calendar/ByEvent.aspx?EventID=...
	-	```
	-
	-	## Sources that are blocked / unavailable (keyless)
	-
	-	\| Source \| Status \| Reason \|
	-	\|---\|---\|---\|
	-	\| Congress.gov committee schedule API \| Key-gated \| Requires `DOCKET_CONGRESS_KEY` \|
	-	\| docs.house.gov Meeting XML download \| JS-only \| `__doPostBack` - no direct URL \|
	-	\| Individual Senate committee pages \| No structured feed \| Per-committee HTML only \|
	-	\| CHRG (hearing transcripts) \| Key-gated (GovInfo API) \| Available in bulk with key \|
	-
	-	## Future /api/schedule endpoint
	-
	-	Suggested query pattern for the frontend:
	-
	-	```sql
	-	SELECT date, chamber, committee, time, datetime_iso, title, type, location, status, source_url
	-	FROM meetings
	-	WHERE date BETWEEN :start AND :end
	-	AND (:chamber IS NULL OR chamber = :chamber)
	-	AND (:committee IS NULL OR committee LIKE :committee)
	-	AND (:type IS NULL OR type = :type)
	-	ORDER BY datetime_iso NULLS LAST, date, committee
	-	LIMIT :limit OFFSET :offset
	-	```
	-
	-	This supports the target UX: "Mon 2026-08-25, 4:55 PM - 4 committee meetings on X, Y, Z."
	-	The `date` index makes date-range queries fast. The `committee` index supports committee-specific filtering.
	-
	-	## Neutrality
	-
	-	All rows carry factual scheduled activity data only: committee name, date, time, subject. No adjectives, no editorial characterization of hearing importance or significance. Source URLs point directly to official `.gov` pages. Meeting subjects are reproduced verbatim from official sources.

services/ingest/STATE_NOTES.md +0 -209

		@@ -1,209 +0,0 @@
	-	# State-Level Ingest - Notes
	-
	-	## What Was Built
	-
	-	State-level granular coverage with honest confidence scoring for all 52 US
	-	jurisdictions (50 states + DC + PR).
	-
	-	### Files created / modified
	-
	-	\| File \| Change \|
	-	\|------\|--------\|
	-	\| `ingest/source_validator.py` \| Added `STATE_LEG_HOSTS` + `_is_state_leg_pattern()` + updated `is_allowlisted()` \|
	-	\| `ingest/pollers/openstates.py` \| Full rewrite - was a stub; now fetches bills + votes + sponsors for all 52 jurisdictions \|
	-	\| `ingest/pollers/legiscan.py` \| New - LegiScan API poller (completeness supplement) \|
	-	\| `ingest/state_coverage.py` \| New - honesty layer; computes + writes state_coverage rows \|
	-	\| `tests/test_state_coverage.py` \| New - 55 tests for coverage logic + state-host allowlist \|
	-
	-	---
	-
	-	## Running
	-
	-	### Compute state coverage (works without any API keys)
	-
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/path/to/docket.db
	-	python -m ingest.state_coverage
	-
	-	# JSON output
	-	python -m ingest.state_coverage --json
	-	```
	-
	-	Expected output (no keys, empty state DB):
	-	```
	-	Jurisdictions tracked : 52 (50 states + DC + PR)
	-	Total coverage rows : 312
	-	ok (fresh) : 0
	-	stale (>48h) : 0
	-	missing (never seen) : 52
	-	```
	-
	-	### Run OpenStates poller (requires key)
	-
	-	```bash
	-	export DOCKET_OPENSTATES_KEY=<your_key>
	-	# MVP subset (12 jurisdictions - conserves 500/day free quota)
	-	python -m ingest.run --source openstates
	-
	-	# All 52 jurisdictions (needs paid tier)
	-	DOCKET_OPENSTATES_ALL=1 python -m ingest.run --source openstates
	-	```
	-
	-	### Run LegiScan poller (requires key)
	-
	-	```bash
	-	export DOCKET_LEGISCAN_KEY=<your_key>
	-	# MVP subset (12 states - ~200 API calls)
	-	python -m ingest.run --source legiscan
	-
	-	# All 52 jurisdictions (needs ~2k API calls - monitor monthly quota)
	-	DOCKET_LEGISCAN_ALL=1 python -m ingest.run --source legiscan
	-	```
	-
	-	### Run tests
	-
	-	```bash
	-	cd services/ingest
	-	python -m pytest tests/ -v
	-	# Expected: 190 passed
	-	```
	-
	-	---
	-
	-	## Getting API Keys (free)
	-
	-	### OpenStates
	-	- URL: https://openstates.org/accounts/login/
	-	- Free tier: 500 requests/day, 10 requests/second
	-	- Env var: `DOCKET_OPENSTATES_KEY`
	-	- Caveat: 500 req/day is too tight for all 52 jurisdictions. MVP polls 12
	-	jurisdictions (the 10 most-populated states + DC + PR) to stay within quota.
	-	Production requires Bronze (5k/day) or Silver (50k/day) - contact
	-	openstates@openstates.org for pricing.
	-
	-	### LegiScan
	-	- URL: https://legiscan.com/legiscan
	-	- Free tier: ~30,000 requests/month (~1,000/day)
	-	- Env var: `DOCKET_LEGISCAN_KEY`
	-	- Efficiency: The `getMasterListRaw` sync strategy minimises API calls -
	-	only bills whose `change_hash` changed are fetched via `getBill`. Cap:
	-	`MAX_BILL_FETCH_PER_STATE=50` per state per run (configurable in code).
	-
	-	---
	-
	-	## Source URL Allowlist Extensions
	-
	-	State legislatures that don't use a `.gov` TLD require explicit allowlisting.
	-	The following were added to `source_validator.py`:
	-
	-	### STATE_LEG_HOSTS (explicit non-.gov hosts)
	-
	-	\| State \| Domain \|
	-	\|-------\|--------\|
	-	\| Pennsylvania \| `www.legis.state.pa.us` \|
	-	\| Texas \| `capitol.state.tx.us`, `legis.state.tx.us` \|
	-	\| Minnesota \| `www.leg.state.mn.us`, `www.house.leg.state.mn.us`, `www.senate.leg.state.mn.us` \|
	-	\| Mississippi \| `billstatus.ls.state.ms.us`, `ls.state.ms.us` \|
	-	\| Nevada \| `www.leg.state.nv.us` \|
	-	\| New Hampshire \| `www.gencourt.state.nh.us` \|
	-	\| New Jersey \| `www.njleg.state.nj.us` \|
	-	\| Alabama \| `alison.legislature.state.al.us` \|
	-	\| Alaska \| `www.legis.state.ak.us` \|
	-	\| Arkansas \| `www.arkleg.state.ar.us` \|
	-	\| Kansas \| `www.kslegislature.org` ← unique .org \|
	-	\| Puerto Rico \| `www.oslpr.org`, `www.camaraderepresentantes.org`, `www.senadopr.us` \|
	-	\| DC Council \| `dccouncil.us` \|
	-
	-	### Pattern rule: `*.state.<XX>.us`
	-
	-	Any host matching `<anything>.state.<2-letter-code>.us` is automatically
	-	allowlisted via `_is_state_leg_pattern()`. This covers:
	-	- `legis.state.pa.us` (Pennsylvania)
	-	- `njleg.state.nj.us` (New Jersey)
	-	- `billstatus.ls.state.ms.us` (Mississippi)
	-	- Any new state portal added at a `*.state.XX.us` subdomain without code change
	-
	-	This pattern does NOT match `govtrack.us`, `example.us`, or similar non-government `.us` domains.
	-
	-	---
	-
	-	## Architecture
	-
	-	### state_coverage table
	-
	-	```sql
	-	CREATE TABLE IF NOT EXISTS state_coverage (
	-	state TEXT NOT NULL,
	-	source TEXT NOT NULL, -- 'openstates' or 'legiscan'
	-	data_type TEXT NOT NULL, -- 'bills', 'votes', 'sponsors'
	-	last_scrape TEXT, -- ISO UTC timestamp of last successful scrape
	-	confidence TEXT NOT NULL DEFAULT 'missing', -- 'ok' \| 'stale' \| 'missing'
	-	PRIMARY KEY (state, source, data_type)
	-	);
	-	```
	-
	-	### scrape_log table
	-
	-	```sql
	-	CREATE TABLE IF NOT EXISTS scrape_log (
	-	source TEXT NOT NULL,
	-	state TEXT NOT NULL,
	-	data_type TEXT NOT NULL,
	-	scraped_at TEXT NOT NULL, -- ISO UTC timestamp
	-	row_count INTEGER NOT NULL DEFAULT 0,
	-	PRIMARY KEY (source, state, data_type)
	-	);
	-	```
	-
	-	### Confidence logic
	-
	-	- `ok`: `scraped_at` within the last 48 hours
	-	- `stale`: `scraped_at` exists but older than 48 hours
	-	- `missing`: no scrape record for this (state, source, data_type) combination
	-
	-	### Poller → coverage flow
	-
	-	```
	-	openstates.poll() / legiscan.poll()
	-	└─ for each jurisdiction:
	-	├─ fetch + store bills + votes
	-	└─ log_scrape(conn, source, state, 'bills', count)
	-	log_scrape(conn, source, state, 'votes', count)
	-	log_scrape(conn, source, state, 'sponsors', count)
	-
	-	state_coverage.compute()
	-	├─ reads scrape_log
	-	├─ fallback: infers from raw_events (OCD/legiscan external_id parsing)
	-	└─ writes 312 rows to state_coverage (52 × 2 × 3)
	-	```
	-
	-	### API surface (GET /api/states)
	-
	-	The `state_coverage` table directly backs the `/api/states` endpoint:
	-	```json
	-	{
	-	"code": "CA",
	-	"name": "California",
	-	"confidence": "ok",
	-	"lastScrape": "2026-06-26T12:00:00Z",
	-	"missingTypes": []
	-	}
	-	```
	-
	-	---
	-
	-	## Known Gaps (surface in-product - never hide)
	-
	-	1. Free quota insufficient for prod: OpenStates 500 req/day covers ~12 states/day;
	-	LegiScan 30k/month covers all states if running daily (tight).
	-	2. Historical coverage: pollers default to `days_back=1`. First run on a state
	-	won't have historical bill history - only recent changes.
	-	3. Worst-lagging states: MA, NE, KY, AL historically 24-48h behind in both
	-	OpenStates and LegiScan (known in DATA_SOURCES.md).
	-	4. Ballot measures / interim committees: poorly covered by both APIs.
	-	5. Amendment text: LegiScan has better amendment completeness than OpenStates,
	-	but pre-2021 coverage is spotty in both.
	-	6. Vote positions (`vote_positions` table): not yet populated - yea/nay counts
	-	only. Individual member positions require `getVote` (LegiScan) per vote - costs
	-	extra API calls. Phase 2 work.

services/ingest/VOTES_NOTES.md +0 -152

		@@ -1,155 +0,0 @@
	-	# Docket - Votes & Members Build Notes
	-
	-	Verified 2026-06-26 (venv `/opt/docket/services/ingest/.venv`, DB `/opt/docket/data/docket.db`).
	-
	-	## Files Created / Modified
	-
	-	\| File \| Status \| Purpose \|
	-	\|---\|---\|---\|
	-	\| `ingest/pollers/members.py` \| new \| Full current member roster (537 members) \|
	-	\| `ingest/pollers/votes_house.py` \| rewritten \| House XML with per-member positions \|
	-	\| `ingest/pollers/votes_enrich.py` \| new \| Senate positions backfill from existing vote source_urls \|
	-	\| `tests/test_votes_house_parser.py` \| new \| 39 tests: XML parser, position enum, schema, both pivots \|
	-
	-	## Schema additions (additive - all CREATE TABLE IF NOT EXISTS / ALTER TABLE)
	-
	-	```sql
	-	CREATE TABLE IF NOT EXISTS members (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	bioguide_id TEXT NOT NULL UNIQUE,
	-	full_name TEXT NOT NULL,
	-	party TEXT, state TEXT, district INTEGER, chamber TEXT,
	-	lis_member_id TEXT, govtrack_id INTEGER, icpsr_id INTEGER,
	-	source_url TEXT NOT NULL -- https://bioguide.congress.gov/search/bio/{id}
	-	);
	-	CREATE INDEX IF NOT EXISTS idx_members_lis ON members(lis_member_id);
	-
	-	CREATE TABLE IF NOT EXISTS vote_positions (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	vote_id INTEGER NOT NULL REFERENCES votes(id),
	-	member_bioguide TEXT,
	-	member_id INTEGER REFERENCES members(id),
	-	position TEXT NOT NULL CHECK(position IN ('Yea','Nay','Present','Not Voting'))
	-	);
	-	CREATE INDEX IF NOT EXISTS idx_vote_positions_vote_id ON vote_positions(vote_id);
	-	CREATE INDEX IF NOT EXISTS idx_vote_positions_member_id ON vote_positions(member_id);
	-	CREATE INDEX IF NOT EXISTS idx_vote_positions_bioguide ON vote_positions(member_bioguide);
	-
	-	ALTER TABLE votes ADD COLUMN congress INTEGER;
	-	ALTER TABLE votes ADD COLUMN session INTEGER;
	-	ALTER TABLE votes ADD COLUMN roll_number INTEGER;
	-	ALTER TABLE votes ADD COLUMN question TEXT;
	-	ALTER TABLE votes ADD COLUMN not_voting INTEGER;
	-	ALTER TABLE votes ADD COLUMN recorded INTEGER DEFAULT 1;
	-	ALTER TABLE votes ADD COLUMN vote_external_id TEXT;
	-	ALTER TABLE votes ADD COLUMN bill_text_ref TEXT;
	-	```
	-
	-	## Data ingested (2026-06-26 baseline)
	-
	-	\| Source \| Count \|
	-	\|---\|---\|
	-	\| Members \| 537 (437 House, 100 Senate) \|
	-	\| House votes (2026, rolls 1-62) \| 50 \|
	-	\| Senate votes (119/1, votes 1-40) \| 40 \|
	-	\| vote_positions total \| 25,572 \|
	-	\| vote_positions (House) \| 21,581 \|
	-	\| vote_positions (Senate) \| 3,991 \|
	-
	-	## Running the pollers
	-
	-	```bash
	-	# Members roster (run first - needed for member_id foreign keys)
	-	python -m ingest.pollers.members --once
	-
	-	# House votes (probes current year sequentially, stops at 404)
	-	python -m ingest.run --source votes_house
	-
	-	# Senate votes (existing poller, no modification)
	-	python -m ingest.run --source votes_senate
	-
	-	# Senate positions backfill (run after members + senate votes)
	-	python -m ingest.pollers.votes_enrich --once
	-	```
	-
	-	## Worked example - both pivot directions
	-
	-	### Pivot 1: "Who voted how on House Roll #62?"
	-
	-	```
	-	Roll #62 (2026-02-11) - On Agreeing to the Resolution, as Amended
	-	H RES 1057 \| Passed \| Y:216 N:215 P:0 NV:1
	-	432 positions recorded
	-	Source: https://clerk.house.gov/evs/2026/roll062.xml
	-
	-	Sample:
	-	Shomari Figures (D-AL): Nay
	-	Terri A. Sewell (D-AL): Nay
	-	Nicholas J. Begich III (R-AK): Yea
	-	Robert B. Aderholt (R-AL): Yea
	-	```
	-
	-	Query:
	-	```sql
	-	SELECT vp.position, m.full_name, m.party, m.state
	-	FROM vote_positions vp
	-	LEFT JOIN members m ON m.bioguide_id = vp.member_bioguide
	-	WHERE vp.vote_id = (SELECT id FROM votes WHERE roll_number=62 AND chamber='house')
	-	ORDER BY m.state, m.full_name;
	-	```
	-
	-	### Pivot 2: "How did Rep. Aderholt vote recently?"
	-
	-	```
	-	Robert B. Aderholt (R-AL) \| bioguide: A000055
	-	Source: https://bioguide.congress.gov/search/bio/A000055
	-	50 recorded votes in DB
	-
	-	Roll #062 \| 2026-02-11 \| On Agreeing to the Resolution, as Amended -> Yea
	-	Roll #061 \| 2026-02-11 \| On Ordering the Previous Question -> Yea
	-	Roll #059 \| 2026-02-10 \| On Ordering the Previous Question -> Yea
	-	Roll #054 \| 2026-02-04 \| On Motion to Recommit -> Nay
	-	```
	-
	-	Query:
	-	```sql
	-	SELECT v.roll_number, v.date, v.question, vp.position
	-	FROM vote_positions vp
	-	JOIN votes v ON v.id = vp.vote_id
	-	WHERE vp.member_bioguide = 'A000055'
	-	ORDER BY v.date DESC, v.roll_number DESC;
	-	```
	-
	-	## Key design decisions
	-
	-	Members source: `unitedstates/congress-legislators` `gh-pages` branch JSON (not `main` - YAML only). Fetch URL is `raw.githubusercontent.com`; stored `source_url` is always `bioguide.congress.gov/search/bio/{id}` (allowlisted .gov).
	-
	-	House XML date format: The Clerk XML uses `"D-Mon-YYYY"` (e.g. `"8-Jan-2026"`), not `MM/DD/YYYY` as some docs suggest. The parser tries `%d-%b-%Y` first, then fallbacks.
	-
	-	House Clerk URL format: `https://clerk.house.gov/evs/{year}/roll{NNN}.xml` (lowercase `roll`, no underscore, zero-padded 3 digits). The prior stub incorrectly used `ROLL_{NNN}.xml`.
	-
	-	Bioguide join: House XML `<legislator name-id="...">` carries the bioguide directly. Senate XML uses `lis_member_id`; the enrich module maps it via `members.lis_member_id`.
	-
	-	Unmatched members: Delegates (DC, territories, etc.) appear in House roll-call XMLs but not in `legislators-current.json`. Their positions are stored with `member_bioguide` populated and `member_id = NULL` - data preserved, nothing fabricated.
	-
	-	Honesty on voice/UC votes: `parse_house_vote_xml()` returns `None` for non-recorded vote types (`VOICE VOTE`, etc.). Those votes are not in this table. The closed-door agent (CREC) owns them.
	-
	-	Senate backfill approach: `votes_enrich.py` re-fetches the `source_url` already stored on existing `votes` rows - no external mapping needed. Idempotent: only processes rows with no existing positions.
	-
	-	## Tests (39 total, 230 total suite)
	-
	-	```
	-	tests/test_votes_house_parser.py - 39 tests
	-
	-	TestNormalizePosition (6) - position enum mapping incl. Aye/No variants
	-	TestParseLegisNum (9) - bill type normalization, all 8 prefixes
	-	TestParseHouseVoteXml (9) - fields, counts, bioguides, voice-vote honesty
	-	TestParseSenateVotePositions (9) - LIS IDs, all 4 positions, malformed XML
	-	TestSchemaAndInsertion (6) - table creation, column presence, enum enforcement,
	-	both pivot directions via JOIN
	-	```
	-
	-	All 230 tests pass (suite includes pre-existing ranking/source_url/state/diff/crec tests).

services/ingest/VOTEVIEW_NOTES.md +0 -220

		@@ -1,220 +0,0 @@
	-	# Voteview Ingest - Implementation Notes
	-
	-	## What this poller does
	-
	-	`pollers/voteview.py` ingests complete per-member voting records from the
	-	Voteview bulk-CSV dataset (Lewis et al., voteview.com). It populates:
	-
	-	- `votes` - one row per recorded roll call, with metadata, year index, and
	-	an official .gov `source_url` (neutrality rule #1)
	-	- `vote_positions` - one row per member per roll call, with canonical position
	-	- `voteview_crosswalk` - cached ICPSR → bioguide mapping
	-
	-	Voteview is the ingestion pipe only. No Voteview URLs appear anywhere in
	-	user-facing data; every `source_url` points to the official government record.
	-
	-	---
	-
	-	## Cast-code mapping
	-
	-	Voteview assigns each member vote a `cast_code` (0-9) per their codebook
	-	(https://voteview.com/articles/data_help_members#castcode):
	-
	-	\| cast_code \| Official meaning \| Canonical position \|
	-	\|-----------\|------------------------------\|--------------------\|
	-	\| 0 \| Not a member / no record \| Not Voting \|
	-	\| 1 \| Yea \| Yea \|
	-	\| 2 \| Paired Yea (absent, declared)\| Yea \|
	-	\| 3 \| Announced Yea (absent, decl.)\| Yea \|
	-	\| 4 \| Announced Nay (absent, decl.)\| Nay \|
	-	\| 5 \| Paired Nay (absent, declared)\| Nay \|
	-	\| 6 \| Nay \| Nay \|
	-	\| 7 \| Present / abstention (v1) \| Present \|
	-	\| 8 \| Present / abstention (v2) \| Present \|
	-	\| 9 \| Not Voting \| Not Voting \|
	-
	-	Paired / announced votes (codes 2-5): The member was physically absent but
	-	announced their intent; the chamber counts them as Yea/Nay. We map them to
	-	Yea/Nay for the same reason - they reflect a declared position.
	-
	-	Codes 7 and 8 are both Present (abstention), not Nay. Some secondary
	-	sources conflate 7/8 with Nay; this is incorrect per the Voteview codebook.
	-
	-	Code 0 vs 9:
	-	- 0 = not a member at the time of the vote (e.g., vacancy) or genuinely missing
	-	- 9 = member was present in the chamber but chose not to vote
	-	Both map to `Not Voting`; the distinction matters for DW-NOMINATE but not for
	-	our per-member position index.
	-
	-	All 10 codes and every enum value are covered by `tests/test_voteview.py`.
	-
	-	---
	-
	-	## ICPSR → bioguide crosswalk
	-
	-	Voteview identifies members by ICPSR code (numeric, chamber-specific: a member
	-	who served in both chambers gets two ICPSR IDs). The Docket DB uses bioguide_id
	-	as the join key.
	-
	-	The crosswalk is built from:
	-	- `legislators-current.json` (unitedstates/congress-legislators, gh-pages branch)
	-	- `legislators-historical.json` (same)
	-
	-	Both are fetched once and cached in the `voteview_crosswalk` table. On
	-	subsequent runs, the cache is used (pass `--refresh-crosswalk` to re-download).
	-
	-	The `id.icpsr` field is the primary mapping. Some members also have
	-	`id.icpsr_hn` (a separate House ICPSR number); both are indexed.
	-
	-	Members not in the crosswalk: stored with `member_bioguide = NULL`.
	-	This happens for:
	-	- Members who served before the digitisation cutoff in congress-legislators
	-	- Delegates, non-voting members, territorial representatives
	-	- A small number of historical mismatches
	-
	-	---
	-
	-	## Official source URLs (neutrality rule #1)
	-
	-	Voteview is an academic aggregator - it is never the user-facing source.
	-
	-	\| Chamber \| URL pattern \|
	-	\|---------\|-------------\|
	-	\| House \| `https://clerk.house.gov/Votes/{year}/roll{NNN:03d}` \|
	-	\| Senate \| `https://www.senate.gov/legislative/LIS/roll_call_votes/vote{congress}{session}/vote_{congress}_{session}_{roll:05d}.xml` \|
	-
	-	For House votes: `clerk_rollnumber` (from the Voteview rollcalls CSV) is used
	-	as `NNN`, because it is the House Clerk's sequential roll number. Voteview's
	-	own `rollnumber` is stored separately in `voteview_rollnumber` for reference.
	-
	-	For Senate votes: Voteview's `rollnumber` equals the Senate's official roll
	-	call number, so it is used directly.
	-
	-	---
	-
	-	## Year index
	-
	-	A `year INTEGER` column is added to the `votes` table and indexed as
	-	`idx_votes_year`. This makes the following queries run in O(log n):
	-
	-	```python
	-	# All votes for member X in year Y (hits idx_votes_year + idx_vote_positions_bioguide)
	-	votes_by_member_year("N000002", 2023)
	-
	-	# All recorded votes in year Y (hits idx_votes_year)
	-	votes_in_year(2023)
	-	```
	-
	-	`year` is derived from the `date` field (first 4 characters) and stored
	-	redundantly to avoid a `substr()` or `CAST` on every range scan.
	-
	-	---
	-
	-	## Completeness
	-
	-	What Voteview covers: Only recorded (roll-call) votes where a member's
	-	name appears in the official tally. This is the complete universe of votes
	-	where per-member records exist in the public record.
	-
	-	What Voteview does NOT cover:
	-	- Voice votes - passage is recorded but no per-member positions exist. This is
	-	~94% of Senate measures and a large fraction of House suspension-of-the-rules
	-	votes. The product note "no individual record exists" must be displayed for
	-	these (handled by `votes_house.py` + `votes_senate.py` which set
	-	`recorded=0`).
	-	- Unanimous consent agreements
	-	- Committee votes
	-
	-	Row-count check: the completeness report (`--report` flag) fetches the
	-	Voteview rollcalls CSV for each (congress, chamber) and compares its row count
	-	against the `seen` table. A match means every rollcall in the CSV was
	-	processed. Gaps are explained (e.g., blank rollnumber rows in the CSV, or an
	-	incomplete run).
	-
	-	---
	-
	-	## Running the ingest
	-
	-	### Proof slice (congresses 118-119, 2023-present)
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.pollers.voteview --once --start 118 --end 119
	-	```
	-
	-	### Full 2019-present (congresses 116-119)
	-	```bash
	-	python -m ingest.pollers.voteview --once --start 116 --end 119
	-	```
	-
	-	### Full historical backfill (1989-present, ~30 congresses, recommended first pass)
	-	```bash
	-	python -m ingest.pollers.voteview --once --start 101 --end 119
	-	```
	-
	-	### Full Voteview archive (1789-present, all ~120 congresses)
	-	```bash
	-	python -m ingest.pollers.voteview --once --start 1 --end 119
	-	```
	-
	-	Add `--report` to any command to print the CSV vs DB completeness table.
	-	Add `--refresh-crosswalk` to re-download the ICPSR crosswalk from GitHub.
	-
	-	---
	-
	-	## Scale estimate (full historical backfill)
	-
	-	Voteview covers all US congresses from 1789. Rough row counts for a full
	-	`--start 1 --end 119` run:
	-
	-	\| Scope \| Congresses \| Approx rollcalls \| Approx positions \|
	-	\|-----------------\|------------\|------------------\|-----------------\|
	-	\| 2019-present \| 116-119 \| ~6,000 \| ~2,000,000 \|
	-	\| 1989-present \| 101-119 \| ~25,000 \| ~8,000,000 \|
	-	\| Full archive \| 1-119 \| ~70,000 \| ~20,000,000 \|
	-
	-	Full archive runtime estimate: 4-6 hours (dominated by CSV downloads and
	-	batch DB inserts). The ingest is fully resumable - the `seen` table prevents
	-	re-processing any rollcall already ingested.
	-
	-	SQLite comfortably handles 20M rows in `vote_positions` with proper indexes;
	-	the `year` and `bioguide` indexes keep member-year queries under 5ms.
	-
	-	---
	-
	-	## Year-index example query
	-
	-	```python
	-	from ingest.pollers.voteview import votes_by_member_year
	-
	-	# Jerry Nadler (D-NY, bioguide N000002) votes in 2023
	-	rows = votes_by_member_year("N000002", 2023)
	-	for r in rows[:5]:
	-	print(r["date"], r["chamber"], r["roll_number"], r["question"],
	-	r["result"], "→", r["position"])
	-	```
	-
	-	Example output (after ingest of congress 118):
	-	```
	-	2023-01-09 house 5 On Ordering the Previous Question Agreed to → Nay
	-	2023-01-09 house 6 On Adopting the Rules of the 118th Congress Agreed to → Nay
	-	2023-01-10 house 10 On Passage Passed → Nay
	-	...
	-	```
	-
	-	To get all votes in a specific year regardless of member:
	-	```python
	-	from ingest.pollers.voteview import votes_in_year
	-	all_2023 = votes_in_year(2023)
	-	print(f"{len(all_2023)} recorded votes in 2023")
	-	```
	-
	-	---
	-
	-	## Files
	-
	-	\| File \| Purpose \|
	-	\|------\|---------\|
	-	\| `ingest/pollers/voteview.py` \| Main poller (this module) \|
	-	\| `tests/test_voteview.py` \| 48 pytest tests (cast_code, crosswalk, schema, year-index, URL validation) \|
	-	\| `VOTEVIEW_NOTES.md` \| This document \|

services/ingest/WATCHER_NOTES.md +0 -182

		@@ -1,182 +0,0 @@
	-	# Docket Watcher - Notes
	-
	-	## What it does
	-
	-	`ingest/watcher.py` wraps the existing pipeline (`ingest.run`) and records
	-	what changed since the previous cycle: items ADDED (new external_id seen for the
	-	first time), UPDATED (same item, newer update date), or REMOVED (a full-set source
	-	stopped returning an item that was previously seen).
	-
	-	Results are stored in `change_events` and are the foundation of Docket's
	-	"what just changed" feed and future alerts.
	-
	-	## Run modes
	-
	-	```bash
	-	# Single cycle - designed for systemd timer (or manual testing)
	-	python -m ingest.watcher --once
	-
	-	# Continuous loop with 30-minute pause
	-	python -m ingest.watcher --loop --interval 1800
	-
	-	# Filter to a single source (fast; good for testing without full pipeline)
	-	python -m ingest.watcher --once --sources federal_register
	-
	-	# Multiple sources
	-	python -m ingest.watcher --once --sources members openomb
	-
	-	# Override DB path
	-	python -m ingest.watcher --once --db /tmp/test.db
	-	```
	-
	-	## Watcher-owned tables
	-
	-	All three tables are created by the watcher (`_ensure_watcher_tables`); they
	-	are additive and never touch existing schema.
	-
	-	\| Table \| Purpose \|
	-	\|---\|---\|
	-	\| `change_events` \| Feed of ADDED / UPDATED / REMOVED events (the primary output) \|
	-	\| `poller_watermarks` \| Last-run timestamp + cycle ID per source (for operational dashboards) \|
	-	\| `watcher_manifest` \| Current-set snapshot for full-set sources (used for removal diffing) \|
	-
	-	### change_events schema
	-
	-	```sql
	-	CREATE TABLE change_events (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	ts TEXT NOT NULL, -- ISO 8601 UTC timestamp of this cycle
	-	source TEXT NOT NULL, -- pipeline step label (e.g. federal_register)
	-	ref_type TEXT NOT NULL, -- normalized type (bill/award/member/…)
	-	ref_id TEXT NOT NULL, -- external_id from the source
	-	change TEXT NOT NULL CHECK(change IN ('added','updated','removed')),
	-	summary TEXT, -- human-readable one-liner
	-	source_url TEXT NOT NULL -- official .gov URL (neutrality rule #1)
	-	);
	-	```
	-
	-	## ADDED / UPDATED detection
	-
	-	The watcher snapshots the `seen` table (which tracks `(source, external_id,
	-	update_date)` for every item processed by `mark_seen`) before and after the
	-	pipeline run:
	-
	-	- ADDED - new `(source, external_id)` appeared in the post-run snapshot.
	-	- UPDATED - same `(source, external_id)`, but `update_date` changed.
	-
	-	This covers every poller that calls `mark_seen()` (federal_register, openomb,
	-	usaspending, treasury, congress, govinfo, bills_bulk, bill_versions, votes_senate,
	-	votes_house, crec, lobbying, openstates, legiscan).
	-
	-	The `members` poller does a blind upsert without `mark_seen`, so its ADDED events
	-	come from the manifest path instead (see below).
	-
	-	## REMOVED detection - full-set sources only
	-
	-	Why only full-set? Paginated or time-windowed sources (federal_register fetches
	-	last-7-days, usaspending returns pages of awards, etc.) cannot distinguish "item
	-	not in this page" from "item genuinely removed". Flagging removal there would
	-	produce a constant stream of false positives.
	-
	-	Full-set sources (configured in `FULL_SET_CONFIGS`):
	-
	-	\| Source \| Data table \| Rationale \|
	-	\|---\|---\|---\|
	-	\| `openomb` \| `apportionments` \| OMB index returns all active FY apportionment files \|
	-	\| `members` \| `members` \| `legislators-current.json` is the authoritative live roster \|
	-
	-	The watcher maintains a `watcher_manifest` table: after each successful cycle it
	-	records every `(source, external_id)` pair that was present in the data table.
	-	The next cycle diffs the new set against this manifest.
	-
	-	### Safety guards
	-
	-	Two guards prevent false REMOVED events:
	-
	-	1. Non-empty guard - if `_query_current_set()` returns an empty dict for a
	-	full-set source, the watcher writes a WARNING log and skips removal detection
	-	for that source. This handles: network failures, table not yet created, source
	-	returning unexpectedly empty HTML, etc. The existing manifest is preserved
	-	unchanged.
	-
	-	2. First-run guard - if there is no prior manifest entry for a source (first
	-	ever cycle), no items are flagged as removed. Without a baseline, there is
	-	nothing to diff against.
	-
	-	Together these ensure: a failed pull or a fresh deployment can never generate a
	-	flood of spurious REMOVED events.
	-
	-	## Watermarks
	-
	-	`poller_watermarks` records `(source, last_run_at, cycle_id)` after every cycle.
	-	Use it to tell an external dashboard "openomb was last checked at T".
	-	The `cycle_id` is a monotonically increasing integer that ties `change_events`
	-	and `watcher_manifest` entries to the same logical run.
	-
	-	## systemd install
	-
	-	```bash
	-	# 1. Copy unit files
	-	cp services/ingest/deploy/docket-watcher.service \
	-	/etc/systemd/system/
	-	cp services/ingest/deploy/docket-watcher.timer \
	-	/etc/systemd/system/
	-
	-	# 2. Reload and enable
	-	systemctl daemon-reload
	-	systemctl enable --now docket-watcher.timer
	-
	-	# 3. Verify
	-	systemctl status docket-watcher.timer
	-	journalctl -u docket-watcher -f
	-
	-	# 4. Trigger a manual cycle immediately (for testing)
	-	systemctl start docket-watcher.service
	-	```
	-
	-	To check the timer schedule:
	-
	-	```bash
	-	systemctl list-timers docket-watcher.timer
	-	```
	-
	-	To query recent change events from the DB:
	-
	-	```bash
	-	sqlite3 /opt/docket/data/docket.db \
	-	"SELECT ts, source, ref_type, change, ref_id FROM change_events ORDER BY id DESC LIMIT 20;"
	-	```
	-
	-	## Testing a single cycle (without full pipeline)
	-
	-	```bash
	-	cd services/ingest
	-	export DOCKET_DB_PATH=/opt/docket/data/docket.db
	-	python -m ingest.watcher --once --sources federal_register
	-	```
	-
	-	## Running the unit tests
	-
	-	```bash
	-	cd services/ingest
	-	python -m pytest tests/test_watcher.py -v
	-	```
	-
	-	## Adding a new full-set source
	-
	-	1. Confirm the source's poll() always returns the complete current live set
	-	(not a time window, not paginated results, not a delta feed).
	-	2. Add an entry to `FULL_SET_CONFIGS` in `ingest/watcher.py`:
	-	```python
	-	"source_label": {
	-	"table": "my_table",
	-	"id_col": "external_id", # or the PK column name
	-	"where": "source = 'source_label'", # or None if table is single-source
	-	"url_col": "source_url",
	-	"ref_type": "my_type",
	-	},
	-	```
	-	3. If the source does not call `mark_seen()`, ensure it is NOT excluded from
	-	the ADDED-via-manifest path (remove it from the exclusion list in
	-	`_detect_full_set_changes`).
	-	4. Add a test in `tests/test_watcher.py`.

services/ingest/ingest/r2.py +1 -1

		@@ -1,7 +1,7 @@
		"""R2 archive uploader (Cloudflare). Outbound-only from the lab; no S3 creds.

		Uses the CF REST object API with the global key (from env CF_ACCOUNT_ID/CF_EMAIL/
	-	CF_GLOBAL_KEY - source /shared/projects/.env.secrets before running). Bucket
	+	CF_GLOBAL_KEY - set these in your environment before running). Bucket
		`docket-archive` is public at https://archive.usdocket.org. Uploads are the
		preservation layer: snapshot official .gov records so they survive when the
		government reorganizes/removes them.