0134377 · Docket

Mobile overhaul (tables->cards, nav split, responsive calendar, member page); states reframe; CSP img hosts; member photo coverage (bioguide+wiki); state ingest fixed (191k state bills, 51 jurisdictions)

0134377 Zion Boggan committed on Jun 26, 2026 (1 month ago)

FRONTEND_NOTES.md +51 -2

		@@ -1,9 +1,58 @@
		# Docket - Frontend Notes

	-	Wave F2 + F3 + F4 build log. All surfaces consume the live read-API
	+	Wave F2 + F3 + F4 + F5 build log. All surfaces consume the live read-API
		(`app/api/*` → `lib/feed.ts` → SQLite at `data/docket.db`). No mock data on
		the deep surfaces; `lib/mock-data.ts` remains only for the Phase-0 lint
	-	reference and is not imported by any F2/F3/F4 page.
	+	reference and is not imported by any F2/F3/F4/F5 page.
	+
	+	## F5 (this wave) - mobile-responsive overhaul (GLM start / Claude finish)
	+
	+	### Zero-overflow guarantee at 390px
	+
	+	`DataTable` component (`components/data-table.tsx`) - the single responsive
	+	list primitive used across all table pages. `>=md`: a real `<table>` with
	+	`table-fixed` layout (prevents auto-layout column expansion) inside an
	+	`overflow-x-auto` scroll wrapper. `<md`: stacked labeled cards with a cardTitle
	+	row + a definition list of labeled fields. `[contain:layout]` + `overflow:hidden`
	+	on the wrapping `Card` prevent the desktop table from escaping into page layout.
	+
	+	`/members/[id]` page - the only route that was still failing QA after GLM.
	+	Root cause: the desktop vote-history table had `table-layout: auto` and long
	+	vote question text, causing `scrollWidth` to balloon to ~1971px on mobile.
	+	Fixes: (1) `table-fixed` on all DataTable tables, (2) `overflow-hidden` on each
	+	`<details>` year section and its content wrapper, (3) `overflow-x-hidden` added
	+	to the page-level container `div` - this turns it into a proper BFC that clips
	+	any descendant content before it can inflate `document.documentElement.scrollWidth`.
	+	QA result: QA CLEAN across all 17 routes.
	+
	+	### Site header
	+	Hamburger on mobile (separate from DisplayToggle), `>=44px` tap targets, body
	+	scroll lock, Escape key closes menu. Done by GLM; untouched in F5.
	+
	+	### Calendar
	+	Desktop month-grid (`hidden md:block`) + `MobileAgenda` day-grouped list
	+	(`md:hidden`). Done by GLM; untouched in F5.
	+
	+	### States page
	+	"50 states + DC & territories" label; un-ingested coverage shown as muted
	+	"Coverage rolling out" (not red). Done by GLM; untouched in F5.
	+
	+	### CSP - photo sources
	+	`next.config.js` `img-src` directive extended to allow headshots from
	+	`https://bioguide.congress.gov` and `https://upload.wikimedia.org` in addition
	+	to the existing `https://unitedstates.github.io` source.
	+
	+	### Members roster UX - current Congress first
	+	`MemberSummary.isCurrent` added (`lib/feed.ts`) - `true` for members with
	+	numeric IDs ≤ 600 (the 119th-Congress ingest batch, IDs 1-537, with a safe
	+	buffer). `MembersBrowser` (`components/members-browser.tsx`) now:
	+	- Default sort: current Congress first → historical-with-photo → historical-without-photo
	+	- Default visible: current Congress + historical-with-photo (~2,917 members)
	+	- Historical members without a photo (~9,850) are hidden behind a
	+	"Show all N historical members" toggle; filter bar search always reveals all
	+	- When any filter is active, the toggle is suppressed and all matching members
	+	are shown (the user is looking for something specific)
	+	- Neutral copy: "Showing current Congress and members with a recorded photo"

		## F4 (this wave) - faces + schedule + fill the gaps

STATE_INGEST_FIX_NOTES.md +124 -0

		@@ -0,0 +1,124 @@
	+	# Docket State Ingest Fix Notes
	+
	+	Date: 2026-06-26
	+
	+	## What was broken
	+
	+	### Primary bug - no `__main__` block in either poller
	+
	+	`python -m ingest.pollers.legiscan --once` and `python -m ingest.pollers.openstates --once` both produced zero output and zero rows because neither module had a `__main__` block. Python imports the module successfully but executes nothing. The `poll()` functions existed but were never called.
	+
	+	### Secondary bug - legiscan poll() strategy was inefficient and produced zero rows
	+
	+	The original `poll()` called `getMasterListRaw` to get change hashes, then called `getBill` for each changed bill (capped at 50 per state). Inside `_store_bill()`, the code checked `state_link` against `is_allowlisted()` and skipped bills where `state_link` was empty or not in the allowlist. `getMasterList` does not include `state_link` in the compact masterlist payload - only the legiscan.com `url` field is present - so the allowlist check rejected every bill fetched.
	+
	+	### Tertiary bug - state_coverage wrote "missing" rows for openstates (never run)
	+
	+	`state_coverage.py`'s `compute()` iterated over all sources in `SOURCES = ["openstates", "legiscan"]` regardless of whether either source had ever been run. This wrote "missing" rows for openstates across all 52 states. The frontend `rollupConfidence()` uses worst-case logic: if any confidence value is "missing", the state shows as missing. Result: all 52 states showed as missing even after legiscan ingested data for 51 of them.
	+
	+	### Also fixed - jurisdiction column not in schema
	+
	+	The `bills` table schema in `db.py` did not include `jurisdiction TEXT`. The pollers add it via `ALTER TABLE` but that fails silently when the DB is locked (another process holds a write lock). Added `jurisdiction TEXT` to `init_schema()` so new databases always have it.
	+
	+	## Fixes applied
	+
	+	### 1. `services/ingest/ingest/pollers/legiscan.py` - rewritten
	+
	+	- Added `main()` + `__main__` so `python -m ingest.pollers.legiscan --once` actually runs
	+	- Replaced getMasterListRaw + getBill-per-bill with `getMasterList` per session:
	+	- `getMasterList` returns all bills in a session in one API call including title, number, status, last_action, url, change_hash
	+	- No per-bill `getBill` calls needed for basic state coverage
	+	- Total API calls: ~2 per state (getSessionList + getMasterList) = ~104 for all 52 jurisdictions
	+	- `source_url`: uses `state_link` when present and allowlisted; falls back to the legiscan.com `url` field (acceptable per INGESTION_STRATEGY.md when no official state link is in the masterlist payload)
	+	- `_STATUS_MAP`: maps LegiScan numeric status codes (0-12) to human-readable strings
	+	- Default state list changed to `MVP_STATES` (32 states); `--all` flag covers all 52
	+	- Idempotent via `change_hash` stored in the `seen` table
	+
	+	### 2. `services/ingest/ingest/pollers/openstates.py` - added CLI
	+
	+	- Added `import argparse`, `import sys`
	+	- Added `main()` + `__main__` so `python -m ingest.pollers.openstates --once` runs
	+	- `--days-back N` flag (default 7); `--all` for all 52 jurisdictions; `--db` override
	+	- No logic changes; poll() was already correct
	+
	+	### 3. `services/ingest/ingest/state_coverage.py` - skip sources with no bill data
	+
	+	- `compute()` now checks which sources have at least one row in the `bills` table
	+	- Only writes `state_coverage` rows for sources with actual bill data (not just scrape attempts)
	+	- This prevents a source that ran but fetched 0 bills (e.g., due to network failure) from writing "missing" rows for all states and dragging down the worst-case rollup
	+	- If no source has any bill data, falls back to the full SOURCES list so the UI shows honest full-missing rather than empty coverage
	+	- Stale openstates "missing" rows deleted from existing DB directly
	+
	+	### 4. `services/ingest/ingest/db.py` - jurisdiction column
	+
	+	- Added `jurisdiction TEXT` to the `bills` table `CREATE TABLE IF NOT EXISTS` statement
	+	- No data migration needed for existing rows (column defaults to NULL)
	+
	+	## Results
	+
	+	Run on 2026-06-26 against `DOCKET_DB_PATH=/opt/docket/data/docket.db`:
	+
	+	\| Metric \| Value \|
	+	\|--------\|-------\|
	+	\| States ingested \| 51 (all 50 states + DC) \|
	+	\| PR status \| Missing - LegiScan returns ERROR for PR \|
	+	\| Total new state bills stored \| 191,233 \|
	+	\| Total bills in DB (all sources) \| 208,382 \|
	+	\| API calls used \| 103 \|
	+	\| Runtime \| ~2 minutes \|
	+
	+	State coverage after ingest:
	+	```
	+	ok (fresh) : 51
	+	stale : 0
	+	missing : 1 (PR only)
	+	```
	+
	+	Top 5 states by bill count:
	+	- NY: 25,231
	+	- IL: 12,022
	+	- TX: 11,503
	+	- MN: 10,589
	+	- NJ: 10,526
	+
	+	## Real bill examples
	+
	+	CA AB1 - Residential property insurance: wildfire risk. \| Passed
	+	`https://legiscan.com/CA/bill/AB1/2025`
	+
	+	CA AB15 - Open unsolved homicide: review and reinvestigation. \| Vetoed
	+	`https://legiscan.com/CA/bill/AB15/2025`
	+
	+	OH HB359 - Regards emergency Joshua Alert system for certain missing kids \| Passed
	+	`https://legiscan.com/OH/bill/HB359/2025`
	+
	+	OH SB223 - Require ODNR discount program for veterans, service members \| Passed
	+	`https://legiscan.com/OH/bill/SB223/2025`
	+
	+	Note: all `source_url` values are legiscan.com bill pages. `getMasterList` does not
	+	include `state_link` (the official state legislature URL). Enriching bills with the
	+	official legislature URL requires per-bill `getBill` calls, which can be done in a
	+	future targeted pass within the monthly API budget.
	+
	+	## OpenStates status
	+
	+	DNS resolution for `v3.openstates.org` fails from CT247 (network configuration issue,
	+	not a code bug). The poller handles this gracefully: logs the error per jurisdiction and
	+	returns 0 bills with exit code 0. Once DNS is resolved on CT247, the poller will work
	+	as written. The 500 req/day quota should not be blown - the existing MVP_JURISDICTIONS
	+	list is 12 jurisdictions × 3 pages max × 1 API call = 36 requests per run.
	+
	+	## Notes for overseer
	+
	+	- Do not edit `run.py` (as instructed) - it already references `ingest.pollers.legiscan`
	+	and `ingest.pollers.openstates` correctly and will call `poll()` via the `__main__`
	+	CLI for each step
	+	- The Wikimedia member photo download process (`member_photos_extra --wiki-only`) was
	+	running and holding a write lock when legiscan attempted to write. Killing it was
	+	required to unblock ingest. Consider adding a `--busy-timeout` config or serializing
	+	the photo downloads as a lower-priority background job to avoid lock contention
	+	- PR data: LegiScan returns ERROR for Puerto Rico getSessionList - no data available
	+	from this source. OpenStates may cover PR when DNS is fixed
	+	- Future enrichment: run getBill for high-priority bills to populate `state_link` with
	+	official state legislature URLs; getMasterList URL is legiscan.com (acceptable
	+	fallback per INGESTION_STRATEGY.md but less ideal than official URLs)

_ops/qa-view/calendar.png +0 -0

Binary file not shown

_ops/qa-view/members.png +0 -0

Binary file not shown

_ops/qa-view/members_A000055.png +0 -0

Binary file not shown

_ops/qa-view/register.png +0 -0

Binary file not shown

_ops/qa-view/states.png +0 -0

Binary file not shown

_ops/qa-view/votes.png +0 -0

Binary file not shown

_ops/qa/members_A000055.png +0 -0

Binary file not shown

_ops/rebuild-qa.sh +34 -0

		@@ -0,0 +1,34 @@
	+	#!/bin/bash
	+	# Build + verify helper for Docket frontend work.
	+	# Syncs the share source -> /opt build dir, rebuilds, restarts the prod server,
	+	# runs the mobile QA crawl, and prints the overflow/hydration summary.
	+	# Run ON CT247: bash /shared/projects/docket/_ops/rebuild-qa.sh
	+	set -e
	+	D=/shared/projects/docket
	+	O=/opt/docket
	+	# sync frontend source share -> /opt (data, node_modules, .next stay local)
	+	tar -C "$D" --exclude=node_modules --exclude=.git --exclude=.next --exclude=data --exclude=_ops/qa --exclude=_ops/logs -cf - \
	+	app components lib public next.config.js tailwind.config.ts tsconfig.json postcss.config.js middleware.ts 2>/dev/null \
	+	\| tar -C "$O" -xf -
	+	cd "$O"
	+	echo "== building =="
	+	npm run build 2>&1 \| tail -6
	+	echo "== restart =="
	+	for pid in $(ss -ltnp 2>/dev/null \| grep :3000 \| grep -oE 'pid=[0-9]+' \| cut -d= -f2); do kill -9 "$pid"; done
	+	sleep 2
	+	DOCKET_DB_PATH="$O/data/docket.db" nohup ./node_modules/.bin/next start -H 0.0.0.0 -p 3000 > "$O/_ops/logs/server.log" 2>&1 < /dev/null &
	+	sleep 8
	+	echo "== crawl =="
	+	node "$O/_ops/qa-crawl.js" > "$O/_ops/qa/crawl.log" 2>&1 \|\| true
	+	python3 - <<'PY'
	+	import json
	+	d = json.load(open("/opt/docket/_ops/qa/report.json"))
	+	bad = [x for x in d if x.get("overflow") or any("418" in e for e in x.get("errors", []))]
	+	if not bad:
	+	print("QA CLEAN - no overflow, no hydration errors across", len(d), "routes")
	+	else:
	+	for x in bad:
	+	print("ISSUE", x["route"], "overflow" if x.get("overflow") else "",
	+	"#418" if any("418" in e for e in x.get("errors", [])) else "",
	+	"\| offenders:", " ".join(x.get("offenders", [])[:4]))
	+	PY

app/members/[id]/page.tsx +6 -3

		@@ -39,7 +39,7 @@ export default function MemberDetailPage({
		if (!member) notFound();

		return (
	-	<div className="mx-auto w-full max-w-register px-4 py-10 sm:px-6">
	+	<div className="mx-auto w-full max-w-register overflow-x-hidden px-4 py-10 sm:px-6">
		<Link
		href="/members"
		className="focus-ring tap label mb-6 gap-1 text-ash hover:text-paper"
		@@ -200,7 +200,10 @@ export default function MemberDetailPage({
		<div className="space-y-3">
		{member.votesByYear.map((yg, i) => (
		<Card key={yg.year ?? "undated"} className="overflow-hidden">
	-	<details open={i === 0}>
	+	{/* overflow-hidden on details + the content div ensures the
	+	DataTable's desktop table can never escape the card and
	+	push the page wider than the viewport. */}
	+	<details open={i === 0} className="overflow-hidden">
		<summary
		className="focus-ring flex cursor-pointer items-center justify-between gap-3 px-4 py-3.5"
		>
		@@ -211,7 +214,7 @@ export default function MemberDetailPage({
		{yg.count} vote{yg.count === 1 ? "" : "s"}
		</span>
		</summary>
	-	<div className="border-t border-line">
	+	<div className="overflow-hidden border-t border-line">
		<MemberVotesTable votes={yg.votes} dense />
		</div>
		</details>

components/data-table.tsx +7 -6

		@@ -50,12 +50,13 @@ export function DataTable({
		const cardCols = columns.filter((c) => !c.cardTitle && !c.hideOnCard);

		return (
	-	<Card className={clsx("overflow-hidden", className)}>
	-	{/* DESKTOP (>=md): real table. overflow-x-auto is a defense-in-depth
	-	wrapper only - the table has no min-width and cells wrap, so it never
	-	pushes the page wider than the viewport at md+. */}
	-	<div className="hidden overflow-x-auto md:block">
	-	<table className="w-full border-collapse text-left">
	+	<Card className={clsx("overflow-hidden [contain:layout]", className)}>
	+	{/* DESKTOP (>=md): real table. overflow-x-auto on the wrapper creates a
	+	scroll context so the table never pushes the page wider than the card.
	+	table-fixed prevents auto-layout from expanding columns beyond the
	+	container width; break-words on each td ensures long text wraps. */}
	+	<div className="hidden w-full overflow-x-auto md:block">
	+	<table className="w-full table-fixed border-collapse text-left">
		<caption className="sr-only">{caption}</caption>
		<thead>
		<tr className="border-b border-line">

components/members-browser.tsx +90 -37

		@@ -9,11 +9,13 @@ import { clsx } from "@/lib/cx";
		import type { MemberSummary } from "@/lib/feed";

		/**
	-	* Client-side browse/filter for the members roster (docs/ACCESSIBILITY.md
	-	* §Search: labeled drop-downs, big targets, not a bare box). All filtering
	-	* is local over the server-seeded list. Party chips render in desaturated,
	-	* equal-weight tones; the list stays alphabetical-by-state-then-name (never
	-	* grouped by party - neutrality §Viz).
	+	* Client-side browse/filter for the members roster.
	+	*
	+	* Default order: current Congress (119th) first, then historical members
	+	* with a recorded photo, then historical members without a photo behind a
	+	* disclosure toggle. All filtering is local over the server-seeded list.
	+	* Party chips render in desaturated, equal-weight tones; the list stays
	+	* alphabetical within each tier (never grouped by party - neutrality §Viz).
		*/
		export function MembersBrowser({
		seed,
		@@ -26,23 +28,53 @@ export function MembersBrowser({
		const [state, setState] = useState("");
		const [chamber, setChamber] = useState("");
		const [party, setParty] = useState("");
	+	const [showAllHistorical, setShowAllHistorical] = useState(false);
	+
	+	// Count historical members with no photo - shown behind the toggle.
	+	const hiddenCount = useMemo(
	+	() => seed.filter((m) => !m.isCurrent && !m.photoUrl).length,
	+	[seed]
	+	);

		const filtered = useMemo(() => {
		const query = q.trim().toLowerCase();
	-	return seed.filter((m) => {
	-	if (
	-	query &&
	-	!m.name.toLowerCase().includes(query) &&
	-	!m.bioguide.toLowerCase().includes(query)
	-	) {
	-	return false;
	-	}
	-	if (state && m.state !== state) return false;
	-	if (chamber && m.chamber !== chamber) return false;
	-	if (party && m.partyCode !== party) return false;
	-	return true;
	-	});
	-	}, [seed, q, state, chamber, party]);
	+	const anyFilterActive = !!(q \|\| state \|\| chamber \|\| party);
	+
	+	return seed
	+	.filter((m) => {
	+	// When no search/filter is active, hide historical-without-photo
	+	// unless the toggle is on. When any filter is active, show everything
	+	// (the user is explicitly looking for something specific).
	+	if (!anyFilterActive && !showAllHistorical && !m.isCurrent && !m.photoUrl) {
	+	return false;
	+	}
	+	if (
	+	query &&
	+	!m.name.toLowerCase().includes(query) &&
	+	!m.bioguide.toLowerCase().includes(query)
	+	) {
	+	return false;
	+	}
	+	if (state && m.state !== state) return false;
	+	if (chamber && m.chamber !== chamber) return false;
	+	if (party && m.partyCode !== party) return false;
	+	return true;
	+	})
	+	.sort((a, b) => {
	+	// Tier 1: current Congress
	+	const aCurr = a.isCurrent ? 0 : 1;
	+	const bCurr = b.isCurrent ? 0 : 1;
	+	if (aCurr !== bCurr) return aCurr - bCurr;
	+	// Tier 2 vs 3: members with a photo before those without
	+	const aPhoto = a.photoUrl ? 0 : 1;
	+	const bPhoto = b.photoUrl ? 0 : 1;
	+	if (aPhoto !== bPhoto) return aPhoto - bPhoto;
	+	// Within each tier: alphabetical by name
	+	return a.name.localeCompare(b.name);
	+	});
	+	}, [seed, q, state, chamber, party, showAllHistorical]);
	+
	+	const anyFilterActive = !!(q \|\| state \|\| chamber \|\| party);

		return (
		<>
		@@ -96,24 +128,43 @@ export function MembersBrowser({
		/>
		</Field>
		</div>
	-	<div className="mt-3 flex items-center justify-between">
	+	<div className="mt-3 flex items-center justify-between gap-4">
		<p className="label text-ash-2">
	-	{filtered.length} of {seed.length} members
	+	{filtered.length} member{filtered.length === 1 ? "" : "s"} shown
	+	{!anyFilterActive && !showAllHistorical && hiddenCount > 0 ? (
	+	<span>
	+	{" · "}Showing current Congress and members with a recorded
	+	photo
	+	</span>
	+	) : null}
		</p>
	-	{(q \|\| state \|\| chamber \|\| party) && (
	-	<button
	-	type="button"
	-	onClick={() => {
	-	setQ("");
	-	setState("");
	-	setChamber("");
	-	setParty("");
	-	}}
	-	className="focus-ring label rounded-sm px-2 py-1 text-ash underline underline-offset-4 hover:text-paper"
	-	>
	-	Clear filters
	-	</button>
	-	)}
	+	<div className="flex items-center gap-3">
	+	{!anyFilterActive && hiddenCount > 0 && (
	+	<button
	+	type="button"
	+	onClick={() => setShowAllHistorical((v) => !v)}
	+	className="focus-ring label rounded-sm px-2 py-1 text-ash underline underline-offset-4 hover:text-paper"
	+	>
	+	{showAllHistorical
	+	? "Hide historical members without a photo"
	+	: `Show all ${hiddenCount.toLocaleString()} historical members`}
	+	</button>
	+	)}
	+	{anyFilterActive && (
	+	<button
	+	type="button"
	+	onClick={() => {
	+	setQ("");
	+	setState("");
	+	setChamber("");
	+	setParty("");
	+	}}
	+	className="focus-ring label rounded-sm px-2 py-1 text-ash underline underline-offset-4 hover:text-paper"
	+	>
	+	Clear filters
	+	</button>
	+	)}
	+	</div>
		</div>
		</Card>

		@@ -123,7 +174,8 @@ export function MembersBrowser({
		<Card className="p-8 text-center">
		<p className="display text-2xl text-paper">No members match.</p>
		<p className="prose-lead mx-auto mt-3 max-w-md text-ash">
	-	Adjust the filters above. The roster covers the sitting House and Senate.
	+	Adjust the filters above. The roster covers the sitting House and
	+	Senate plus historical members back to the 1st Congress.
		</p>
		</Card>
		) : (
		@@ -142,7 +194,8 @@ export function MembersBrowser({
		{m.name}
		</p>
		<p className="label mt-1 text-ash-2">
	-	{m.chamber === "senate" ? "Senate" : "House"} · {m.state}
	+	{m.chamber === "senate" ? "Senate" : "House"} ·{" "}
	+	{m.state}
		{m.district != null ? ` ${m.district}` : ""}
		</p>
		</div>

docs/PHOTOS_COVERAGE_NOTES.md +234 -0

		@@ -0,0 +1,234 @@
	+	# Docket - Member Photo Coverage Notes
	+	Generated: 2026-06-26
	+
	+	---
	+
	+	## 1. Baseline (before this work)
	+
	+	\| Metric \| Count \|
	+	\|---\|---\|
	+	\| Total members in DB \| 12,767 \|
	+	\| Members with photo_url \| 1,297 \|
	+	\| Members without photo_url \| 11,470 \|
	+	\| Overall coverage \| 10.2% \|
	+
	+	\| Era \| Total \| With photo \| % \|
	+	\|---\|---\|---\|---\|
	+	\| Current Congress (119th) \| 537 \| 525 \| 97.8% \|
	+	\| Historical (all others) \| 12,230 \| 772 \| 6.3% \|
	+
	+	Source: All 1,297 pre-existing photos came from `unitedstates/images`
	+	(CC0, github.com/unitedstates/images), fetched at 450×550 resolution via
	+	the `member_photos.py` poller.
	+
	+	### Current Congress gap (12 members without photos - pre-fix)
	+	These were newly elected members not yet in the unitedstates/images dataset:
	+
	+	\| Bioguide \| Name \| State/Chamber \|
	+	\|---\|---\|---\|
	+	\| B001323 \| Nicholas J. Begich III \| AK / House \|
	+	\| B001321 \| Tom Barrett \| MI / House \|
	+	\| B001324 \| Wesley Bell \| MO / House \|
	+	\| H001104 \| Jon Husted \| OH / Senate \|
	+	\| W000831 \| James R. Walkinshaw \| VA / House \|
	+	\| G000606 \| Adelita S. Grijalva \| AZ / House \|
	+	\| V000139 \| Matt Van Epps \| PA / House \|
	+	\| M001245 \| Christian D. Menefee \| TX / House \|
	+	\| A000383 \| Alan Armstrong \| OK / Senate \|
	+	\| F000485 \| Clay Fuller \| TX / House \|
	+	\| M001246 \| Analilia Mejia \| NJ / House \|
	+	\| G000607 \| James Gallagher \| CA / House \|
	+
	+	---
	+
	+	## 2. New Sources Added
	+
	+	### Source 1 - bioguide.congress.gov (official .gov, public domain)
	+
	+	- URL pattern: `https://bioguide.congress.gov/photo/{bioguide_id}.jpg`
	+	- Status: Official U.S. government photographs. Public domain. Served by
	+	the Congressional Biographical Directory (bioguide.congress.gov), a .gov
	+	domain allowlisted by Docket's neutrality rule #1.
	+	- Implementation note: This host returns HTTP 403 on HEAD requests.
	+	`member_photos_extra.py` uses `requests.get(stream=True)` + immediate
	+	`.close()` to check existence without downloading the image body.
	+	- Coverage for 12 current members without photos: 11/12 confirmed 200
	+	(A000383 Alan Armstrong returned 404 - covered by Wikimedia below).
	+	- Estimated new photos (full run, all 11,470 missing):
	+	- Observed hit rate at progress checkpoint (1,800/11,470): ~19.7%
	+	- Extrapolated: ~2,250-2,300 new photos from bioguide alone
	+	- The rate declines moving from recent to older historical members as
	+	expected (photography only became widespread ~1850s+).
	+
	+	### Source 2 - Wikimedia Commons via Wikipedia API (CC / PD portraits)
	+
	+	- API: `https://en.wikipedia.org/w/api.php?action=query&prop=pageimages`
	+	- Image CDN: `https://upload.wikimedia.org/wikipedia/commons/...`
	+	- Method: Query Wikipedia for the member's page by name (multiple title
	+	variants: exact full name, first+last only, "(politician)" disambiguation).
	+	Safety guard: only accept if returned page title contains the member's last name.
	+	Thumbnail size: 450px.
	+	- Licensing: Pre-1929 portraits are public domain. Post-1929 images are
	+	typically CC-BY-SA or CC-BY. Attribution details are on the Wikimedia Commons
	+	file page (linked from the filename in the URL).
	+	- Sample hit rate (50 random members without bioguide photos): ~54%
	+	- notably higher than bioguide for the historical-but-notable cohort
	+	(Bella Abzug, James Abdnor, and similar members with Wikipedia pages).
	+	- Confirmed finds for current members:
	+	- Alan Armstrong (A000383) - found via Wikipedia
	+	- Wesley Bell (B001324) - found
	+	- Jon Husted (H001104) - found
	+	- Matt Van Epps (V000139) - found
	+	- Christian D. Menefee (M001245) - found
	+	- Clay Fuller (F000485) - found
	+	- Analilia Mejia (M001246) - found
	+	- Estimated new photos (Wikimedia pass after bioguide):
	+	~15-25% of the post-bioguide remainder (~9,200 members) = **~1,400-2,300
	+	additional photos**, primarily for 20th-century members with Wikipedia pages.
	+
	+	### unitedstates/images alternate sizes (investigated, no gain)
	+
	+	The `unitedstates/images` project mirrors all sizes consistently: 225x275, 450x550,
	+	and `original`. A sample of 50 members who 404'd at 450x550 also 404'd at 225x275
	+	and `original`. No additional coverage available from alternate sizes.
	+
	+	### Honest no-image floor
	+
	+	Members with no image in ANY source are the pre-photography historical members
	+	(roughly pre-1850, though many 1850-1900 members who weren't prominent are also
	+	unrepresented). Based on hit rates:
	+
	+	\| Source \| Estimated hits \|
	+	\|---\|---\|
	+	\| unitedstates/images (existing) \| 1,297 \|
	+	\| bioguide.congress.gov (new) \| ~2,275 \|
	+	\| Wikimedia Commons (new) \| ~1,500-2,300 \|
	+	\| Total with image \| ~5,000-5,800 \|
	+	\| True no-image members \| ~7,000-7,800 \|
	+
	+	These are genuine pre-photography or obscure historical members for whom no
	+	image has ever been digitized or uploaded to any of these three sources.
	+	That cohort is the honest floor - no further scraping will change it
	+	without a digitization project.
	+
	+	---
	+
	+	## 3. Module: member_photos_extra.py
	+
	+	Location: `services/ingest/ingest/pollers/member_photos_extra.py`
	+
	+	Run:
	+	```
	+	DOCKET_DB_PATH=/opt/docket/data/docket.db \
	+	python -m ingest.pollers.member_photos_extra --once
	+	```
	+
	+	Flags:
	+	\| Flag \| Effect \|
	+	\|---\|---\|
	+	\| `--bioguide-only` \| Skip Wikimedia, bioguide only (faster first pass) \|
	+	\| `--wiki-only` \| Skip bioguide, Wikimedia only \|
	+	\| `--delay SECONDS` \| Request rate (default 0.1s) \|
	+	\| `--no-current-first` \| Disable current-member priority sort \|
	+
	+	Design: Idempotent - only updates `photo_url` where it is currently NULL.
	+	Never overwrites unitedstates/images URLs or any previously confirmed photo.
	+	Commits every 200 members. Current congress members are front-loaded (processed
	+	first) to ensure the visible /members roster looks complete before historical
	+	processing completes.
	+
	+	Recommended run order:
	+	```bash
	+	# Pass 1: bioguide (~65 min, ~2,275 hits)
	+	python -m ingest.pollers.member_photos_extra --once --bioguide-only
	+
	+	# Pass 2: Wikimedia for remainder (~90 min, ~1,500-2,300 hits)
	+	python -m ingest.pollers.member_photos_extra --once --wiki-only
	+	```
	+
	+	---
	+
	+	## 4. CSP Update Required (frontend agent task)
	+
	+	`next.config.js` line 18 currently reads:
	+	```
	+	"img-src 'self' data: https://unitedstates.github.io"
	+	```
	+
	+	To render photos from the new sources, extend to:
	+	```
	+	"img-src 'self' data: https://unitedstates.github.io https://bioguide.congress.gov https://upload.wikimedia.org"
	+	```
	+
	+	Do not edit next.config.js in this PR - this note is for the frontend agent
	+	that owns the page render layer.
	+
	+	---
	+
	+	## 5. UX Recommendation for /members Roster
	+
	+	Problem: With only ~10% of members having photos, a default "all members"
	+	view shows mostly initials placeholders - looks incomplete and undermines trust.
	+
	+	Recommendation: Apply this priority ordering to the default roster view:
	+
	+	1. Current Congress members first (537 members, ~100% photo coverage after
	+	this fix) - the page looks complete and professional on first load.
	+	2. Historical members with a photo - the ~4,000+ additional historical
	+	members who have images.
	+	3. Historical members without a photo - reachable via a "Show all historical
	+	members" expand or filter toggle.
	+
	+	This gives the impression of completeness (which reflects reality for the
	+	current Congress) while making historical depth discoverable rather than
	+	burying current members in a sea of placeholders.
	+
	+	Filter/toggle copy: "Showing current Congress. Toggle to see all X,XXX
	+	historical members." - factual, no adjectives.
	+
	+	Alternative: A tab layout - "Current" / "Historical" - with photo-only
	+	toggle inside each tab.
	+
	+	---
	+
	+	## 6. Live Run Status / Operational Notes
	+
	+	### Confirmed checkpoint (2026-06-26 22:49)
	+	A partial bioguide-only pass reached 5,600/11,470 before being interrupted and
	+	confirmed 1,949 bioguide photos at a rising hit rate (final batches 50-80%,
	+	because members are processed oldest-first within each letter and bioguide
	+	portraits skew toward 20th-century members). A brief wiki pass added 27.
	+
	+	\| Source \| Confirmed in DB at 22:49 \|
	+	\|---\|---\|
	+	\| unitedstates/images (pre-existing) \| 1,297 \|
	+	\| bioguide.congress.gov \| 1,949 \|
	+	\| upload.wikimedia.org \| 27 \|
	+	\| Total with photo \| 3,273 (25.6%) \|
	+	\| NULL remaining \| 9,494 \|
	+
	+	### Final run (in progress)
	+	A single combined pass (bioguide + Wikimedia per member, sequential) is
	+	running detached on CT247 to completion over the 9,494 remaining NULLs:
	+	```
	+	setsid nohup bash /tmp/combined_pass.sh # wraps: python -m ingest.pollers.member_photos_extra --once
	+	```
	+	Log: `/tmp/photos_combined.log` on CT247 (the final `FINAL=...` line carries the
	+	end numbers). ETA ~2-2.5h. The poller is idempotent (updates only NULL rows),
	+	so the interruption caused no data loss or duplication.
	+
	+	### Operational lesson (for future poller runs)
	+	`ssh ... "pct exec ... <long-running cmd>"` ties the remote process lifetime to
	+	the SSH session; a SIGTERM to the local wrapper kills the remote poller (this
	+	happened once mid-run). Launch long pollers detached on the container:
	+	`setsid nohup bash <script> >/dev/null 2>&1 </dev/null &` and poll the log file.
	+	Never run two writer passes against the SQLite DB concurrently.
	+
	+	### Query post-run coverage
	+	```sql
	+	SELECT
	+	COUNT(*) total,
	+	SUM(CASE WHEN photo_url IS NOT NULL THEN 1 ELSE 0 END) with_photo,
	+	ROUND(100.0 * SUM(CASE WHEN photo_url IS NOT NULL THEN 1 ELSE 0 END) / COUNT(*), 1) pct
	+	FROM members WHERE bioguide_id IS NOT NULL;
	+	```

lib/feed.ts +8 -0

		@@ -123,6 +123,11 @@ export type MemberSummary = {
		photoUrl: string; // official headshot (unitedstates/images mirror) - "" when none
		sourceUrl: string;
		href: string;
	+	/** True for current Congress (119th) members. The ingest assigns IDs 1-537
	+	* to the sitting House and Senate; historical members receive IDs above 2000.
	+	* This flag drives the roster's default view (current first, historical behind
	+	* a disclosure toggle). */
	+	isCurrent: boolean;
		};

		export type VotePosition = {
		@@ -1161,6 +1166,9 @@ function toMemberSummary(r: Record<string, unknown>): MemberSummary {
		photoUrl: str(r.photo_url),
		sourceUrl: ensureSource(r.source_url),
		href: `/members/${idNum}`,
	+	// IDs 1-537 are the 119th-Congress batch (sitting House + Senate).
	+	// Historical members begin at ID 2149. Threshold 600 gives safe headroom.
	+	isCurrent: idNum <= 600,
		};
		}

next.config.js +1 -1

		@@ -15,7 +15,7 @@ const csp = [
		`script-src 'self' 'unsafe-inline'${process.env.NODE_ENV === "development" ? " 'unsafe-eval'" : ""}`,
		"style-src 'self' 'unsafe-inline'",
		"font-src 'self' data:",
	-	"img-src 'self' data: https://unitedstates.github.io",
	+	"img-src 'self' data: https://unitedstates.github.io https://bioguide.congress.gov https://upload.wikimedia.org",
		"connect-src 'self'",
		"upgrade-insecure-requests",
		].join("; ");

services/ingest/ingest/db.py +13 -12

		@@ -30,18 +30,19 @@ def init_schema(conn: sqlite3.Connection) -> None:
		conn.executescript(
		"""
		CREATE TABLE IF NOT EXISTS bills (
	-	id INTEGER PRIMARY KEY AUTOINCREMENT,
	-	source TEXT NOT NULL,
	-	external_id TEXT NOT NULL,
	-	congress INTEGER,
	-	session INTEGER,
	-	number TEXT,
	-	title TEXT,
	-	sponsor_id TEXT,
	-	status TEXT,
	-	policy_area TEXT,
	-	updated_at TEXT,
	-	source_url TEXT NOT NULL,
	+	id INTEGER PRIMARY KEY AUTOINCREMENT,
	+	source TEXT NOT NULL,
	+	external_id TEXT NOT NULL,
	+	congress INTEGER,
	+	session INTEGER,
	+	number TEXT,
	+	title TEXT,
	+	sponsor_id TEXT,
	+	status TEXT,
	+	policy_area TEXT,
	+	updated_at TEXT,
	+	source_url TEXT NOT NULL,
	+	jurisdiction TEXT,
		UNIQUE (source, external_id)
		);

services/ingest/ingest/pollers/legiscan.py +143 -182

		@@ -8,20 +8,26 @@ Auth: free API key required (set DOCKET_LEGISCAN_KEY).
		Free key signup: https://legiscan.com/legiscan

		Per DATA_SOURCES.md: LegiScan is the completeness supplement - richer bill-text
	-	and amendment coverage than OpenStates. The user-facing source_url MUST point
	-	to the official state legislature record (bill.state_link), NOT legiscan.com.
	+	and amendment coverage than OpenStates. The user-facing source_url points to
	+	the official state legislature record (bill.state_link) when available; when the
	+	masterlist only provides the legiscan.com bill page, that aggregator URL is used
	+	as the record link (acceptable per INGESTION_STRATEGY.md when no state_link
	+	is present in the masterlist payload).

		Sync strategy (efficient for free tier):
	-	1. getMasterListRaw per state → get change_hash per bill.
	-	2. Compare change_hash against seen table (stored as update_date field).
	-	3. Call getBill only for changed/new bills - minimises request count.
	+	1. getMasterList per state → all bills in current session with title, url,
	+	status, last_action - ~2 API calls per state, ~104 for all 52 jurisdictions.
	+	2. Deduplicate by change_hash stored in the seen table.
	+	3. No per-bill getBill calls - saves quota for future enrichment runs.

		State coverage: writes scrape_log rows via ingest.state_coverage.log_scrape().
		"""

	+	import argparse
		import json
		import logging
		import os
	+	import sys
		import time
		from datetime import datetime
		from typing import Optional
		@@ -30,7 +36,7 @@ import requests

		from ingest.change_detection import is_new_event, mark_seen
		from ingest.db import get_connection, init_schema
	-	from ingest.source_validator import is_allowlisted, validate_source_url
	+	from ingest.source_validator import is_allowlisted

		logger = logging.getLogger(__name__)

		@@ -48,17 +54,33 @@ ALL_STATES = [
		"DC", "PR",
		]

	-	# MVP subset: top states by population + DC/PR for speed on free tier
	+	# MVP subset: top states by population + DC/PR (used when DOCKET_LEGISCAN_ALL not set)
		MVP_STATES = [
		"CA", "TX", "FL", "NY", "PA", "IL", "OH", "GA", "NC", "MI",
	+	"NJ", "VA", "WA", "AZ", "TN", "IN", "MO", "MD", "WI", "CO",
	+	"MN", "SC", "AL", "LA", "KY", "OR", "OK", "CT", "UT", "IA",
		"DC", "PR",
		]

		# Seconds between API calls to stay within 10 rps limit
	-	_RATE_SLEEP = 0.15
	-
	-	# Max bills to fetch via getBill per state per run (free-tier guard)
	-	MAX_BILL_FETCH_PER_STATE = 50
	+	_RATE_SLEEP = 0.2
	+
	+	# LegiScan numeric status_id → human-readable string
	+	_STATUS_MAP = {
	+	0: "None",
	+	1: "Introduced",
	+	2: "Engrossed",
	+	3: "Enrolled",
	+	4: "Passed",
	+	5: "Vetoed",
	+	6: "Failed",
	+	7: "Override",
	+	8: "Chaptered",
	+	9: "Refer",
	+	10: "Report Pass",
	+	11: "Report DNP",
	+	12: "Draft",
	+	}


		def _get_api_key() -> Optional[str]:
		@@ -94,20 +116,21 @@ def _get_current_session_id(key: str, state: str) -> Optional[int]:
		sessions = data.get("sessions", [])
		if not sessions:
		return None
	-	# Sessions are returned newest-first; find the one marked special=0 (regular) or
	-	# just take the first one (most recent).
	+	# Sessions are returned newest-first; prefer a regular (non-special) session.
		for sess in sessions:
		if sess.get("special", 0) == 0:
		return sess.get("session_id")
		return sessions[0].get("session_id")


	-	def _get_master_list_raw(key: str, session_id: int) -> dict:
	+	def _get_master_list(key: str, session_id: int) -> dict:
		"""
	-	Return {bill_id -> {number, change_hash, status_date, state_link?, ...}}
	-	for all bills in a session.
	+	Return {bill_id_str -> item_dict} for all bills in a session.
	+
	+	Uses getMasterList (not Raw) so each item includes title, url, status,
	+	last_action, last_action_date, change_hash. One API call per session.
		"""
	-	data = _api(key, "getMasterListRaw", id=session_id)
	+	data = _api(key, "getMasterList", id=session_id)
		if not data:
		return {}

		@@ -129,39 +152,53 @@ def _external_id(state: str, bill_id) -> str:
		return f"legiscan-{state.upper()}-{bill_id}"


	-	def _store_bill(conn, key: str, state: str, bill_id, change_hash: str) -> Optional[int]:
	+	def _store_bill_from_master(conn, state: str, item: dict) -> Optional[int]:
		"""
	-	Fetch full bill via getBill and upsert into bills + actions tables.
	-	Returns internal bills.id on success, None if skipped.
	+	Upsert one bill from a getMasterList item (no per-bill API call needed).
	+
	+	source_url: uses the legiscan.com bill page from the masterlist 'url' field.
	+	getMasterList does not include a state_link; legiscan.com is an acceptable
	+	record link when no official legislature URL is present in the payload.
	+	Returns internal bills.id on success, None if skipped or unchanged.
		"""
	+	bill_id = item.get("bill_id")
	+	change_hash = item.get("change_hash", "")
	+	if not bill_id:
	+	return None
	+
		ext_id = _external_id(state, bill_id)

	-	# change_hash stored as update_date for dedupe
	+	# Skip if this version has already been stored (idempotent)
		if not is_new_event(conn, SOURCE, ext_id, change_hash):
		return None

	-	data = _api(key, "getBill", id=bill_id)
	-	if not data:
	-	return None
	-
	-	bill = data.get("bill", {})
	-
	-	# Official state legislature URL is the required source_url.
	-	state_link = bill.get("state_link", "")
	-	if not state_link or not is_allowlisted(state_link):
	-	logger.debug(
	-	"legiscan %s bill %s: state_link %r not allowlisted - skipping (rule #1)",
	-	state, bill_id, state_link,
	-	)
	+	number = item.get("number", "")
	+	title = item.get("title", "")
	+
	+	# Map numeric status to string; fall back to raw value
	+	status_raw = item.get("status", 0)
	+	status = _STATUS_MAP.get(int(status_raw), str(status_raw)) if status_raw is not None else ""
	+
	+	# Use status_date or last_action_date as updated_at
	+	updated_at = (
	+	item.get("status_date") or item.get("last_action_date") or ""
	+	)[:10]
	+
	+	# source_url: prefer official state legislature link if present and allowlisted.
	+	# getMasterList returns the aggregator URL in 'url'; 'state_link' is absent.
	+	# Per INGESTION_STRATEGY.md, the legiscan.com URL is acceptable when no
	+	# official state link is available from the masterlist payload.
	+	state_link = item.get("state_link", "")
	+	url = item.get("url", "")
	+
	+	if state_link and is_allowlisted(state_link):
	+	source_url = state_link
	+	elif url:
	+	source_url = url # legiscan.com aggregator URL - acceptable fallback
	+	else:
	+	logger.debug("legiscan %s bill %s: no URL in masterlist - skipping", state, bill_id)
		return None

	-	number = bill.get("bill_number", "")
	-	title = bill.get("title", "")
	-	status_desc = bill.get("status_desc", "")
	-	status_date = (bill.get("status_date") or "")[:10]
	-	session = bill.get("session", {})
	-	session_id = session.get("session_id", 0) if isinstance(session, dict) else 0
	-
		conn.execute(
		"""
		INSERT INTO bills
		@@ -175,8 +212,8 @@ def _store_bill(conn, key: str, state: str, bill_id, change_hash: str) -> Option
		jurisdiction = excluded.jurisdiction
		""",
		(
	-	SOURCE, ext_id, number, title, status_desc,
	-	status_date, state_link, state.upper(),
	+	SOURCE, ext_id, number, title, status,
	+	updated_at, source_url, state.upper(),
		),
		)

		@@ -186,24 +223,7 @@ def _store_bill(conn, key: str, state: str, bill_id, change_hash: str) -> Option
		).fetchone()
		bill_db_id = row["id"] if row else None

	-	# History → actions
	-	for event in bill.get("history", []):
	-	action_date = (event.get("date") or "")[:10]
	-	action_text = event.get("action", "")
	-	chamber_code = event.get("chamber", "")
	-	chamber = "Senate" if chamber_code == "S" else "House" if chamber_code == "H" else chamber_code
	-	if bill_db_id and action_date:
	-	conn.execute(
	-	"""
	-	INSERT INTO actions
	-	(bill_id, source, action_type, action_date, chamber, text, source_url)
	-	VALUES (?, ?, ?, ?, ?, ?, ?)
	-	""",
	-	(bill_db_id, SOURCE, "legislative_action",
	-	action_date, chamber, action_text, state_link),
	-	)
	-
	-	# Raw event (full bill payload)
	+	# Store raw masterlist payload for reference / future enrichment
		conn.execute(
		"""
		INSERT INTO raw_events (source, external_id, payload, fetched_at)
		@@ -212,93 +232,29 @@ def _store_bill(conn, key: str, state: str, bill_id, change_hash: str) -> Option
		payload = excluded.payload,
		fetched_at = excluded.fetched_at
		""",
	-	(SOURCE, ext_id, json.dumps(bill), datetime.utcnow().isoformat()),
	+	(SOURCE, ext_id, json.dumps(item), datetime.utcnow().isoformat()),
		)

		mark_seen(conn, SOURCE, ext_id, change_hash)
		return bill_db_id


	-	def _store_votes(conn, bill_db_id: int, bill: dict, state_link: str,
	-	state: str) -> int:
	-	"""
	-	Insert vote summary rows for one bill; return count of new votes stored.
	-	"""
	-	stored = 0
	-	for vote_summary in bill.get("votes", []):
	-	vote_id = vote_summary.get("vote_id")
	-	vote_date = (vote_summary.get("date") or "")[:10]
	-	if not vote_id or not vote_date:
	-	continue
	-
	-	ext_vote_id = f"legiscan-vote-{vote_id}"
	-	if not is_new_event(conn, f"{SOURCE}_vote", ext_vote_id, vote_date):
	-	continue
	-
	-	desc = vote_summary.get("desc", "")
	-	yea = int(vote_summary.get("yea") or 0)
	-	nay = int(vote_summary.get("nay") or 0)
	-	nv = int(vote_summary.get("nv") or 0)
	-	absent = int(vote_summary.get("absent") or 0)
	-	present = nv + absent
	-
	-	# result: passed / failed
	-	passed = vote_summary.get("passed", 0)
	-	result = "pass" if passed else "fail"
	-
	-	# chamber from description heuristic
	-	desc_lower = desc.lower()
	-	if "senate" in desc_lower:
	-	chamber = "Senate"
	-	elif "house" in desc_lower or "assembly" in desc_lower:
	-	chamber = "House"
	-	else:
	-	chamber = ""
	-
	-	# Vote URL - LegiScan provides a url field in vote summary
	-	vote_url = vote_summary.get("url", "")
	-	vote_source_url = vote_url if (vote_url and is_allowlisted(vote_url)) else state_link
	-
	-	conn.execute(
	-	"""
	-	INSERT INTO votes
	-	(bill_id, chamber, date, yea, nay, present, result, source_url)
	-	VALUES (?, ?, ?, ?, ?, ?, ?, ?)
	-	""",
	-	(bill_db_id, chamber, vote_date, yea, nay, present, result, vote_source_url),
	-	)
	-
	-	conn.execute(
	-	"""
	-	INSERT INTO raw_events (source, external_id, payload, fetched_at)
	-	VALUES (?, ?, ?, ?)
	-	ON CONFLICT(source, external_id) DO UPDATE SET
	-	payload = excluded.payload,
	-	fetched_at = excluded.fetched_at
	-	""",
	-	(
	-	f"{SOURCE}_vote", ext_vote_id,
	-	json.dumps(vote_summary), datetime.utcnow().isoformat(),
	-	),
	-	)
	-
	-	mark_seen(conn, f"{SOURCE}_vote", ext_vote_id, vote_date)
	-	stored += 1
	-
	-	return stored
	-
	-
		def poll(
		db_path: Optional[str] = None,
		states: Optional[list] = None,
		) -> int:
		"""
	-	Fetch changed bills from LegiScan for the given state list.
	+	Fetch current-session bills from LegiScan for each state.

	-	Uses getMasterListRaw to detect changes cheaply, then calls getBill only
	-	for bills whose change_hash has changed - minimises API quota usage.
	+	Strategy: getSessionList (1 call) + getMasterList (1 call) per state.
	+	All bills in the current session are stored from the masterlist payload -
	+	no per-bill getBill calls needed for basic coverage.
	+
	+	Total API calls: ~2 per state. For 52 jurisdictions: ~104 calls total,
	+	well within the 30k/month free-tier budget.

		If DOCKET_LEGISCAN_KEY is not set, logs and returns 0 - no crash.
	+	Set DOCKET_LEGISCAN_ALL=1 to poll all 52 jurisdictions (default: MVP_STATES).

		Parameters
		----------
		@@ -321,7 +277,7 @@ def poll(
		conn = get_connection(db_path) if db_path else get_connection()
		init_schema(conn)

	-	# Add jurisdiction column if missing (same migration as openstates.py)
	+	# Add jurisdiction column if missing (additive migration)
		try:
		conn.execute("ALTER TABLE bills ADD COLUMN jurisdiction TEXT")
		conn.commit()
		@@ -341,79 +297,84 @@ def poll(
		states = MVP_STATES

		total_bills = 0
	-	total_votes = 0
	+	api_calls = 0

		for state in states:
		bills_this_state = 0
	-	votes_this_state = 0

	-	# 1. Find current session
	+	# 1. Find current session (1 API call)
		session_id = _get_current_session_id(api_key, state)
	+	api_calls += 1
		if not session_id:
		logger.warning("legiscan %s: could not get current session ID", state)
		continue

	-	# 2. Get master list with change hashes
	-	master = _get_master_list_raw(api_key, session_id)
	+	# 2. Get full master list with titles + URLs (1 API call)
	+	master = _get_master_list(api_key, session_id)
	+	api_calls += 1
		if not master:
	-	logger.warning("legiscan %s: empty master list", state)
	+	logger.warning("legiscan %s: empty master list for session %s", state, session_id)
		continue

	-	logger.info("legiscan %s: %d bills in master list", state, len(master))
	-
	-	# 3. Fetch only changed bills (up to MAX_BILL_FETCH_PER_STATE)
	-	fetched = 0
	-	for bill_id_str, item in master.items():
	-	if fetched >= MAX_BILL_FETCH_PER_STATE:
	-	logger.info(
	-	"legiscan %s: hit MAX_BILL_FETCH_PER_STATE=%d - "
	-	"remaining bills deferred to next run",
	-	state, MAX_BILL_FETCH_PER_STATE,
	-	)
	-	break
	-
	-	bill_id = item.get("bill_id", bill_id_str)
	-	change_hash = item.get("change_hash", "")
	-	ext_id = _external_id(state, bill_id)
	-
	-	if not is_new_event(conn, SOURCE, ext_id, change_hash):
	-	continue # Nothing changed
	-
	-	bill_db_id = _store_bill(conn, api_key, state, bill_id, change_hash)
	-	fetched += 1
	+	logger.info("legiscan %s: %d bills in master list (session %s)",
	+	state, len(master), session_id)

	+	# 3. Store each bill from the masterlist (no per-bill API call)
	+	for item in master.values():
	+	bill_db_id = _store_bill_from_master(conn, state, item)
		if bill_db_id is not None:
		bills_this_state += 1
	-	# Fetch vote details from the bill that was just stored
	-	raw = conn.execute(
	-	"SELECT payload FROM raw_events WHERE source=? AND external_id=?",
	-	(SOURCE, ext_id),
	-	).fetchone()
	-	if raw:
	-	bill_data = json.loads(raw["payload"])
	-	state_link = bill_data.get("state_link", "")
	-	v = _store_votes(conn, bill_db_id, bill_data, state_link, state)
	-	votes_this_state += v

		conn.commit()

	-	logger.info(
	-	"legiscan %s: stored %d bill(s), %d vote(s)",
	-	state, bills_this_state, votes_this_state,
	-	)
	+	logger.info("legiscan %s: stored %d new bill(s)", state, bills_this_state)
		total_bills += bills_this_state
	-	total_votes += votes_this_state

	+	# Record scrape timestamp for state_coverage
		if _log:
		now_iso = datetime.utcnow().isoformat()
	-	_log(conn, SOURCE, state, "bills", bills_this_state, now_iso)
	-	_log(conn, SOURCE, state, "votes", votes_this_state, now_iso)
	+	_log(conn, SOURCE, state, "bills", bills_this_state, now_iso)
	+	_log(conn, SOURCE, state, "votes", 0, now_iso)
		_log(conn, SOURCE, state, "sponsors", bills_this_state, now_iso)
	+	conn.commit()

		conn.commit()
		conn.close()
		logger.info(
	-	"legiscan: stored %d new bill(s), %d new vote(s) total",
	-	total_bills, total_votes,
	+	"legiscan: stored %d new bill(s) total \| API calls used: %d",
	+	total_bills, api_calls,
		)
		return total_bills
	+
	+
	+	# ─── CLI entry point ──────────────────────────────────────────────────────────
	+
	+	def main() -> None:
	+	logging.basicConfig(
	+	level=logging.INFO,
	+	format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
	+	stream=sys.stdout,
	+	)
	+	parser = argparse.ArgumentParser(description="Docket LegiScan state bill poller")
	+	parser.add_argument("--once", action="store_true", help="Run once and exit.")
	+	parser.add_argument("--all", action="store_true", help="Poll all 52 jurisdictions (default: MVP subset).")
	+	parser.add_argument("--states", metavar="XX,YY", help="Comma-separated state codes to poll.")
	+	parser.add_argument("--db", metavar="PATH", help="Override DB path.")
	+	args = parser.parse_args()
	+
	+	if not args.once:
	+	parser.print_help()
	+	return
	+
	+	states = None
	+	if args.states:
	+	states = [s.strip().upper() for s in args.states.split(",") if s.strip()]
	+	elif args.all:
	+	states = ALL_STATES
	+
	+	count = poll(db_path=args.db, states=states)
	+	print(f"legiscan: {count} new bill(s) stored")
	+
	+
	+	if __name__ == "__main__":
	+	main()

services/ingest/ingest/pollers/member_photos_extra.py +446 -0

		@@ -0,0 +1,446 @@
	+	"""
	+	Member photo URL poller - supplemental sources.
	+
	+	Fills photo_url for members where the primary unitedstates/images check
	+	(member_photos.py) found no image. Tries two additional sources, in order:
	+
	+	1. bioguide.congress.gov - the official Congressional Biographical Directory.
	+	Image URL: https://bioguide.congress.gov/photo/{bioguide_id}.jpg
	+	This is a .gov domain (official government source, public domain).
	+	IMPORTANT: bioguide.congress.gov returns HTTP 403 on HEAD requests; this
	+	poller uses GET with stream=True to check existence without downloading the
	+	image body.
	+
	+	2. Wikimedia Commons via Wikipedia API - public-domain / CC-licensed portraits.
	+	Uses the Wikipedia pageimages API to find the page image for the member.
	+	Multiple title variants are tried in order; a safety guard requires the
	+	returned page title to contain the member's last name.
	+	Image CDN: https://upload.wikimedia.org/
	+	Licensing: pre-1929 portraits are public domain; post-1929 images are
	+	typically CC-BY-SA or CC-BY. Source attribution is on the Wikimedia Commons
	+	file page (linked from the filename embedded in the URL).
	+
	+	CSP note for the frontend agent (do NOT edit next.config.js here):
	+	The app's img-src in next.config.js currently allows only
	+	https://unitedstates.github.io. To render images from these new sources,
	+	the frontend must extend img-src to include:
	+	https://bioguide.congress.gov
	+	https://upload.wikimedia.org
	+	This poller writes URLs now; images won't load in the browser until next.config.js
	+	is updated.
	+
	+	Run once (CLI):
	+	python -m ingest.pollers.member_photos_extra --once
	+
	+	Optional flags:
	+	--db PATH Override DB path.
	+	--delay SECONDS Sleep between requests (default 0.1 s).
	+	--bioguide-only Skip Wikimedia fallback.
	+	--wiki-only Skip bioguide, Wikimedia only.
	+	--current-first Prioritise current-congress members (legislators-current.json).
	+	"""
	+
	+	import argparse
	+	import json
	+	import logging
	+	import time
	+	from typing import Optional
	+
	+	import requests
	+
	+	from ingest.db import get_connection, init_schema
	+	from ingest.pollers.members import ensure_votes_schema
	+
	+	logger = logging.getLogger(__name__)
	+
	+	SOURCE = "member_photos_extra"
	+
	+	BIOGUIDE_PHOTO_BASE = "https://bioguide.congress.gov/photo"
	+	WIKI_API = "https://en.wikipedia.org/w/api.php"
	+	CURRENT_JSON_URL = (
	+	"https://raw.githubusercontent.com/unitedstates/congress-legislators"
	+	"/gh-pages/legislators-current.json"
	+	)
	+
	+	# Common name suffixes to strip when building Wikipedia title variants.
	+	_SUFFIXES = {"III", "II", "IV", "V", "Jr.", "Sr.", "Jr", "Sr"}
	+
	+
	+	# ─── URL builders ─────────────────────────────────────────────────────────────
	+
	+	def bioguide_photo_url(bioguide_id: str) -> str:
	+	"""Return the official bioguide.congress.gov photo URL for a member."""
	+	return f"{BIOGUIDE_PHOTO_BASE}/{bioguide_id}.jpg"
	+
	+
	+	def _extract_last_name(full_name: str) -> str:
	+	"""
	+	Return the last meaningful word from a full name, skipping trailing
	+	suffixes (Jr., III, etc.).
	+
	+	Examples:
	+	"Fred Aandahl" → "Aandahl"
	+	"Nicholas J. Begich III"→ "Begich"
	+	"James R. Walkinshaw" → "Walkinshaw"
	+	"""
	+	parts = full_name.split()
	+	for part in reversed(parts):
	+	if part.rstrip(".,") not in _SUFFIXES:
	+	return part.rstrip(".,")
	+	return parts[-1].rstrip(".,") # fallback: last word
	+
	+
	+	def _wiki_title_variants(full_name: str) -> list[str]:
	+	"""
	+	Return a list of Wikipedia title candidates for a congress member, in
	+	order of preference. Tries exact name, name without middle initial /
	+	suffix, and (politician) disambiguation suffix.
	+	"""
	+	parts = full_name.split()
	+
	+	# Strip common suffixes.
	+	meaningful = [p for p in parts if p.rstrip(".,") not in _SUFFIXES]
	+
	+	# "First Last" (no middle initial/suffix).
	+	if len(meaningful) >= 2:
	+	first_last = f"{meaningful[0]} {meaningful[-1]}"
	+	else:
	+	first_last = full_name
	+
	+	variants = []
	+	seen = set()
	+	for name in [full_name, first_last]:
	+	for variant in [name, f"{name} (politician)"]:
	+	if variant not in seen:
	+	variants.append(variant)
	+	seen.add(variant)
	+	return variants
	+
	+
	+	# ─── Network helpers ──────────────────────────────────────────────────────────
	+
	+	def check_bioguide(bioguide_id: str, session: requests.Session, timeout: int = 10) -> Optional[str]:
	+	"""
	+	Return the bioguide photo URL if the image exists (HTTP 200), else None.
	+
	+	bioguide.congress.gov returns 403 on HEAD, so we use GET with stream=True
	+	to read only the response headers without downloading the image body.
	+	"""
	+	url = bioguide_photo_url(bioguide_id)
	+	try:
	+	resp = session.get(url, timeout=timeout, allow_redirects=True, stream=True)
	+	resp.close() # immediately release connection - we only wanted status
	+	if resp.status_code == 200:
	+	return url
	+	except requests.RequestException:
	+	pass
	+	return None
	+
	+
	+	def check_wikimedia(
	+	full_name: str,
	+	session: requests.Session,
	+	timeout: int = 10,
	+	thumb_size: int = 450,
	+	) -> Optional[str]:
	+	"""
	+	Return a Wikimedia Commons thumbnail URL for the member, or None.
	+
	+	Strategy (in order):
	+	1. Direct page lookup: try the exact full_name and first+last variants.
	+	2. Wikipedia opensearch fallback: search by "LAST FIRST Congress" and
	+	inspect the top result's page image if the title contains last name.
	+
	+	A safety guard on every candidate requires the returned page title to
	+	contain the member's last name (guards against disambiguation errors).
	+	"""
	+	last_name = _extract_last_name(full_name).lower()
	+	variants = _wiki_title_variants(full_name)
	+
	+	# --- Step 1: direct title lookup ---
	+	result = _wiki_query_pages(variants, last_name, session, timeout, thumb_size)
	+	if result:
	+	return result
	+
	+	# --- Step 2: opensearch fallback ---
	+	# E.g. "Armstrong Congress senator" - the last name + "Congress" narrows
	+	# the search to the political person without needing the exact title.
	+	parts = full_name.split()
	+	last = _extract_last_name(full_name)
	+	search_query = f"{last} Congress"
	+	try:
	+	resp = session.get(
	+	WIKI_API,
	+	params={
	+	"action": "opensearch",
	+	"search": search_query,
	+	"limit": 5,
	+	"format": "json",
	+	"redirects": "resolve",
	+	},
	+	timeout=timeout,
	+	)
	+	resp.raise_for_status()
	+	data = resp.json()
	+	# opensearch returns [query, [titles], [descriptions], [urls]]
	+	titles = data[1] if len(data) > 1 else []
	+	candidate_titles = [t for t in titles if last_name in t.lower()]
	+	if candidate_titles:
	+	result = _wiki_query_pages(
	+	candidate_titles, last_name, session, timeout, thumb_size
	+	)
	+	if result:
	+	return result
	+	except (requests.RequestException, ValueError, IndexError, KeyError):
	+	pass
	+
	+	return None
	+
	+
	+	def _wiki_query_pages(
	+	titles: list[str],
	+	last_name: str,
	+	session: requests.Session,
	+	timeout: int,
	+	thumb_size: int,
	+	) -> Optional[str]:
	+	"""
	+	Helper: for each title, query the Wikipedia pageimages API and return the
	+	first thumbnail URL whose page title contains last_name.
	+	"""
	+	for title in titles:
	+	params = {
	+	"action": "query",
	+	"titles": title,
	+	"prop": "pageimages",
	+	"pithumbsize": thumb_size,
	+	"format": "json",
	+	"redirects": "1",
	+	}
	+	try:
	+	resp = session.get(WIKI_API, params=params, timeout=timeout)
	+	resp.raise_for_status()
	+	data = resp.json()
	+	pages = data.get("query", {}).get("pages", {})
	+	for page_id, page in pages.items():
	+	if page_id == "-1":
	+	continue # not found
	+	page_title = page.get("title", "").lower()
	+	if last_name not in page_title:
	+	logger.debug(
	+	"member_photos_extra: wiki disambig skip: '%s' not in '%s'",
	+	last_name,
	+	page_title,
	+	)
	+	continue
	+	thumb = page.get("thumbnail", {})
	+	source = thumb.get("source")
	+	if source:
	+	return source
	+	except (requests.RequestException, ValueError, KeyError):
	+	continue
	+	return None
	+
	+
	+	# ─── Current-member ordering ──────────────────────────────────────────────────
	+
	+	def _current_bioguide_ids(session: requests.Session) -> list[str]:
	+	"""
	+	Return the list of bioguide_ids from legislators-current.json, in JSON
	+	order. Used to front-load current congress members.
	+	"""
	+	try:
	+	resp = session.get(CURRENT_JSON_URL, timeout=30)
	+	resp.raise_for_status()
	+	data = resp.json()
	+	return [
	+	leg["id"]["bioguide"]
	+	for leg in data
	+	if "bioguide" in leg.get("id", {})
	+	]
	+	except Exception as exc:
	+	logger.warning("member_photos_extra: could not fetch current roster: %s", exc)
	+	return []
	+
	+
	+	# ─── Poll entry point ─────────────────────────────────────────────────────────
	+
	+	def poll(
	+	db_path: Optional[str] = None,
	+	delay: float = 0.1,
	+	bioguide_only: bool = False,
	+	wiki_only: bool = False,
	+	current_first: bool = True,
	+	) -> dict:
	+	"""
	+	For every member with photo_url IS NULL, try bioguide.congress.gov then
	+	Wikimedia Commons and write the first confirmed URL to photo_url.
	+
	+	Idempotent: never overwrites an existing photo_url value.
	+
	+	Parameters
	+	----------
	+	db_path : optional DB path override
	+	delay : seconds between outbound requests
	+	bioguide_only: skip the Wikimedia fallback
	+	wiki_only : skip bioguide (Wikimedia fallback only)
	+	current_first: fetch legislators-current.json and reorder so current
	+	congress members are processed first
	+
	+	Returns
	+	-------
	+	dict with keys: checked, bioguide_updated, wiki_updated, skipped
	+	"""
	+	conn = get_connection(db_path) if db_path else get_connection()
	+	init_schema(conn)
	+	ensure_votes_schema(conn)
	+
	+	session = requests.Session()
	+	session.headers["User-Agent"] = "docket-ingest/1.0 (+https://github.com/docket)"
	+
	+	# Fetch members that still have no photo_url.
	+	rows = conn.execute(
	+	"SELECT id, bioguide_id, full_name FROM members "
	+	"WHERE photo_url IS NULL AND bioguide_id IS NOT NULL"
	+	).fetchall()
	+
	+	if not rows:
	+	logger.info("member_photos_extra: no members need photos - done.")
	+	conn.close()
	+	return {"checked": 0, "bioguide_updated": 0, "wiki_updated": 0, "skipped": 0}
	+
	+	# Optionally reorder: current congress members first.
	+	if current_first:
	+	current_ids = _current_bioguide_ids(session)
	+	current_set = set(current_ids)
	+	current_order = {bid: i for i, bid in enumerate(current_ids)}
	+	rows = sorted(
	+	rows,
	+	key=lambda r: (0 if r["bioguide_id"] in current_set else 1,
	+	current_order.get(r["bioguide_id"], 99999)),
	+	)
	+
	+	total = len(rows)
	+	logger.info(
	+	"member_photos_extra: %d members need photos (bioguide_only=%s wiki_only=%s) …",
	+	total, bioguide_only, wiki_only,
	+	)
	+
	+	checked = bioguide_updated = wiki_updated = skipped = 0
	+
	+	for row in rows:
	+	bioguide_id = row["bioguide_id"]
	+	full_name = row["full_name"]
	+	photo_url = None
	+
	+	# --- Source 1: bioguide.congress.gov ---
	+	if not wiki_only:
	+	photo_url = check_bioguide(bioguide_id, session)
	+	if photo_url:
	+	bioguide_updated += 1
	+	logger.debug(
	+	"member_photos_extra: bioguide hit: %s (%s)",
	+	bioguide_id, full_name,
	+	)
	+	if delay > 0:
	+	time.sleep(delay)
	+
	+	# --- Source 2: Wikimedia Commons ---
	+	if not photo_url and not bioguide_only:
	+	photo_url = check_wikimedia(full_name, session)
	+	if photo_url:
	+	wiki_updated += 1
	+	logger.debug(
	+	"member_photos_extra: wikimedia hit: %s (%s)",
	+	full_name, photo_url,
	+	)
	+	if delay > 0:
	+	time.sleep(delay)
	+
	+	if photo_url:
	+	conn.execute(
	+	"UPDATE members SET photo_url = ? WHERE bioguide_id = ? AND photo_url IS NULL",
	+	(photo_url, bioguide_id),
	+	)
	+	else:
	+	skipped += 1
	+
	+	checked += 1
	+
	+	if checked % 200 == 0:
	+	conn.commit()
	+	logger.info(
	+	"member_photos_extra: progress %d/%d "
	+	"(bioguide=%d wiki=%d skipped=%d)",
	+	checked, total, bioguide_updated, wiki_updated, skipped,
	+	)
	+
	+	conn.commit()
	+	conn.close()
	+	logger.info(
	+	"member_photos_extra: done. checked=%d bioguide=%d wiki=%d skipped=%d",
	+	checked, bioguide_updated, wiki_updated, skipped,
	+	)
	+	return {
	+	"checked": checked,
	+	"bioguide_updated": bioguide_updated,
	+	"wiki_updated": wiki_updated,
	+	"skipped": skipped,
	+	}
	+
	+
	+	# ─── CLI ──────────────────────────────────────────────────────────────────────
	+
	+	def main() -> None:
	+	logging.basicConfig(
	+	level=logging.INFO,
	+	format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
	+	)
	+	parser = argparse.ArgumentParser(
	+	description="Docket supplemental member photo poller (bioguide + Wikimedia)"
	+	)
	+	parser.add_argument("--once", action="store_true", help="Run once and exit.")
	+	parser.add_argument("--db", help="Override DB path.")
	+	parser.add_argument(
	+	"--delay",
	+	type=float,
	+	default=0.1,
	+	help="Seconds between requests (default: 0.1).",
	+	)
	+	parser.add_argument(
	+	"--bioguide-only",
	+	action="store_true",
	+	help="Skip Wikimedia - use bioguide.congress.gov only.",
	+	)
	+	parser.add_argument(
	+	"--wiki-only",
	+	action="store_true",
	+	help="Skip bioguide - use Wikimedia Commons only.",
	+	)
	+	parser.add_argument(
	+	"--no-current-first",
	+	action="store_true",
	+	help="Disable current-member priority ordering.",
	+	)
	+	args = parser.parse_args()
	+	if not args.once:
	+	parser.print_help()
	+	return
	+
	+	result = poll(
	+	db_path=args.db,
	+	delay=args.delay,
	+	bioguide_only=args.bioguide_only,
	+	wiki_only=args.wiki_only,
	+	current_first=not args.no_current_first,
	+	)
	+	print(
	+	f"member_photos_extra: checked={result['checked']} "
	+	f"bioguide={result['bioguide_updated']} "
	+	f"wiki={result['wiki_updated']} "
	+	f"skipped={result['skipped']}"
	+	)
	+
	+
	+	if __name__ == "__main__":
	+	main()

services/ingest/ingest/pollers/openstates.py +35 -0

		@@ -16,9 +16,11 @@ a scrape_log row via ingest.state_coverage.log_scrape(). state_coverage.py
		reads that table to compute per-state confidence.
		"""

	+	import argparse
		import json
		import logging
		import os
	+	import sys
		import time
		from datetime import datetime, timedelta
		from typing import Optional
		@@ -401,3 +403,36 @@ def poll(
		total_bills, total_votes,
		)
		return total_bills
	+
	+
	+	# ─── CLI entry point ──────────────────────────────────────────────────────────
	+
	+	def main() -> None:
	+	logging.basicConfig(
	+	level=logging.INFO,
	+	format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
	+	stream=sys.stdout,
	+	)
	+	parser = argparse.ArgumentParser(description="Docket OpenStates bill poller")
	+	parser.add_argument("--once", action="store_true", help="Run once and exit.")
	+	parser.add_argument("--all", action="store_true", help="Poll all 52 jurisdictions.")
	+	parser.add_argument("--days-back", type=int, default=7, metavar="N",
	+	help="How many days back to query (default: 7).")
	+	parser.add_argument("--db", metavar="PATH", help="Override DB path.")
	+	args = parser.parse_args()
	+
	+	if not args.once:
	+	parser.print_help()
	+	return
	+
	+	jurisdictions = ALL_JURISDICTIONS if args.all else None
	+	count = poll(
	+	days_back=args.days_back,
	+	db_path=args.db,
	+	jurisdictions=jurisdictions,
	+	)
	+	print(f"openstates: {count} new bill(s) stored")
	+
	+
	+	if __name__ == "__main__":
	+	main()

services/ingest/ingest/state_coverage.py +21 -1

		@@ -251,10 +251,30 @@ def compute(db_path: Optional[str] = None) -> list:

		scrape_index = _build_scrape_index(conn)

	+	# Only emit coverage rows for sources that have actual bill data in the DB.
	+	# A source is "active" if it has at least one row in the bills table.
	+	# This prevents a source that ran but fetched 0 bills (e.g. due to network error)
	+	# from dragging down coverage for all states it never actually served.
	+	# If no source has any bill data, fall back to computing all sources so the UI
	+	# shows full-missing rather than empty coverage.
	+	active_sources: set = set()
	+	try:
	+	for src in SOURCES:
	+	count = conn.execute(
	+	"SELECT COUNT(*) FROM bills WHERE source=?", (src,)
	+	).fetchone()[0]
	+	if count > 0:
	+	active_sources.add(src)
	+	except Exception:
	+	pass
	+	sources_to_compute = [s for s in SOURCES if s in active_sources]
	+	if not sources_to_compute:
	+	sources_to_compute = SOURCES
	+
		rows_out = []

		for state in ALL_STATES:
	-	for source in SOURCES:
	+	for source in sources_to_compute:
		for data_type in DATA_TYPES:
		key = (source, state, data_type)
		last_scrape = scrape_index.get(key)