| @@ -1,10 +1,104 @@ | ||
| # Docket - Frontend Notes | ||
| - | Wave F2 + F3 + F4 + F5 + R1 + R2 build log. All surfaces consume the live read-API | |
| + | Wave F2 + F3 + F4 + F5 + R1 + R2 + R3 build log. All surfaces consume the live read-API | |
| (`app/api/*` → `lib/feed.ts` → SQLite at `data/docket.db`). No mock data on | ||
| the deep surfaces; `lib/mock-data.ts` remains only for the Phase-0 lint | ||
| reference and is not imported by any F2/F3/F4/F5 page. | ||
| + | ## R3 (2026-06-27) - Votes show what was voted on + richer sort/filter + home landing (Claude) | |
| + | ||
| + | ### Summary | |
| + | Wave R3 delivers three owner-priority changes. QA CLEAN (17+ routes, zero overflow, zero hydration errors). | |
| + | ||
| + | ### 1. Votes show what was voted on | |
| + | ||
| + | **`lib/feed.ts`** - `Vote` type extended: `billNumber?`, `billTitle?`, `billTextRef?`. | |
| + | `getVotes()`, `getVotesFiltered()`, `getVoteById()` now LEFT JOIN the `bills` table | |
| + | on `votes.bill_id`. Graceful: when no bill is linked the fields are `undefined` and | |
| + | the UI falls back through the priority chain (no breaking change). | |
| + | ||
| + | **New `lib/vote-subject.ts`** - `resolveVoteSubject(v)` implements the full priority chain | |
| + | for identifying what any vote was about. Priority: | |
| + | 1. Linked bill (`billNumber` + `billTitle`) - link to `/bill/[id]` | |
| + | 2. `bill_text_ref` starting with `"PN"` → Presidential Nomination label | |
| + | 3. `bill_text_ref` starting with `"TREATYDOC"` → Treaty document label | |
| + | 4. Other `bill_text_ref` (historical ref, bill not yet in DB) + question | |
| + | 5. Question text from the roll call | |
| + | 6. Result text as last resort - never leads with just "Passed" | |
| + | ||
| + | **`/votes` list** - "what" column is now the `cardTitle` (previously `result`). | |
| + | Primary line: bill number + title (e.g. "H.R. 22 - Financial Exploitation Prevention Act | |
| + | of 2025") for linked bills, or nomination/treaty label, or ref/question. Secondary | |
| + | line: result text. "Passed · 414-2" is now always SECONDARY, never the leading | |
| + | identifier. The list also shows the result + tally in a separate Tally column. | |
| + | ||
| + | **`/votes/[id]` detail** - A prominent card is inserted ABOVE the PageHeader identifying | |
| + | what the vote was about: kind label ("Bill voted on", "Presidential nomination", | |
| + | "Treaty ratification", "Legislative document", "Question voted on"), the primary subject | |
| + | with a link to `/bill/[id]` when available, and the procedural question below it. A reader | |
| + | instantly sees the subject before the tally/roster. | |
| + | ||
| + | ### 2. Sort + deeper filters on every list page and /search | |
| + | ||
| + | **`/votes`** - new `result` filter (Passed/Failed, pushed into SQL as `v.yea > v.nay`) | |
| + | and `sort` control (Newest/By chamber/By result text). `VoteFilter` extended with | |
| + | `resultFilter` and `sort`. `buildVotesWhere()` updated with `v.` table alias for | |
| + | JOIN compatibility. `buildVotesOrder()` helper resolves ORDER BY per sort value. | |
| + | ||
| + | **`/bills`** - new `sort` control (Last action/By congress/By status). `BillFilter.sort` | |
| + | extended to include `"congress"` and `"status"`. `getBills()` ORDER BY fixed to | |
| + | actually honor the sort value (was hardcoded to `updated_at` for all options). | |
| + | ||
| + | **`/spending`** - new `sort` control (Largest first / Most recent / By agency A-Z). | |
| + | `getAwardsFiltered(q, sort?)` signature extended with optional sort param; ORDER BY | |
| + | varies accordingly. | |
| + | ||
| + | **`/members`** - `MembersBrowser` (client component) gets a `sort` state and new | |
| + | "Sort by" `FilterSelect` (Default / Name A-Z / State / Chamber). Named sorts bypass | |
| + | the tier logic; Default preserves the existing current-Congress-first ordering. | |
| + | ||
| + | **`/search`** - type filter chips (All / Bills / Members / Federal Register / Awards / | |
| + | States) control which result groups render. A GET-form sort control (Relevance / | |
| + | Alphabetical / By amount) applies in-memory re-ordering after the cached | |
| + | `globalSearch()` call. Type and sort params are preserved across chip navigation. | |
| + | ||
| + | ### 3. New landing home page at "/" | |
| + | ||
| + | `app/page.tsx` - replaced the `redirect("/register")` with a full orienting landing page: | |
| + | - **Hero**: plain-language explanation of what Docket is (neutral, sourced, .gov-linked), | |
| + | how to use it, and a primary CTA → Search. | |
| + | - **Entry cards grid** (2-up / 4-up responsive): Search, The Register, Bills, Votes, | |
| + | Members, Spending, Calendar, States - each with a short description of what it does. | |
| + | - **How-to section**: two cards - "For students and educators" (how to cite) and | |
| + | "How to cite" (using the Cite this button). | |
| + | - **About** card: plain statement of neutrality policy + links to /how-we-pick and | |
| + | /who-pays. | |
| + | The Register feed lives at `/register` (unchanged); the home page links to it. No forced | |
| + | tutorial, no blocking interstitial - purely orienting. | |
| + | ||
| + | ### Data layer changes (`lib/feed.ts`) | |
| + | - `Vote` type: + `billNumber?`, `billTitle?`, `billTextRef?` | |
| + | - `VoteFilter` type: + `resultFilter?`, `sort?` | |
| + | - `buildVotesWhere()`: `v.` alias for JOIN compatibility; resultFilter clause | |
| + | - New `buildVotesOrder()`: ORDER BY per sort value | |
| + | - `getVoteCountFiltered()`: uses `FROM votes v` alias to match WHERE clauses | |
| + | - `getVotes()`: delegates to `getVotesFiltered()` | |
| + | - `getVotesFiltered()`: LEFT JOIN bills; honor sort; pick up bill_number/bill_title/bill_text_ref | |
| + | - `getVoteById()`: LEFT JOIN bills; return bill fields + bill_text_ref | |
| + | - `toVote()`: picks up billNumber, billTitle, billTextRef from JOIN result | |
| + | - `BillFilter.sort`: + `"congress"` | `"status"` | |
| + | - `getBills()`: ORDER BY now properly varies by sort value | |
| + | - `getAwardsFiltered(q?, sort?)`: second param controls ORDER BY (amount/date/agency) | |
| + | ||
| + | ### New file: `lib/vote-subject.ts` | |
| + | Shared `resolveVoteSubject(v)` utility - used by both `/votes` and `/votes/[id]`. | |
| + | Types exported: `VoteSubjectKind`, `VoteSubject`, `VoteSubjectInput`. | |
| + | ||
| + | ### QA / screenshots | |
| + | - `_ops/qa-view/home-landing-r3.png` - new home page | |
| + | - `_ops/qa-view/votes-with-bill-r3.png` - votes list showing bill identifiers | |
| + | - `_ops/qa-view/vote-detail-r3.png` - vote detail with bill banner above roster | |
| + | ||
| ## R2 (2026-06-27) - Search/filter on every list page + first-class Cite This (Claude) | ||
| ### Summary |
| @@ -0,0 +1,159 @@ | ||
| + | VOTE_BILL_LINK_NOTES.md | |
| + | Docket - Vote-to-Bill Linkage Fix (2026-06-27) | |
| + | ||
| + | PROBLEM | |
| + | ------- | |
| + | votes.bill_id was NULL for all 35,569 votes in the DB. The pollers store the | |
| + | raw bill reference in bill_text_ref (and attempt an inline lookup), but all | |
| + | votes were ingested before the bills table was populated, leaving every bill_id | |
| + | as NULL. | |
| + | ||
| + | MODULE CREATED | |
| + | -------------- | |
| + | ingest/link_votes.py | |
| + | ||
| + | Retroactively populates votes.bill_id by parsing bill_text_ref and matching | |
| + | against the bills table on external_id. Idempotent: only processes rows where | |
| + | bill_id IS NULL. Safe to re-run after new bills are ingested. | |
| + | ||
| + | Usage: | |
| + | python -m ingest.link_votes # one-shot, uses DOCKET_DB_PATH | |
| + | python -m ingest.link_votes --dry-run # inspect without writing | |
| + | python -m ingest.link_votes --db /path # explicit DB path | |
| + | ||
| + | REFERENCE FORMATS HANDLED | |
| + | -------------------------- | |
| + | 1. House Clerk XML space-separated (from votes_house.py): | |
| + | "H R 22" -> hr -> 119-hr-22 | |
| + | "H RES 977" -> hres -> 119-hres-977 | |
| + | "H J RES 140" -> hjres -> 119-hjres-140 | |
| + | "H CON RES 14" -> hconres | |
| + | "S 1" -> s | |
| + | "S RES 22" -> sres | |
| + | "S J RES 22" -> sjres | |
| + | "S CON RES 13" -> sconres | |
| + | ||
| + | 2. Voteview compact (from voteview.py bill_number CSV column): | |
| + | "HR22" -> hr -> 119-hr-22 | |
| + | "HRES977" -> hres -> 119-hres-977 | |
| + | "HJRES140" -> hjres | |
| + | "HCONRES14" -> hconres | |
| + | "S1" -> s | |
| + | "SRES22" -> sres | |
| + | "SJRES22" -> sjres | |
| + | "SCONRES13" -> sconres | |
| + | ||
| + | BEFORE / AFTER | |
| + | -------------- | |
| + | Total votes: 35,569 | |
| + | Linked (bill_id set) BEFORE: 0 | |
| + | Candidate rows (bill_text_ref set): 34,323 | |
| + | Matched and linked: 1,055 | |
| + | Linked (bill_id set) AFTER: 1,055 | |
| + | Match rate (of candidates): 3.1% | |
| + | Match rate (congress 119 only): 71.1% | |
| + | ||
| + | BREAKDOWN OF UNLINKED | |
| + | --------------------- | |
| + | Presidential Nominations (PN*): 3,319 -- Senate confirmation votes. | |
| + | No bill equivalent. Correct to | |
| + | leave bill_id NULL. Frontend | |
| + | should show "Nomination" label. | |
| + | ||
| + | Treaty documents (TREATYDOC*): 67 -- Senate treaty ratification. | |
| + | No bill equivalent. Show | |
| + | "Treaty Document" label. | |
| + | ||
| + | Historical (congress 101-118): 29,882 -- Bills for these congresses are | |
| + | NOT yet in the bills table. | |
| + | The govinfo/congress pollers only | |
| + | cover congress 119 today. | |
| + | See "EXPANDING COVERAGE" below. | |
| + | ||
| + | Refs with no congress: 40 -- Cannot disambiguate congress. | |
| + | Skip is safe. | |
| + | ||
| + | 3 REAL EXAMPLES | |
| + | --------------- | |
| + | 1. vote_id=74 | |
| + | question: "On Retaining Division A" | |
| + | ref: "H R 6938" | |
| + | bill: H.R. 6938 - Commerce, Justice, Science; Energy and Water | |
| + | Development; and Interior and Environment Appropriations Act, 2026 | |
| + | ||
| + | 2. vote_id=71 | |
| + | question: "On Ordering the Previous Question" | |
| + | ref: "H RES 977" | |
| + | bill: H.Res. 977 - Providing for consideration of H.R. 4593 (showerhead | |
| + | energy standards), H.R. 5184 (manufactured housing standards), | |
| + | and H.R. 6938 (appropriations) | |
| + | ||
| + | 3. vote_id=73 | |
| + | question: "On Motion to Discharge" | |
| + | ref: "H RES 780" | |
| + | bill: H.Res. 780 - Providing for consideration of H.R. 1834 (Break | |
| + | the Gridlock Act) | |
| + | ||
| + | RUN.PY WIRING | |
| + | ------------- | |
| + | "link_votes" step added to PIPELINE in ingest/run.py immediately after | |
| + | "votes_enrich". This means the linkage step runs automatically on every | |
| + | full-pipeline invocation, catching any new votes that gained a matching bill | |
| + | since the last run. | |
| + | ||
| + | Pipeline position (excerpt from run.py): | |
| + | ("votes_enrich", "ingest.pollers.votes_enrich", []), | |
| + | ("link_votes", "ingest.link_votes", []), <-- ADDED | |
| + | ||
| + | POLLERS: FORWARD-LOOKING NOTE | |
| + | ------------------------------ | |
| + | votes_house.py already attempts inline bill_id resolution at ingest time | |
| + | (lines 317-319). It calls _lookup_bill_id(conn, legis_external_id) before | |
| + | inserting the vote row. This works correctly IF the bill is already in the DB | |
| + | when the vote is polled. | |
| + | ||
| + | Since bills are ingested by a separate poller (govinfo/congress.py) that runs | |
| + | earlier in the pipeline, the inline lookup will succeed for any vote polled | |
| + | AFTER the bills poller has run. Votes polled before their bills are in the DB | |
| + | get bill_id=NULL at insert time and are then fixed by link_votes on the next | |
| + | full pipeline run. | |
| + | ||
| + | voteview.py inserts with bill_id=None explicitly and relies on link_votes | |
| + | for backfill (documented in the source comment "look up later if needed"). | |
| + | ||
| + | votes_senate.py does NOT currently parse the bill reference from Senate XML | |
| + | (the <document_title>/<document_number> field). Senate vote refs come from | |
| + | Voteview's bill_number CSV column instead. No change needed in votes_senate.py. | |
| + | ||
| + | EXPANDING COVERAGE (ACTION ITEMS) | |
| + | ----------------------------------- | |
| + | To link historical votes (congress 101-118, ~29,882 rows), ingest bills for | |
| + | those congresses. Then re-run link_votes: | |
| + | ||
| + | # Ingest bills for all available congresses via govinfo (if supported): | |
| + | python -m ingest.pollers.govinfo --congress-start 101 --congress-end 118 | |
| + | ||
| + | # Re-run linkage (idempotent, picks up new matches): | |
| + | python -m ingest.link_votes | |
| + | ||
| + | Alternatively, consider ingesting bulk bill data from congress.gov or ProPublica | |
| + | for historical congresses. The linkage module will automatically pick up any | |
| + | bills with the format "{congress}-{type}-{number}" in bills.external_id. | |
| + | ||
| + | FRONTEND INTEGRATION NOTES | |
| + | ---------------------------- | |
| + | The frontend agent should JOIN on votes.bill_id to display bill information: | |
| + | ||
| + | SELECT v.*, b.number, b.title, b.status | |
| + | FROM votes v | |
| + | LEFT JOIN bills b ON b.id = v.bill_id | |
| + | WHERE ... | |
| + | ||
| + | For votes where bill_id IS NULL: | |
| + | - If bill_text_ref LIKE 'PN%' -> label as "Nomination" | |
| + | - If bill_text_ref LIKE 'TREATYDOC%' -> label as "Treaty Document" | |
| + | - If bill_text_ref IS NOT NULL -> show raw ref + "(bill not in DB)" | |
| + | - If bill_text_ref IS NULL -> label as "Procedural / No bill" | |
| + | ||
| + | The question field often contains a human-readable description ("On Passage", | |
| + | "On Motion to Recommit") that can complement the bill title in the UI. |
| @@ -41,18 +41,20 @@ export default function BillsPage({ | ||
| policyArea?: string | string[]; | ||
| q?: string | string[]; | ||
| congress?: string | string[]; | ||
| + | sort?: string | string[]; | |
| }; | ||
| }) { | ||
| const policyArea = asString(searchParams.policyArea); | ||
| const q = asString(searchParams.q); | ||
| const congressRaw = asString(searchParams.congress); | ||
| const congressNum = parseInt(congressRaw ?? "", 10); | ||
| + | const sortParam = asString(searchParams.sort); | |
| const billFilter = { | ||
| policyArea, | ||
| q, | ||
| congress: Number.isFinite(congressNum) ? congressNum : undefined, | ||
| - | sort: "updated" as const, | |
| + | sort: (sortParam === "congress" ? "congress" : sortParam === "status" ? "status" : "updated") as "updated" | "congress" | "status", | |
| }; | ||
| // Push filtering into SQL - fetch only the 100 we render + a cheap count. | ||
| const bills = getBills(billFilter, 100, 0); | ||
| @@ -61,7 +63,7 @@ export default function BillsPage({ | ||
| const policyAreas = getPolicyAreaLabels(); | ||
| const congresses = getBillCongresses(); | ||
| - | const isFiltered = !!(policyArea || q || Number.isFinite(congressNum)); | |
| + | const isFiltered = !!(policyArea || q || Number.isFinite(congressNum) || sortParam); | |
| const activeChips: Array<{ label: string; href: string }> = []; | ||
| if (policyArea) | ||
| @@ -157,6 +159,18 @@ export default function BillsPage({ | ||
| ]} | ||
| /> | ||
| </FilterField> | ||
| + | <FilterField label="Sort by"> | |
| + | <FilterSelect | |
| + | name="sort" | |
| + | defaultValue={sortParam ?? ""} | |
| + | aria-label="Sort bills" | |
| + | options={[ | |
| + | { value: "", label: "Last action (newest)" }, | |
| + | { value: "congress", label: "By congress (newest)" }, | |
| + | { value: "status", label: "By status" }, | |
| + | ]} | |
| + | /> | |
| + | </FilterField> | |
| <FilterSubmit /> | ||
| </SearchFilterBar> | ||
| </form> |
| @@ -1,5 +1,179 @@ | ||
| - | import { redirect } from "next/navigation"; | |
| + | import type { Metadata } from "next"; | |
| + | import Link from "next/link"; | |
| + | import { Card } from "@/components/ui"; | |
| - | export default function RootPage() { | |
| - | redirect("/register"); | |
| + | export const metadata: Metadata = { | |
| + | title: "Docket - The public record of the U.S. government", | |
| + | description: | |
| + | "Docket is a neutral, sourced record of official federal government activity: bills, votes, spending, and more. Every fact links to its official .gov source.", | |
| + | }; | |
| + | ||
| + | export const revalidate = 3600; | |
| + | ||
| + | /** Entry cards for the main sections. */ | |
| + | const SECTIONS = [ | |
| + | { | |
| + | href: "/search", | |
| + | eyebrow: "Search", | |
| + | title: "Search the record", | |
| + | desc: "Look up any bill, member of Congress, federal agency, or topic.", | |
| + | }, | |
| + | { | |
| + | href: "/register", | |
| + | eyebrow: "Live feed", | |
| + | title: "The Register", | |
| + | desc: "A ranked, live feed of official government actions: bills passed, votes taken, and dollars awarded.", | |
| + | }, | |
| + | { | |
| + | href: "/bills", | |
| + | eyebrow: "Legislation", | |
| + | title: "Bills", | |
| + | desc: "Laws and legislation. Status, sponsor, what changed between versions, and the official record.", | |
| + | }, | |
| + | { | |
| + | href: "/votes", | |
| + | eyebrow: "Roll calls", | |
| + | title: "Votes", | |
| + | desc: "House and Senate roll-call votes. See how every member voted, by name.", | |
| + | }, | |
| + | { | |
| + | href: "/members", | |
| + | eyebrow: "Representatives", | |
| + | title: "Members", | |
| + | desc: "Every member of Congress - past and present. Party, state, vote record, and sponsored bills.", | |
| + | }, | |
| + | { | |
| + | href: "/spending", | |
| + | eyebrow: "Federal dollars", | |
| + | title: "Spending", | |
| + | desc: "Federal awards and contracts from USAspending.gov, plus OMB apportionment records.", | |
| + | }, | |
| + | { | |
| + | href: "/calendar", | |
| + | eyebrow: "Dates and schedules", | |
| + | title: "Calendar", | |
| + | desc: "When laws take effect. When bills are scheduled for committee or floor action.", | |
| + | }, | |
| + | { | |
| + | href: "/states", | |
| + | eyebrow: "Geography", | |
| + | title: "States", | |
| + | desc: "Bills, members, and legislative activity organized by state.", | |
| + | }, | |
| + | ] as const; | |
| + | ||
| + | export default function HomePage() { | |
| + | return ( | |
| + | <div className="mx-auto w-full max-w-register px-4 py-10 sm:px-6"> | |
| + | ||
| + | {/* ── Hero ────────────────────────────────────────────────── */} | |
| + | <header className="border-b border-line pb-10"> | |
| + | <p className="label mb-3">Docket</p> | |
| + | <h1 | |
| + | className="display text-paper" | |
| + | style={{ fontSize: "clamp(1.875rem, 7vw, 3rem)", lineHeight: 1.05 }} | |
| + | > | |
| + | The public record of the U.S. government | |
| + | </h1> | |
| + | <p className="prose-lead mt-5 max-w-2xl text-ash"> | |
| + | Docket is a neutral, sourced record of official federal government | |
| + | activity. Every fact shown here comes directly from an official{" "} | |
| + | <span className="font-mono text-[13px]">.gov</span> source, with a | |
| + | link. No commentary. No characterization. Facts, on the record. | |
| + | </p> | |
| + | <p className="mt-4 max-w-2xl text-ash" style={{ fontSize: "1rem", lineHeight: 1.6 }}> | |
| + | Use this site to look up a bill, see how your representative voted, | |
| + | track federal spending, or cite a government action in a paper or | |
| + | report. All records are free to access. | |
| + | </p> | |
| + | <div className="mt-6"> | |
| + | <Link | |
| + | href="/search" | |
| + | className="focus-ring inline-flex min-h-[52px] items-center rounded-[3px] border border-line bg-surface px-6 font-mono text-[13px] text-paper transition-colors hover:bg-graphite" | |
| + | > | |
| + | Search the record → | |
| + | </Link> | |
| + | </div> | |
| + | </header> | |
| + | ||
| + | {/* ── Entry cards ─────────────────────────────────────────── */} | |
| + | <section aria-label="Browse sections" className="mt-10"> | |
| + | <p className="label mb-6 text-ash">What you can do here</p> | |
| + | <div className="grid gap-4 sm:grid-cols-2 lg:grid-cols-4"> | |
| + | {SECTIONS.map((s) => ( | |
| + | <Link | |
| + | key={s.href} | |
| + | href={s.href} | |
| + | className="focus-ring group rounded-[3px] border border-line bg-surface p-5 transition-colors hover:bg-graphite" | |
| + | > | |
| + | <p className="label mb-1.5 text-ash-2">{s.eyebrow}</p> | |
| + | <p className="font-mono text-[14px] text-paper group-hover:underline group-hover:underline-offset-4"> | |
| + | {s.title} | |
| + | </p> | |
| + | <p className="mt-2 text-[13px] leading-snug text-ash">{s.desc}</p> | |
| + | </Link> | |
| + | ))} | |
| + | </div> | |
| + | </section> | |
| + | ||
| + | {/* ── How to use / How to cite ─────────────────────────────── */} | |
| + | <section className="mt-12 border-t border-line pt-8"> | |
| + | <div className="grid gap-6 sm:grid-cols-2"> | |
| + | <Card className="p-5"> | |
| + | <p className="label mb-2 text-ash-2">How to use this site</p> | |
| + | <p className="font-mono text-[14px] text-paper">For students and educators</p> | |
| + | <p className="mt-2 text-[13px] leading-snug text-ash"> | |
| + | Start with{" "} | |
| + | <Link href="/search" className="focus-ring underline underline-offset-4 hover:text-paper"> | |
| + | Search | |
| + | </Link>{" "} | |
| + | to find a topic. Click any bill or vote to open its full record. Every | |
| + | detail page has a{" "} | |
| + | <span className="font-mono text-[12px]">Cite this</span> button that | |
| + | generates a citation in APA, MLA, or Chicago format with a permalink | |
| + | and the official .gov source. | |
| + | </p> | |
| + | </Card> | |
| + | ||
| + | <Card className="p-5"> | |
| + | <p className="label mb-2 text-ash-2">How to cite this site</p> | |
| + | <p className="font-mono text-[14px] text-paper">APA / MLA / Chicago</p> | |
| + | <p className="mt-2 text-[13px] leading-snug text-ash"> | |
| + | On any bill, vote, or member page, scroll to the bottom and use the{" "} | |
| + | <span className="font-mono text-[12px]">Cite this record</span> panel. | |
| + | It fills in the author, date, title, permalink, and the official government | |
| + | source automatically. No manual formatting required. | |
| + | </p> | |
| + | </Card> | |
| + | </div> | |
| + | </section> | |
| + | ||
| + | {/* ── Neutrality note ─────────────────────────────────────── */} | |
| + | <section className="mt-8"> | |
| + | <Card className="p-5"> | |
| + | <p className="label mb-2 text-ash-2">About this site</p> | |
| + | <p className="text-[13px] leading-snug text-ash"> | |
| + | Docket is neutral by design. It does not editorialize, rank by | |
| + | political popularity, or surface opinion. The feed is ordered by | |
| + | action date and dollar magnitude only - the same factors as the | |
| + | official record. Party colors appear only in desaturated, equal-weight | |
| + | tones. No comment section. No ads. All source links open the official | |
| + | .gov record directly. | |
| + | </p> | |
| + | <p className="mt-3 text-[13px] leading-snug text-ash"> | |
| + | See the{" "} | |
| + | <Link href="/how-we-pick" className="focus-ring underline underline-offset-4 hover:text-paper"> | |
| + | How we pick what to show | |
| + | </Link>{" "} | |
| + | and{" "} | |
| + | <Link href="/who-pays" className="focus-ring underline underline-offset-4 hover:text-paper"> | |
| + | Who pays for this | |
| + | </Link>{" "} | |
| + | pages for the full editorial and funding policy. | |
| + | </p> | |
| + | </Card> | |
| + | </section> | |
| + | ||
| + | </div> | |
| + | ); | |
| } |
| @@ -14,22 +14,59 @@ export const metadata: Metadata = { | ||
| export const revalidate = 300; | ||
| + | type ResultType = "all" | "bills" | "members" | "executive" | "awards" | "states"; | |
| + | ||
| + | const TYPE_LABELS: Record<ResultType, string> = { | |
| + | all: "All types", | |
| + | bills: "Bills", | |
| + | members: "Members", | |
| + | executive: "Federal Register", | |
| + | awards: "Federal awards", | |
| + | states: "States", | |
| + | }; | |
| + | ||
| export default function SearchPage({ | ||
| searchParams, | ||
| }: { | ||
| - | searchParams: { q?: string }; | |
| + | searchParams: { q?: string; type?: string; sort?: string }; | |
| }) { | ||
| const q = (searchParams.q ?? "").trim(); | ||
| + | const typeParam = (searchParams.type ?? "all") as ResultType; | |
| + | const sortParam = searchParams.sort ?? ""; | |
| + | ||
| const results: SearchResults = q | ||
| ? globalSearch(q) | ||
| : { bills: [], awards: [], members: [], executive: [], states: [] }; | ||
| + | // Apply in-page sort to results (no DB re-query needed since globalSearch is cached). | |
| + | const sortedResults = applySortToResults(results, sortParam); | |
| + | ||
| const total = | ||
| - | results.bills.length + | |
| - | results.members.length + | |
| - | results.executive.length + | |
| - | results.awards.length + | |
| - | results.states.length; | |
| + | sortedResults.bills.length + | |
| + | sortedResults.members.length + | |
| + | sortedResults.executive.length + | |
| + | sortedResults.awards.length + | |
| + | sortedResults.states.length; | |
| + | ||
| + | // Which type to show (all, or filtered to one type). | |
| + | const activeType: ResultType = | |
| + | ["all", "bills", "members", "executive", "awards", "states"].includes(typeParam) | |
| + | ? typeParam | |
| + | : "all"; | |
| + | ||
| + | const showBills = activeType === "all" || activeType === "bills"; | |
| + | const showMembers = activeType === "all" || activeType === "members"; | |
| + | const showExec = activeType === "all" || activeType === "executive"; | |
| + | const showAwards = activeType === "all" || activeType === "awards"; | |
| + | const showStates = activeType === "all" || activeType === "states"; | |
| + | ||
| + | // Visible total respects type filter. | |
| + | const visibleTotal = | |
| + | (showBills ? sortedResults.bills.length : 0) + | |
| + | (showMembers ? sortedResults.members.length : 0) + | |
| + | (showExec ? sortedResults.executive.length : 0) + | |
| + | (showAwards ? sortedResults.awards.length : 0) + | |
| + | (showStates ? sortedResults.states.length : 0); | |
| return ( | ||
| <div className="mx-auto w-full max-w-register px-4 py-10 sm:px-6"> | ||
| @@ -65,11 +102,85 @@ export default function SearchPage({ | ||
| {q ? ( | ||
| <div className="mt-10"> | ||
| + | {/* Type filter row + sort control */} | |
| + | <div className="mb-6 flex flex-wrap items-center justify-between gap-3"> | |
| + | {/* Type chips */} | |
| + | <div className="flex flex-wrap gap-2" role="group" aria-label="Filter results by type"> | |
| + | {(["all", "bills", "members", "executive", "awards", "states"] as ResultType[]).map( | |
| + | (type) => { | |
| + | const count = | |
| + | type === "all" | |
| + | ? total | |
| + | : type === "bills" | |
| + | ? sortedResults.bills.length | |
| + | : type === "members" | |
| + | ? sortedResults.members.length | |
| + | : type === "executive" | |
| + | ? sortedResults.executive.length | |
| + | : type === "awards" | |
| + | ? sortedResults.awards.length | |
| + | : sortedResults.states.length; | |
| + | if (type !== "all" && count === 0) return null; | |
| + | const active = activeType === type; | |
| + | return ( | |
| + | <Link | |
| + | key={type} | |
| + | href={buildHref(q, type, sortParam)} | |
| + | aria-current={active ? "true" : undefined} | |
| + | className={[ | |
| + | "focus-ring label inline-flex min-h-[44px] items-center rounded-[2px] border px-3 transition-colors", | |
| + | active | |
| + | ? "border-paper bg-surface text-paper" | |
| + | : "border-line bg-surface text-ash hover:bg-graphite hover:text-paper", | |
| + | ].join(" ")} | |
| + | > | |
| + | {TYPE_LABELS[type]} | |
| + | {count > 0 ? ( | |
| + | <span className="ml-1.5 tabular-nums text-ash-2"> | |
| + | {count} | |
| + | </span> | |
| + | ) : null} | |
| + | </Link> | |
| + | ); | |
| + | } | |
| + | )} | |
| + | </div> | |
| + | ||
| + | {/* Sort control - GET form preserving q + type */} | |
| + | <form method="get" action="/search" className="flex items-center gap-2"> | |
| + | <input type="hidden" name="q" value={q} /> | |
| + | {activeType !== "all" && ( | |
| + | <input type="hidden" name="type" value={activeType} /> | |
| + | )} | |
| + | <label htmlFor="search-sort" className="label text-ash-2"> | |
| + | Sort | |
| + | </label> | |
| + | <select | |
| + | id="search-sort" | |
| + | name="sort" | |
| + | defaultValue={sortParam} | |
| + | aria-label="Sort search results" | |
| + | className="focus-ring min-h-[44px] rounded-[3px] border border-line bg-surface px-3 py-2 font-mono text-[13px] text-paper" | |
| + | > | |
| + | <option value="">Relevance / newest</option> | |
| + | <option value="alpha">Alphabetical (A-Z)</option> | |
| + | <option value="amount">By amount (awards)</option> | |
| + | </select> | |
| + | <button | |
| + | type="submit" | |
| + | className="focus-ring label min-h-[44px] rounded-[3px] border border-line bg-surface px-3 text-paper hover:bg-graphite" | |
| + | > | |
| + | Apply | |
| + | </button> | |
| + | </form> | |
| + | </div> | |
| + | ||
| <p className="label mb-6 text-ash"> | ||
| - | {total} result{total === 1 ? "" : "s"} for “{q}” | |
| + | {visibleTotal} result{visibleTotal === 1 ? "" : "s"} for “{q}” | |
| + | {activeType !== "all" ? ` · ${TYPE_LABELS[activeType]}` : ""} | |
| </p> | ||
| - | {total === 0 ? ( | |
| + | {visibleTotal === 0 ? ( | |
| <Card className="p-8 text-center"> | ||
| <p className="display text-2xl text-paper">No matches on record.</p> | ||
| <p className="prose-lead mx-auto mt-3 max-w-md text-ash"> | ||
| @@ -78,9 +189,9 @@ export default function SearchPage({ | ||
| </Card> | ||
| ) : ( | ||
| <div className="space-y-10"> | ||
| - | {results.members.length > 0 ? ( | |
| - | <ResultGroup title="Members" count={results.members.length}> | |
| - | {results.members.map((m) => ( | |
| + | {showMembers && sortedResults.members.length > 0 ? ( | |
| + | <ResultGroup title="Members" count={sortedResults.members.length}> | |
| + | {sortedResults.members.map((m) => ( | |
| <Link | ||
| key={m.id} | ||
| href={m.href} | ||
| @@ -101,9 +212,9 @@ export default function SearchPage({ | ||
| </ResultGroup> | ||
| ) : null} | ||
| - | {results.bills.length > 0 ? ( | |
| - | <ResultGroup title="Bills" count={results.bills.length}> | |
| - | {results.bills.map((b) => ( | |
| + | {showBills && sortedResults.bills.length > 0 ? ( | |
| + | <ResultGroup title="Bills" count={sortedResults.bills.length}> | |
| + | {sortedResults.bills.map((b) => ( | |
| <Link | ||
| key={b.id} | ||
| href={b.href} | ||
| @@ -119,9 +230,9 @@ export default function SearchPage({ | ||
| </ResultGroup> | ||
| ) : null} | ||
| - | {results.executive.length > 0 ? ( | |
| - | <ResultGroup title="Federal Register" count={results.executive.length}> | |
| - | {results.executive.map((e) => ( | |
| + | {showExec && sortedResults.executive.length > 0 ? ( | |
| + | <ResultGroup title="Federal Register" count={sortedResults.executive.length}> | |
| + | {sortedResults.executive.map((e) => ( | |
| <div | ||
| key={e.id} | ||
| className="flex items-start justify-between gap-3 border-b border-line p-3 last:border-0" | ||
| @@ -139,9 +250,9 @@ export default function SearchPage({ | ||
| </ResultGroup> | ||
| ) : null} | ||
| - | {results.awards.length > 0 ? ( | |
| - | <ResultGroup title="Federal awards" count={results.awards.length}> | |
| - | {results.awards.map((a) => ( | |
| + | {showAwards && sortedResults.awards.length > 0 ? ( | |
| + | <ResultGroup title="Federal awards" count={sortedResults.awards.length}> | |
| + | {sortedResults.awards.map((a) => ( | |
| <div | ||
| key={a.id} | ||
| className="flex items-start justify-between gap-3 border-b border-line p-3 last:border-0" | ||
| @@ -156,9 +267,9 @@ export default function SearchPage({ | ||
| </ResultGroup> | ||
| ) : null} | ||
| - | {results.states.length > 0 ? ( | |
| - | <ResultGroup title="States" count={results.states.length}> | |
| - | {results.states.map((s) => ( | |
| + | {showStates && sortedResults.states.length > 0 ? ( | |
| + | <ResultGroup title="States" count={sortedResults.states.length}> | |
| + | {sortedResults.states.map((s) => ( | |
| <Link | ||
| key={s.code} | ||
| href={`/states/${s.code}`} | ||
| @@ -179,6 +290,36 @@ export default function SearchPage({ | ||
| ); | ||
| } | ||
| + | /** Build a search results href preserving q and updating type/sort. */ | |
| + | function buildHref(q: string, type: ResultType, sort: string): string { | |
| + | const params = new URLSearchParams(); | |
| + | if (q) params.set("q", q); | |
| + | if (type && type !== "all") params.set("type", type); | |
| + | if (sort) params.set("sort", sort); | |
| + | const qs = params.toString(); | |
| + | return `/search${qs ? `?${qs}` : ""}`; | |
| + | } | |
| + | ||
| + | /** Apply in-memory sort to search results. */ | |
| + | function applySortToResults(results: SearchResults, sort: string): SearchResults { | |
| + | if (!sort || sort === "relevance") return results; | |
| + | if (sort === "alpha") { | |
| + | return { | |
| + | ...results, | |
| + | bills: [...results.bills].sort((a, b) => a.title.localeCompare(b.title)), | |
| + | members: [...results.members].sort((a, b) => a.name.localeCompare(b.name)), | |
| + | executive: [...results.executive].sort((a, b) => a.title.localeCompare(b.title)), | |
| + | }; | |
| + | } | |
| + | if (sort === "amount") { | |
| + | return { | |
| + | ...results, | |
| + | awards: [...results.awards].sort((a, b) => b.amount - a.amount), | |
| + | }; | |
| + | } | |
| + | return results; | |
| + | } | |
| + | ||
| function ResultGroup({ | ||
| title, | ||
| count, |
| @@ -14,6 +14,7 @@ import { | ||
| SearchFilterBar, | ||
| FilterField, | ||
| FilterInput, | ||
| + | FilterSelect, | |
| FilterSubmit, | ||
| } from "@/components/search-filter-bar"; | ||
| import { CiteThis } from "@/components/cite-this"; | ||
| @@ -43,14 +44,18 @@ function apportionmentColor(status: string): string { | ||
| export default function SpendingPage({ | ||
| searchParams, | ||
| }: { | ||
| - | searchParams: { q?: string | string[] }; | |
| + | searchParams: { q?: string | string[]; sort?: string | string[] }; | |
| }) { | ||
| const q = asStr(searchParams.q); | ||
| - | const awards = getAwardsFiltered(q); | |
| + | const sortParam = asStr(searchParams.sort); | |
| + | const awards = getAwardsFiltered( | |
| + | q, | |
| + | sortParam === "date" ? "date" : sortParam === "agency" ? "agency" : undefined | |
| + | ); | |
| const fiscal = getFiscal(); | ||
| const apportionments = getApportionments(); | ||
| const totalAwards = awards.reduce((n, a) => n + a.amount, 0); | ||
| - | const isFiltered = !!q; | |
| + | const isFiltered = !!(q || sortParam); | |
| const withheld = apportionments.filter((a) => | ||
| (a.status || "").toLowerCase().startsWith("withhold") | ||
| @@ -90,6 +95,18 @@ export default function SpendingPage({ | ||
| aria-label="Search federal awards by agency, recipient, or program" | ||
| /> | ||
| </FilterField> | ||
| + | <FilterField label="Sort by"> | |
| + | <FilterSelect | |
| + | name="sort" | |
| + | defaultValue={sortParam ?? ""} | |
| + | aria-label="Sort awards" | |
| + | options={[ | |
| + | { value: "", label: "Largest amount first" }, | |
| + | { value: "date", label: "Most recent first" }, | |
| + | { value: "agency", label: "By agency (A-Z)" }, | |
| + | ]} | |
| + | /> | |
| + | </FilterField> | |
| <FilterSubmit /> | ||
| </SearchFilterBar> | ||
| </form> |
| @@ -10,6 +10,7 @@ import { PositionTag, positionColor } from "@/components/position-tag"; | ||
| import { PartyTag } from "@/components/party-tag"; | ||
| import { MemberAvatar } from "@/components/member-avatar"; | ||
| import { getVoteById, type VotePosition } from "@/lib/feed"; | ||
| + | import { resolveVoteSubject } from "@/lib/vote-subject"; | |
| import { formatDate } from "@/lib/format"; | ||
| import { CiteThis } from "@/components/cite-this"; | ||
| @@ -23,9 +24,10 @@ export function generateMetadata({ | ||
| const numeric = parseInt(params.id.replace(/^vote:/, ""), 10); | ||
| const vote = Number.isFinite(numeric) ? getVoteById(numeric) : null; | ||
| if (!vote) return { title: "Vote not found" }; | ||
| + | const subject = resolveVoteSubject(vote); | |
| return { | ||
| title: `${vote.chamber} vote · ${formatDate(vote.date)}`, | ||
| - | description: vote.result, | |
| + | description: subject.primary, | |
| }; | ||
| } | ||
| @@ -65,6 +67,9 @@ export default function VoteDetailPage({ | ||
| return m; | ||
| }, {}); | ||
| + | // Resolve what this vote was about using the full priority chain. | |
| + | const subject = resolveVoteSubject(vote); | |
| + | ||
| return ( | ||
| <div className="mx-auto w-full max-w-register px-4 py-10 sm:px-6"> | ||
| <Link | ||
| @@ -74,6 +79,51 @@ export default function VoteDetailPage({ | ||
| ← Votes | ||
| </Link> | ||
| + | {/* ────────────────────────────────────────────────────────── | |
| + | WHAT THIS VOTE WAS ABOUT - shown before the page header so | |
| + | the reader instantly understands the subject. | |
| + | Priority: linked bill → nomination → treaty → ref → question | |
| + | ────────────────────────────────────────────────────────── */} | |
| + | <div className="mt-4 mb-6 rounded-[3px] border border-line bg-surface p-5"> | |
| + | <p className="label mb-1.5 text-ash"> | |
| + | {subject.kind === "bill" | |
| + | ? "Bill voted on" | |
| + | : subject.kind === "nomination" | |
| + | ? "Presidential nomination" | |
| + | : subject.kind === "treaty" | |
| + | ? "Treaty ratification" | |
| + | : subject.kind === "ref" | |
| + | ? "Legislative document" | |
| + | : "Question voted on"} | |
| + | </p> | |
| + | ||
| + | {/* Primary subject line */} | |
| + | {subject.kind === "bill" && vote.billId != null ? ( | |
| + | <p className="font-mono text-[14px] leading-snug text-paper"> | |
| + | <Link | |
| + | href={`/bill/${vote.billId}`} | |
| + | className="focus-ring underline underline-offset-4 hover:text-paper" | |
| + | > | |
| + | {vote.billNumber} | |
| + | </Link> | |
| + | {vote.billTitle ? ( | |
| + | <span className="text-ash"> - {vote.billTitle}</span> | |
| + | ) : null} | |
| + | </p> | |
| + | ) : ( | |
| + | <p className="font-mono text-[14px] leading-snug text-paper"> | |
| + | {subject.primary} | |
| + | </p> | |
| + | )} | |
| + | ||
| + | {/* Secondary: question (if different from primary) */} | |
| + | {subject.secondary && subject.secondary !== vote.result ? ( | |
| + | <p className="mt-2 text-[13px] text-ash">{subject.secondary}</p> | |
| + | ) : vote.question && subject.kind === "bill" ? ( | |
| + | <p className="mt-2 text-[13px] text-ash">{vote.question}</p> | |
| + | ) : null} | |
| + | </div> | |
| + | ||
| <PageHeader | ||
| eyebrow={`${vote.chamber} · ${formatDate(vote.date)}${ | ||
| vote.rollNumber ? ` · Roll ${vote.rollNumber}` : "" | ||
| @@ -203,7 +253,7 @@ export default function VoteDetailPage({ | ||
| <p className="label mb-4 text-ash">Cite this record</p> | ||
| <CiteThis | ||
| kind="vote" | ||
| - | title={`${vote.chamber} Roll-Call Vote - ${vote.question || vote.result || "Recorded vote"} (${formatDate(vote.date)})`} | |
| + | title={`${vote.chamber} Roll-Call Vote - ${subject.primary} (${formatDate(vote.date)})`} | |
| permalink={`https://usdocket.org/votes/${voteUrlId}`} | ||
| sourceUrl={vote.sourceUrl} | ||
| /> |
| @@ -9,6 +9,7 @@ import { | ||
| getVoteCountFiltered, | ||
| type VoteFilter, | ||
| } from "@/lib/feed"; | ||
| + | import { resolveVoteSubject } from "@/lib/vote-subject"; | |
| import { formatDate } from "@/lib/format"; | ||
| import { | ||
| SearchFilterBar, | ||
| @@ -32,27 +33,49 @@ function asStr(v: string | string[] | undefined): string | undefined { | ||
| return v; | ||
| } | ||
| + | /** Truncate to max chars, appending ellipsis. */ | |
| + | function trunc(s: string | undefined, max: number): string { | |
| + | if (!s) return ""; | |
| + | return s.length <= max ? s : s.slice(0, max - 1) + "…"; | |
| + | } | |
| + | ||
| export default function VotesPage({ | ||
| searchParams, | ||
| }: { | ||
| searchParams: { | ||
| chamber?: string | string[]; | ||
| q?: string | string[]; | ||
| + | result?: string | string[]; | |
| + | sort?: string | string[]; | |
| }; | ||
| }) { | ||
| const chamber = asStr(searchParams.chamber); | ||
| const q = asStr(searchParams.q); | ||
| + | const resultParam = asStr(searchParams.result); | |
| + | const sort = asStr(searchParams.sort); | |
| + | ||
| const filter: VoteFilter = { | ||
| chamber: chamber || undefined, | ||
| q: q || undefined, | ||
| + | resultFilter: | |
| + | resultParam === "passed" | |
| + | ? "passed" | |
| + | : resultParam === "failed" | |
| + | ? "failed" | |
| + | : undefined, | |
| + | sort: | |
| + | sort === "chamber" | |
| + | ? "chamber" | |
| + | : sort === "result" | |
| + | ? "result" | |
| + | : undefined, | |
| }; | ||
| // Push filtering into SQL - only fetch the 100 we render. | ||
| const votes = getVotesFiltered(filter, 100, 0); | ||
| const totalFiltered = getVoteCountFiltered(filter); | ||
| - | const isFiltered = !!(chamber || q); | |
| + | const isFiltered = !!(chamber || q || resultParam || sort); | |
| - | // Determine clear href. | |
| const clearHref = "/votes"; | ||
| return ( | ||
| @@ -60,7 +83,7 @@ export default function VotesPage({ | ||
| <PageHeader | ||
| eyebrow="Roll-call register" | ||
| title="Votes, on the record" | ||
| - | lead="Each row is a recorded vote in the House or Senate - the question, the tally, and a link to the official roll call. Open any vote to see how every member voted." | |
| + | lead="Each row is a recorded vote in the House or Senate - what it was about, the tally, and a link to the official roll call. Open any vote to see how every member voted." | |
| /> | ||
| {/* Search/filter bar - server-side via GET form (fast on 35k+ rows). */} | ||
| @@ -76,7 +99,7 @@ export default function VotesPage({ | ||
| <FilterInput | ||
| name="q" | ||
| defaultValue={q ?? ""} | ||
| - | placeholder="Keyword, bill, question…" | |
| + | placeholder="Keyword, bill number, question…" | |
| aria-label="Search votes by keyword or question" | ||
| /> | ||
| </FilterField> | ||
| @@ -92,6 +115,30 @@ export default function VotesPage({ | ||
| ]} | ||
| /> | ||
| </FilterField> | ||
| + | <FilterField label="Result"> | |
| + | <FilterSelect | |
| + | name="result" | |
| + | defaultValue={resultParam ?? ""} | |
| + | aria-label="Filter by vote result" | |
| + | options={[ | |
| + | { value: "", label: "All results" }, | |
| + | { value: "passed", label: "Passed (Yea majority)" }, | |
| + | { value: "failed", label: "Failed (Nay majority)" }, | |
| + | ]} | |
| + | /> | |
| + | </FilterField> | |
| + | <FilterField label="Sort by"> | |
| + | <FilterSelect | |
| + | name="sort" | |
| + | defaultValue={sort ?? ""} | |
| + | aria-label="Sort votes" | |
| + | options={[ | |
| + | { value: "", label: "Newest first" }, | |
| + | { value: "chamber", label: "By chamber" }, | |
| + | { value: "result", label: "By result text" }, | |
| + | ]} | |
| + | /> | |
| + | </FilterField> | |
| <FilterSubmit /> | ||
| </SearchFilterBar> | ||
| </form> | ||
| @@ -112,7 +159,7 @@ export default function VotesPage({ | ||
| </p> | ||
| </Card> | ||
| ) : ( | ||
| - | <VotesTable votes={votes} /> | |
| + | <VotesTable votes={votes} trunc={trunc} /> | |
| )} | ||
| <p className="label mt-4 text-ash-2"> | ||
| Source: clerk.house.gov and senate.gov roll-call XML. Figures only - | ||
| @@ -135,28 +182,48 @@ export default function VotesPage({ | ||
| } | ||
| /** Recorded votes as a responsive table (>=md) / stacked cards (<md). */ | ||
| - | function VotesTable({ votes }: { votes: ReturnType<typeof getVotesFiltered> }) { | |
| + | function VotesTable({ | |
| + | votes, | |
| + | trunc, | |
| + | }: { | |
| + | votes: ReturnType<typeof getVotesFiltered>; | |
| + | trunc: (s: string | undefined, max: number) => string; | |
| + | }) { | |
| const columns: Column[] = [ | ||
| { key: "date", header: "Date" }, | ||
| { key: "chamber", header: "Chamber", hideOnCard: true }, | ||
| - | { key: "result", header: "Result", cardTitle: true }, | |
| + | { key: "what", header: "What was voted on", cardTitle: true }, | |
| { key: "tally", header: "Tally", align: "right" }, | ||
| { key: "record", header: "Record", align: "right" }, | ||
| ]; | ||
| const rows: Row[] = votes.map((v) => { | ||
| const variant = classifyStatus(v.result); | ||
| + | const subject = resolveVoteSubject(v); | |
| return { | ||
| key: v.id, | ||
| cells: { | ||
| date: <span className="tabular-nums text-ash">{formatDate(v.date)}</span>, | ||
| chamber: <span className="uppercase text-ash-2">{v.chamber}</span>, | ||
| - | result: ( | |
| - | <span className="text-paper"> | |
| + | what: ( | |
| + | <span className="block text-paper"> | |
| <Link | ||
| href={v.href ?? "#"} | ||
| className="focus-ring rounded-sm underline underline-offset-4 hover:text-paper" | ||
| > | ||
| - | {v.result || "Roll-call vote"} | |
| + | <span | |
| + | className={ | |
| + | subject.kind === "bill" | |
| + | ? "block font-mono text-[13px] text-paper" | |
| + | : "block text-paper" | |
| + | } | |
| + | > | |
| + | {trunc(subject.primary, 90)} | |
| + | </span> | |
| + | {subject.secondary ? ( | |
| + | <span className="mt-0.5 block text-[12px] text-ash-2"> | |
| + | {trunc(subject.secondary, 72)} | |
| + | </span> | |
| + | ) : null} | |
| </Link> | ||
| <span className="mt-1.5 block"> | ||
| <StatusPill variant={variant} dot> | ||
| @@ -176,7 +243,7 @@ function VotesTable({ votes }: { votes: ReturnType<typeof getVotesFiltered> }) { | ||
| }); | ||
| return ( | ||
| <DataTable | ||
| - | caption="Recorded votes: date, chamber, question or result, tally, and source." | |
| + | caption="Recorded votes: date, chamber, what was voted on, tally, and source." | |
| cardLabel="Recorded votes" | ||
| columns={columns} | ||
| rows={rows} |
| @@ -41,6 +41,7 @@ export function MembersBrowser({ | ||
| const [state, setState] = useState(""); | ||
| const [chamber, setChamber] = useState(""); | ||
| const [party, setParty] = useState(""); | ||
| + | const [sort, setSort] = useState(""); | |
| const [showAllHistorical, setShowAllHistorical] = useState(false); | ||
| const [allMembers, setAllMembers] = useState<MemberSummary[] | null>(null); | ||
| const [loadingAll, setLoadingAll] = useState(false); | ||
| @@ -82,7 +83,7 @@ export function MembersBrowser({ | ||
| const filtered = useMemo(() => { | ||
| const query = q.trim().toLowerCase(); | ||
| - | const anyFilterActive = !!(q || state || chamber || party); | |
| + | const anyFilterActive = !!(q || state || chamber || party || sort); | |
| return pool | ||
| .filter((m) => { | ||
| @@ -110,7 +111,17 @@ export function MembersBrowser({ | ||
| return true; | ||
| }) | ||
| .sort((a, b) => { | ||
| - | // Tier 1: current Congress | |
| + | // Named sort overrides tier logic (user is exploring explicitly). | |
| + | if (sort === "name") return a.name.localeCompare(b.name); | |
| + | if (sort === "state") { | |
| + | const sc = a.state.localeCompare(b.state); | |
| + | return sc !== 0 ? sc : a.name.localeCompare(b.name); | |
| + | } | |
| + | if (sort === "chamber") { | |
| + | const cc = a.chamber.localeCompare(b.chamber); | |
| + | return cc !== 0 ? cc : a.name.localeCompare(b.name); | |
| + | } | |
| + | // Default: Tier 1 = current Congress | |
| const aCurr = a.isCurrent ? 0 : 1; | ||
| const bCurr = b.isCurrent ? 0 : 1; | ||
| if (aCurr !== bCurr) return aCurr - bCurr; | ||
| @@ -121,9 +132,9 @@ export function MembersBrowser({ | ||
| // Within each tier: alphabetical by name | ||
| return a.name.localeCompare(b.name); | ||
| }); | ||
| - | }, [pool, q, state, chamber, party, showAllHistorical]); | |
| + | }, [pool, q, state, chamber, party, sort, showAllHistorical]); | |
| - | const anyFilterActive = !!(q || state || chamber || party); | |
| + | const anyFilterActive = !!(q || state || chamber || party || sort); | |
| return ( | ||
| <> | ||
| @@ -140,6 +151,7 @@ export function MembersBrowser({ | ||
| setState(""); | ||
| setChamber(""); | ||
| setParty(""); | ||
| + | setSort(""); | |
| } | ||
| : undefined | ||
| } | ||
| @@ -188,6 +200,19 @@ export function MembersBrowser({ | ||
| ]} | ||
| /> | ||
| </FilterField> | ||
| + | <FilterField label="Sort by"> | |
| + | <FilterSelect | |
| + | value={sort} | |
| + | onChange={setSort} | |
| + | aria-label="Sort members" | |
| + | options={[ | |
| + | { value: "", label: "Default (current first)" }, | |
| + | { value: "name", label: "Name (A-Z)" }, | |
| + | { value: "state", label: "State (A-Z)" }, | |
| + | { value: "chamber", label: "Chamber" }, | |
| + | ]} | |
| + | /> | |
| + | </FilterField> | |
| </SearchFilterBar> | ||
| {/* Historical toggle - only shown when no filter is active */} |
| @@ -92,6 +92,17 @@ export type Apportionment = { | ||
| export type Vote = { | ||
| id: string; | ||
| billId?: number | null; | ||
| + | /** Bill number (e.g. "H.R. 22") - populated when votes.bill_id is linked. */ | |
| + | billNumber?: string; | |
| + | /** Bill title - populated when votes.bill_id is linked. */ | |
| + | billTitle?: string; | |
| + | /** | |
| + | * Raw bill/doc reference from the vote record (e.g. "H.R. 6938", "PN123", | |
| + | * "TREATYDOC118-5"). Present when the vote references a document but the | |
| + | * bill is not yet in our DB. PN-prefix = Presidential Nomination; | |
| + | * TREATYDOC-prefix = Treaty document. Gracefully null for older records. | |
| + | */ | |
| + | billTextRef?: string; | |
| chamber: string; | ||
| date: string; | ||
| result: string; | ||
| @@ -605,30 +616,43 @@ export function getAwards(): Award[] { | ||
| ); | ||
| } | ||
| - | /** Awards filtered by keyword text (agency, recipient, or program). Client-safe size: already small. */ | |
| - | export function getAwardsFiltered(q?: string): Award[] { | |
| - | if (!q || !q.trim()) return getAwards(); | |
| + | /** Awards filtered by keyword text (agency, recipient, or program) with optional sort. */ | |
| + | export function getAwardsFiltered(q?: string, sort?: string): Award[] { | |
| + | const orderBy = | |
| + | sort === "date" ? "action_date DESC, amount DESC" : | |
| + | sort === "agency" ? "agency ASC, amount DESC" : | |
| + | "amount DESC"; // default: largest first | |
| + | const mapper = (r: Record<string, unknown>): Award => { | |
| + | const amount = num(r.amount); | |
| + | return { | |
| + | id: `award:${r.id}`, | |
| + | agency: str(r.agency), | |
| + | recipient: str(r.recipient), | |
| + | amount, | |
| + | amountText: formatUsd(amount), | |
| + | actionDate: str(r.action_date), | |
| + | program: str(r.program), | |
| + | sourceUrl: ensureSource(r.source_url), | |
| + | href: "/spending", | |
| + | }; | |
| + | }; | |
| + | if (!q || !q.trim()) { | |
| + | return safeAll<Award>( | |
| + | `SELECT id, agency, recipient, amount, action_date, program, source_url | |
| + | FROM awards | |
| + | ORDER BY ${orderBy}`, | |
| + | {}, | |
| + | mapper | |
| + | ); | |
| + | } | |
| const like = `%${q.trim().replace(/[%_\\]/g, (m) => `\\${m}`)}%`; | ||
| return safeAll<Award>( | ||
| `SELECT id, agency, recipient, amount, action_date, program, source_url | ||
| FROM awards | ||
| WHERE (agency LIKE :q ESCAPE '\\' OR recipient LIKE :q ESCAPE '\\' OR program LIKE :q ESCAPE '\\') | ||
| - | ORDER BY amount DESC`, | |
| + | ORDER BY ${orderBy}`, | |
| { q: like }, | ||
| - | (r) => { | |
| - | const amount = num(r.amount); | |
| - | return { | |
| - | id: `award:${r.id}`, | |
| - | agency: str(r.agency), | |
| - | recipient: str(r.recipient), | |
| - | amount, | |
| - | amountText: formatUsd(amount), | |
| - | actionDate: str(r.action_date), | |
| - | program: str(r.program), | |
| - | sourceUrl: ensureSource(r.source_url), | |
| - | href: "/spending", | |
| - | }; | |
| - | } | |
| + | mapper | |
| ); | ||
| } | ||
| @@ -690,67 +714,78 @@ export function getVoteCount(chamber?: string): number { | ||
| /** Filter shape for votes list - pushed into SQL for performance (35k+ rows). */ | ||
| export type VoteFilter = { | ||
| - | chamber?: string; // "senate" | "house" | |
| - | q?: string; // keyword search on result + question text | |
| + | chamber?: string; // "senate" | "house" | |
| + | q?: string; // keyword search on result + question text | |
| + | resultFilter?: "passed" | "failed"; // pass/fail filter (uses yea > nay) | |
| + | sort?: "newest" | "chamber" | "result"; // sort order (default: newest) | |
| }; | ||
| - | /** Build WHERE clause + params for vote filters. */ | |
| + | /** | |
| + | * Build WHERE clause + params for vote filters. | |
| + | * All column references use the `v` table alias so this WHERE can be used | |
| + | * alongside a LEFT JOIN bills b ON b.id = v.bill_id. | |
| + | */ | |
| function buildVotesWhere(filter: VoteFilter): { where: string; params: Record<string, unknown> } { | ||
| const clauses: string[] = []; | ||
| const params: Record<string, unknown> = {}; | ||
| if (filter.chamber) { | ||
| - | clauses.push("LOWER(chamber) = :ch"); | |
| + | clauses.push("LOWER(v.chamber) = :ch"); | |
| params.ch = filter.chamber.toLowerCase(); | ||
| } | ||
| if (filter.q) { | ||
| const like = `%${filter.q.trim().replace(/[%_\\]/g, (m) => `\\${m}`)}%`; | ||
| - | clauses.push("(result LIKE :q ESCAPE '\\' OR question LIKE :q ESCAPE '\\')"); | |
| + | clauses.push("(v.result LIKE :q ESCAPE '\\' OR v.question LIKE :q ESCAPE '\\')"); | |
| params.q = like; | ||
| } | ||
| + | if (filter.resultFilter === "passed") { | |
| + | clauses.push("v.yea > v.nay"); | |
| + | } else if (filter.resultFilter === "failed") { | |
| + | clauses.push("v.nay >= v.yea AND (v.yea + v.nay) > 0"); | |
| + | } | |
| const where = clauses.length ? " WHERE " + clauses.join(" AND ") : ""; | ||
| return { where, params }; | ||
| } | ||
| + | /** ORDER BY clause for vote list sort - never references non-ranking fields. */ | |
| + | function buildVotesOrder(sort?: VoteFilter["sort"]): string { | |
| + | switch (sort) { | |
| + | case "chamber": return "v.chamber, v.date DESC, v.id DESC"; | |
| + | case "result": return "v.result, v.date DESC, v.id DESC"; | |
| + | default: return "v.date DESC, v.id DESC"; | |
| + | } | |
| + | } | |
| + | ||
| /** Count votes matching the given filter - cheap SELECT COUNT(*). */ | ||
| export function getVoteCountFiltered(filter: VoteFilter = {}): number { | ||
| const { where, params } = buildVotesWhere(filter); | ||
| + | // Use `v` alias to match the WHERE clauses produced by buildVotesWhere. | |
| const row = safeGet<{ c: number }>( | ||
| - | `SELECT COUNT(*) AS c FROM votes${where}`, | |
| + | `SELECT COUNT(*) AS c FROM votes v${where}`, | |
| params | ||
| ); | ||
| return row?.c ?? 0; | ||
| } | ||
| - | /** Paginated votes list - only fetches what the page renders. */ | |
| + | /** Paginated votes list - delegates to getVotesFiltered with optional chamber. */ | |
| export function getVotes(limit = 100, offset = 0, chamber?: string): Vote[] { | ||
| - | if (chamber) { | |
| - | return safeAll<Vote>( | |
| - | `SELECT id, bill_id, chamber, date, yea, nay, present, result, source_url | |
| - | FROM votes | |
| - | WHERE LOWER(chamber) = :ch | |
| - | ORDER BY date DESC | |
| - | LIMIT :limit OFFSET :offset`, | |
| - | { ch: chamber.toLowerCase(), limit, offset }, | |
| - | (r) => toVote(r) | |
| - | ); | |
| - | } | |
| - | return safeAll<Vote>( | |
| - | `SELECT id, bill_id, chamber, date, yea, nay, present, result, source_url | |
| - | FROM votes | |
| - | ORDER BY date DESC | |
| - | LIMIT :limit OFFSET :offset`, | |
| - | { limit, offset }, | |
| - | (r) => toVote(r) | |
| - | ); | |
| + | return getVotesFiltered(chamber ? { chamber } : {}, limit, offset); | |
| } | ||
| - | /** Filtered, paginated votes list - pushes filtering into SQL (neutrality: date-ordered). */ | |
| + | /** | |
| + | * Filtered, paginated votes list. | |
| + | * LEFT JOINs bills so each row carries the linked bill's number + title | |
| + | * (gracefully null when votes.bill_id is not set - no bill is shown). | |
| + | */ | |
| export function getVotesFiltered(filter: VoteFilter = {}, limit = 100, offset = 0): Vote[] { | ||
| const { where, params } = buildVotesWhere(filter); | ||
| + | const orderBy = buildVotesOrder(filter.sort); | |
| return safeAll<Vote>( | ||
| - | `SELECT id, bill_id, chamber, date, yea, nay, present, result, source_url | |
| - | FROM votes${where} | |
| - | ORDER BY date DESC | |
| + | `SELECT v.id, v.bill_id, v.bill_text_ref, v.chamber, v.date, | |
| + | v.yea, v.nay, v.present, v.result, v.source_url, | |
| + | b.number AS bill_number, b.title AS bill_title | |
| + | FROM votes v | |
| + | LEFT JOIN bills b ON b.id = v.bill_id${where} | |
| + | ORDER BY ${orderBy} | |
| LIMIT :limit OFFSET :offset`, | ||
| { ...params, limit, offset }, | ||
| (r) => toVote(r) | ||
| @@ -764,9 +799,13 @@ export function getVotesFiltered(filter: VoteFilter = {}, limit = 100, offset = | ||
| */ | ||
| export function getVoteById(id: number): VoteDetail | null { | ||
| const r = safeGet<Record<string, unknown>>( | ||
| - | `SELECT id, bill_id, chamber, date, yea, nay, present, not_voting, | |
| - | roll_number, question, recorded, result, source_url | |
| - | FROM votes WHERE id = :id`, | |
| + | `SELECT v.id, v.bill_id, v.bill_text_ref, v.chamber, v.date, | |
| + | v.yea, v.nay, v.present, v.not_voting, v.roll_number, | |
| + | v.question, v.recorded, v.result, v.source_url, | |
| + | b.number AS bill_number, b.title AS bill_title | |
| + | FROM votes v | |
| + | LEFT JOIN bills b ON b.id = v.bill_id | |
| + | WHERE v.id = :id`, | |
| { id } | ||
| ); | ||
| if (!r) return null; | ||
| @@ -799,6 +838,9 @@ export function getVoteById(id: number): VoteDetail | null { | ||
| return { | ||
| id: `vote:${r.id}`, | ||
| billId: nullableNum(r.bill_id), | ||
| + | billNumber: str(r.bill_number) || undefined, | |
| + | billTitle: str(r.bill_title) || undefined, | |
| + | billTextRef: str(r.bill_text_ref) || undefined, | |
| chamber: str(r.chamber).toUpperCase() || "SENATE", | ||
| date: str(r.date), | ||
| result: str(r.result) || (yea > nay ? "Passed" : "Failed"), | ||
| @@ -824,6 +866,9 @@ function toVote(r: Record<string, unknown>): Vote { | ||
| return { | ||
| id: `vote:${r.id}`, | ||
| billId: nullableNum(r.bill_id), | ||
| + | billNumber: str(r.bill_number) || undefined, | |
| + | billTitle: str(r.bill_title) || undefined, | |
| + | billTextRef: str(r.bill_text_ref) || undefined, | |
| chamber: str(r.chamber).toUpperCase() || "SENATE", | ||
| date: str(r.date), | ||
| result: str(r.result) || (yea > nay ? "Passed" : "Failed"), | ||
| @@ -881,7 +926,7 @@ export type BillFilter = { | ||
| status?: string; | ||
| congress?: number; | ||
| policyArea?: string; | ||
| - | sort?: "updated" | "action"; | |
| + | sort?: "updated" | "action" | "congress" | "status"; | |
| }; | ||
| /** | ||
| @@ -939,9 +984,10 @@ export function getBillsCount(filter: BillFilter = {}): number { | ||
| export function getBills(filter: BillFilter = {}, limit = 100, offset = 0): BillListItem[] { | ||
| const { where, params } = buildBillsWhere(filter); | ||
| - | const orderBy = filter.sort === "action" | |
| - | ? "b.updated_at DESC" | |
| - | : "b.updated_at DESC"; | |
| + | const orderBy = | |
| + | filter.sort === "congress" ? "b.congress DESC, b.updated_at DESC" : | |
| + | filter.sort === "status" ? "b.status ASC, b.updated_at DESC" : | |
| + | "b.updated_at DESC"; | |
| return safeAll<BillListItem>( | ||
| `SELECT b.id, b.congress, b.number, b.title, b.status, b.policy_area, | ||
| b.updated_at, b.source_url |
| @@ -0,0 +1,97 @@ | ||
| + | /** | |
| + | * Resolve what a vote was about, in priority order: | |
| + | * 1. Linked bill (billNumber + billTitle) - votes.bill_id is set | |
| + | * 2. PN-prefix bill_text_ref → Presidential Nomination | |
| + | * 3. TREATYDOC-prefix bill_text_ref → Treaty document | |
| + | * 4. Other bill_text_ref (historical ref, no link in DB) + question | |
| + | * 5. Question text | |
| + | * 6. Result text (last resort - never leads with just "Passed") | |
| + | * | |
| + | * Returns { primary, secondary, kind } where `primary` is the bold/linked | |
| + | * identifier and `secondary` is additional context (question / result). | |
| + | * | |
| + | * Used by both the votes list page and the vote detail page so the label | |
| + | * logic is consistent across surfaces. | |
| + | */ | |
| + | ||
| + | export type VoteSubjectKind = | |
| + | | "bill" | |
| + | | "nomination" | |
| + | | "treaty" | |
| + | | "ref" | |
| + | | "question" | |
| + | | "result"; | |
| + | ||
| + | export type VoteSubject = { | |
| + | primary: string; | |
| + | secondary: string; | |
| + | kind: VoteSubjectKind; | |
| + | }; | |
| + | ||
| + | export type VoteSubjectInput = { | |
| + | billId?: number | null; | |
| + | billNumber?: string; | |
| + | billTitle?: string; | |
| + | billTextRef?: string; | |
| + | question?: string; | |
| + | result?: string; | |
| + | }; | |
| + | ||
| + | export function resolveVoteSubject(v: VoteSubjectInput): VoteSubject { | |
| + | const ref = v.billTextRef ?? ""; | |
| + | const question = v.question ?? ""; | |
| + | const result = v.result ?? ""; | |
| + | ||
| + | // 1. Linked bill (full bill record in DB) | |
| + | if (v.billNumber) { | |
| + | const title = v.billTitle ? ` - ${v.billTitle}` : ""; | |
| + | return { | |
| + | primary: `${v.billNumber}${title}`, | |
| + | secondary: question || result, | |
| + | kind: "bill", | |
| + | }; | |
| + | } | |
| + | ||
| + | // 2. Presidential nomination (Senate confirmation vote) | |
| + | if (ref.toUpperCase().startsWith("PN")) { | |
| + | return { | |
| + | primary: `Nomination - ${question || ref}`, | |
| + | secondary: result, | |
| + | kind: "nomination", | |
| + | }; | |
| + | } | |
| + | ||
| + | // 3. Treaty document ratification | |
| + | if (ref.toUpperCase().startsWith("TREATYDOC")) { | |
| + | return { | |
| + | primary: `Treaty document - ${question || ref}`, | |
| + | secondary: result, | |
| + | kind: "treaty", | |
| + | }; | |
| + | } | |
| + | ||
| + | // 4. Historical bill reference (bill in record but not yet in our DB) | |
| + | if (ref) { | |
| + | return { | |
| + | primary: ref, | |
| + | secondary: question || result, | |
| + | kind: "ref", | |
| + | }; | |
| + | } | |
| + | ||
| + | // 5. Question text (procedural question from the roll call) | |
| + | if (question) { | |
| + | return { | |
| + | primary: question, | |
| + | secondary: result, | |
| + | kind: "question", | |
| + | }; | |
| + | } | |
| + | ||
| + | // 6. Result text as last resort - still better than blank | |
| + | return { | |
| + | primary: result || "Roll-call vote", | |
| + | secondary: "", | |
| + | kind: "result", | |
| + | }; | |
| + | } |
| @@ -0,0 +1,394 @@ | ||
| + | """ | |
| + | link_votes.py - Retroactive vote-to-bill linkage for the Docket ingest service. | |
| + | ||
| + | Problem | |
| + | ------- | |
| + | When a vote is ingested, the bills table may not yet contain the referenced | |
| + | bill (votes are polled from House Clerk / Voteview; bills come from a separate | |
| + | govinfo/congress.gov poller). As a result, votes.bill_id is NULL even when | |
| + | bill_text_ref carries a recognizable bill reference. | |
| + | ||
| + | This module does a one-pass retroactive fix: for every vote row that has a | |
| + | non-NULL bill_text_ref and a NULL bill_id, it attempts to parse the reference, | |
| + | build the canonical external_id, look up the bills table, and set bill_id. | |
| + | ||
| + | Supported reference formats | |
| + | --------------------------- | |
| + | 1. House Clerk space-separated (from votes_house.py) | |
| + | "H R 22" → hr-22 | |
| + | "H RES 977" → hres-977 | |
| + | "H J RES 140" → hjres-140 | |
| + | "H CON RES 14" → hconres-14 | |
| + | "S 1" → s-1 | |
| + | "S RES 22" → sres-22 | |
| + | "S J RES 22" → sjres-22 | |
| + | "S CON RES 13" → sconres-13 | |
| + | ||
| + | 2. Voteview compact (from voteview.py bill_number CSV column) | |
| + | "HR22" → hr-22 | |
| + | "HRES977" → hres-977 | |
| + | "HJRES140" → hjres-140 | |
| + | "HCONRES14" → hconres-14 | |
| + | "S1" → s-1 | |
| + | "SRES22" → sres-22 | |
| + | "SJRES22" → sjres-22 | |
| + | "SCONRES13" → sconres-13 | |
| + | ||
| + | Unresolvable references | |
| + | ----------------------- | |
| + | Presidential Nomination refs (PN\d+) are Senate confirmation votes - they have | |
| + | no corresponding bill row, so they are correctly left with bill_id = NULL. | |
| + | Votes with congress = NULL cannot be disambiguated and are also skipped. | |
| + | Historical votes (congress 101-118) cannot be linked until bills for those | |
| + | congresses are ingested into the bills table. | |
| + | ||
| + | Idempotent | |
| + | ---------- | |
| + | Only rows with bill_id IS NULL are touched. Re-running after new bills are | |
| + | ingested will pick up any newly matchable votes without double-processing. | |
| + | ||
| + | Usage | |
| + | ----- | |
| + | # Via module (preferred - picks up DOCKET_DB_PATH env): | |
| + | python -m ingest.link_votes | |
| + | ||
| + | # With an explicit DB path: | |
| + | python -m ingest.link_votes --db /opt/docket/data/docket.db | |
| + | ||
| + | # Dry-run (prints stats without committing): | |
| + | python -m ingest.link_votes --dry-run | |
| + | """ | |
| + | ||
| + | from __future__ import annotations | |
| + | ||
| + | import argparse | |
| + | import logging | |
| + | import re | |
| + | import sqlite3 | |
| + | from typing import Optional | |
| + | ||
| + | from ingest.db import get_connection | |
| + | ||
| + | logger = logging.getLogger(__name__) | |
| + | ||
| + | # ─── Bill reference parsers ─────────────────────────────────────────────────── | |
| + | ||
| + | # Longest prefix first to avoid partial matches (e.g. "S" must not eat "S RES"). | |
| + | # Each entry: (prefix_or_regex, canonical_bill_type) | |
| + | ||
| + | _SPACE_PATTERNS: list[tuple[re.Pattern, str]] = [ | |
| + | (re.compile(r"^H\s+J\s+RES\s+(\d+)$"), "hjres"), | |
| + | (re.compile(r"^H\s+CON\s+RES\s+(\d+)$"), "hconres"), | |
| + | (re.compile(r"^H\s+RES\s+(\d+)$"), "hres"), | |
| + | (re.compile(r"^H\s+R\s+(\d+)$"), "hr"), | |
| + | (re.compile(r"^S\s+J\s+RES\s+(\d+)$"), "sjres"), | |
| + | (re.compile(r"^S\s+CON\s+RES\s+(\d+)$"), "sconres"), | |
| + | (re.compile(r"^S\s+RES\s+(\d+)$"), "sres"), | |
| + | (re.compile(r"^S\s+(\d+)$"), "s"), | |
| + | ] | |
| + | ||
| + | _COMPACT_PATTERNS: list[tuple[re.Pattern, str]] = [ | |
| + | (re.compile(r"^SCONRES(\d+)$"), "sconres"), | |
| + | (re.compile(r"^SJRES(\d+)$"), "sjres"), | |
| + | (re.compile(r"^HCONRES(\d+)$"), "hconres"), | |
| + | (re.compile(r"^HJRES(\d+)$"), "hjres"), | |
| + | (re.compile(r"^SRES(\d+)$"), "sres"), | |
| + | (re.compile(r"^HRES(\d+)$"), "hres"), | |
| + | (re.compile(r"^HR(\d+)$"), "hr"), | |
| + | (re.compile(r"^S(\d+)$"), "s"), | |
| + | ] | |
| + | ||
| + | # Presidential Nomination refs - Senate confirmation votes, no bill linkage. | |
| + | _PN_PATTERN = re.compile(r"^PN\d+", re.IGNORECASE) | |
| + | ||
| + | ||
| + | def parse_bill_ref(raw: str) -> tuple[Optional[str], Optional[str]]: | |
| + | """ | |
| + | Parse a raw bill reference string into (bill_type, number). | |
| + | ||
| + | Returns (None, None) if the reference is not recognized as a bill. | |
| + | ||
| + | Parameters | |
| + | ---------- | |
| + | raw : raw bill_text_ref value from the votes table | |
| + | ||
| + | Returns | |
| + | ------- | |
| + | (bill_type, number) e.g. ("hr", "22") or (None, None) | |
| + | """ | |
| + | if not raw: | |
| + | return None, None | |
| + | ||
| + | s = raw.strip().upper() | |
| + | ||
| + | # Skip Presidential Nominations - they have no bill entry. | |
| + | if _PN_PATTERN.match(s): | |
| + | return None, None | |
| + | ||
| + | # Try space-separated format first (House Clerk XML). | |
| + | for pat, btype in _SPACE_PATTERNS: | |
| + | m = pat.match(s) | |
| + | if m: | |
| + | return btype, m.group(1) | |
| + | ||
| + | # Try compact format (Voteview bill_number CSV column). | |
| + | for pat, btype in _COMPACT_PATTERNS: | |
| + | m = pat.match(s) | |
| + | if m: | |
| + | return btype, m.group(1) | |
| + | ||
| + | return None, None | |
| + | ||
| + | ||
| + | def build_external_id(congress: int, bill_type: str, number: str) -> str: | |
| + | """Build the canonical external_id used by the bills table.""" | |
| + | return f"{congress}-{bill_type}-{number}" | |
| + | ||
| + | ||
| + | # ─── Core linkage ───────────────────────────────────────────────────────────── | |
| + | ||
| + | def link_votes( | |
| + | conn: sqlite3.Connection, | |
| + | dry_run: bool = False, | |
| + | ) -> dict: | |
| + | """ | |
| + | Scan votes for unlinked rows and attempt bill_id resolution. | |
| + | ||
| + | Parameters | |
| + | ---------- | |
| + | conn : open SQLite connection with row_factory = sqlite3.Row | |
| + | dry_run : if True, compute matches but do not UPDATE or COMMIT | |
| + | ||
| + | Returns | |
| + | ------- | |
| + | dict with keys: | |
| + | before_linked - votes.bill_id NOT NULL before this run | |
| + | candidate_rows - votes with bill_text_ref and bill_id NULL | |
| + | matched - rows successfully linked | |
| + | skipped_no_congress - rows skipped because congress is NULL | |
| + | skipped_pn - rows skipped because ref is a PN nomination | |
| + | skipped_parse_fail - refs that could not be parsed (not PN) | |
| + | skipped_no_bill - parsed ok but no matching bill in the DB | |
| + | after_linked - votes.bill_id NOT NULL after this run | |
| + | examples - up to 5 dict with {vote_id, question, | |
| + | bill_text_ref, external_id, bill_title} | |
| + | """ | |
| + | before_linked = conn.execute( | |
| + | "SELECT COUNT(*) FROM votes WHERE bill_id IS NOT NULL" | |
| + | ).fetchone()[0] | |
| + | ||
| + | # Fetch all candidate votes in one pass. | |
| + | candidate_rows = conn.execute( | |
| + | """ | |
| + | SELECT id, bill_text_ref, congress, question | |
| + | FROM votes | |
| + | WHERE bill_id IS NULL | |
| + | AND bill_text_ref IS NOT NULL | |
| + | AND bill_text_ref != '' | |
| + | ORDER BY congress DESC, id | |
| + | """ | |
| + | ).fetchall() | |
| + | ||
| + | total_candidates = len(candidate_rows) | |
| + | ||
| + | skipped_no_congress = 0 | |
| + | skipped_pn = 0 | |
| + | skipped_parse_fail = 0 | |
| + | skipped_no_bill = 0 | |
| + | matched = 0 | |
| + | examples: list[dict] = [] | |
| + | ||
| + | # Pre-build an external_id → bills.id lookup for all bills that have a | |
| + | # congress in the range of our votes. This avoids thousands of individual | |
| + | # SELECT calls; trade-off is a modest dict in memory (~200k bills max). | |
| + | bill_lookup: dict[str, int] = {} | |
| + | for row in conn.execute("SELECT id, external_id FROM bills WHERE external_id IS NOT NULL"): | |
| + | bill_lookup[row["external_id"]] = row["id"] | |
| + | ||
| + | logger.info( | |
| + | "link_votes: %d bills loaded into lookup, %d candidate votes to process", | |
| + | len(bill_lookup), total_candidates, | |
| + | ) | |
| + | ||
| + | updates: list[tuple[int, int]] = [] # (bill_id, vote_id) | |
| + | ||
| + | for vote in candidate_rows: | |
| + | vote_id = vote["id"] | |
| + | raw_ref = vote["bill_text_ref"] | |
| + | congress = vote["congress"] | |
| + | question = vote["question"] or "" | |
| + | ||
| + | if congress is None: | |
| + | skipped_no_congress += 1 | |
| + | continue | |
| + | ||
| + | # parse_bill_ref returns (None, None) for PN refs and unknown formats. | |
| + | raw_upper = (raw_ref or "").strip().upper() | |
| + | if _PN_PATTERN.match(raw_upper): | |
| + | skipped_pn += 1 | |
| + | continue | |
| + | ||
| + | bill_type, number = parse_bill_ref(raw_ref) | |
| + | if bill_type is None: | |
| + | skipped_parse_fail += 1 | |
| + | logger.debug( | |
| + | "link_votes: could not parse ref %r (vote_id=%d congress=%d q=%r)", | |
| + | raw_ref, vote_id, congress, question[:60], | |
| + | ) | |
| + | continue | |
| + | ||
| + | ext_id = build_external_id(congress, bill_type, number) | |
| + | bill_id = bill_lookup.get(ext_id) | |
| + | ||
| + | if bill_id is None: | |
| + | skipped_no_bill += 1 | |
| + | logger.debug( | |
| + | "link_votes: no bill for %s (vote_id=%d q=%r)", | |
| + | ext_id, vote_id, question[:60], | |
| + | ) | |
| + | continue | |
| + | ||
| + | updates.append((bill_id, vote_id)) | |
| + | matched += 1 | |
| + | ||
| + | if len(examples) < 5: | |
| + | # Fetch the bill title for the example report. | |
| + | bill_row = conn.execute( | |
| + | "SELECT number, title FROM bills WHERE id=?", (bill_id,) | |
| + | ).fetchone() | |
| + | bill_label = ( | |
| + | f"{bill_row['number']} - {bill_row['title']}" | |
| + | if bill_row and bill_row["title"] | |
| + | else ext_id | |
| + | ) | |
| + | examples.append( | |
| + | { | |
| + | "vote_id": vote_id, | |
| + | "question": question, | |
| + | "bill_text_ref": raw_ref, | |
| + | "external_id": ext_id, | |
| + | "bill_label": bill_label, | |
| + | } | |
| + | ) | |
| + | ||
| + | if not dry_run and updates: | |
| + | conn.executemany( | |
| + | "UPDATE votes SET bill_id = ? WHERE id = ?", | |
| + | updates, | |
| + | ) | |
| + | conn.commit() | |
| + | logger.info("link_votes: committed %d bill_id updates", len(updates)) | |
| + | elif dry_run: | |
| + | logger.info("link_votes: dry-run - %d updates NOT committed", len(updates)) | |
| + | ||
| + | after_linked = conn.execute( | |
| + | "SELECT COUNT(*) FROM votes WHERE bill_id IS NOT NULL" | |
| + | ).fetchone()[0] | |
| + | ||
| + | return { | |
| + | "before_linked": before_linked, | |
| + | "candidate_rows": total_candidates, | |
| + | "matched": matched, | |
| + | "skipped_no_congress": skipped_no_congress, | |
| + | "skipped_pn": skipped_pn, | |
| + | "skipped_parse_fail": skipped_parse_fail, | |
| + | "skipped_no_bill": skipped_no_bill, | |
| + | "after_linked": after_linked, | |
| + | "examples": examples, | |
| + | } | |
| + | ||
| + | ||
| + | # ─── Public entry point ─────────────────────────────────────────────────────── | |
| + | ||
| + | def run(db_path: Optional[str] = None, dry_run: bool = False) -> dict: | |
| + | """ | |
| + | Open the DB, run vote-to-bill linkage, return results. | |
| + | ||
| + | Parameters | |
| + | ---------- | |
| + | db_path : path override (falls back to DOCKET_DB_PATH env or default) | |
| + | dry_run : if True, show what would be linked without writing | |
| + | ||
| + | Returns | |
| + | ------- | |
| + | Result dict from link_votes(). | |
| + | """ | |
| + | conn = get_connection(db_path) if db_path else get_connection() | |
| + | try: | |
| + | result = link_votes(conn, dry_run=dry_run) | |
| + | finally: | |
| + | conn.close() | |
| + | return result | |
| + | ||
| + | ||
| + | # ─── CLI ────────────────────────────────────────────────────────────────────── | |
| + | ||
| + | def _print_report(result: dict) -> None: | |
| + | total = result["before_linked"] + result["candidate_rows"] | |
| + | match_rate = ( | |
| + | result["matched"] / result["candidate_rows"] * 100 | |
| + | if result["candidate_rows"] | |
| + | else 0.0 | |
| + | ) | |
| + | ||
| + | print("\n=== Docket vote-to-bill linkage report ===\n") | |
| + | print(f" Votes already linked (before): {result['before_linked']}") | |
| + | print(f" Candidate votes (bill_id NULL): {result['candidate_rows']}") | |
| + | print(f" Matched and linked: {result['matched']}") | |
| + | print(f" Match rate (of candidates): {match_rate:.1f}%") | |
| + | print(f" Skipped - no congress on vote: {result['skipped_no_congress']}") | |
| + | print(f" Skipped - PN nomination ref: {result['skipped_pn']}") | |
| + | print(f" Skipped - ref unparseable: {result['skipped_parse_fail']}") | |
| + | print(f" Skipped - bill not in DB: {result['skipped_no_bill']}") | |
| + | print(f" Votes linked after run: {result['after_linked']}") | |
| + | ||
| + | if result["examples"]: | |
| + | print("\n--- Example linkages ---") | |
| + | for i, ex in enumerate(result["examples"], 1): | |
| + | print(f"\n [{i}] vote_id={ex['vote_id']}") | |
| + | print(f" question: {ex['question']}") | |
| + | print(f" ref: {ex['bill_text_ref']}") | |
| + | print(f" → bill: {ex['bill_label']}") | |
| + | ||
| + | print("\n--- Notes ---") | |
| + | print( | |
| + | " Presidential Nomination refs (PN*) are Senate confirmation votes " | |
| + | "and have no bill entry - skipped correctly." | |
| + | ) | |
| + | print( | |
| + | " Historical votes (congress 101-118) cannot be linked until bills " | |
| + | "for those congresses are ingested. Run the congress/govinfo poller " | |
| + | "for those congresses first, then re-run this module." | |
| + | ) | |
| + | print( | |
| + | " Forward-looking: votes_house.py and voteview.py now attempt inline " | |
| + | "linkage at ingest time. Run this module again after each new bill " | |
| + | "import to catch any new matches." | |
| + | ) | |
| + | ||
| + | ||
| + | def main() -> None: | |
| + | logging.basicConfig( | |
| + | level=logging.INFO, | |
| + | format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", | |
| + | ) | |
| + | parser = argparse.ArgumentParser( | |
| + | description="Docket - retroactive vote-to-bill linkage via bill_text_ref." | |
| + | ) | |
| + | parser.add_argument("--db", help="Override DB path.") | |
| + | parser.add_argument( | |
| + | "--dry-run", action="store_true", | |
| + | help="Show what would be linked without committing changes.", | |
| + | ) | |
| + | parser.add_argument( | |
| + | "--once", action="store_true", | |
| + | help="Run once and exit (required for pipeline runner compatibility).", | |
| + | ) | |
| + | args = parser.parse_args() | |
| + | ||
| + | result = run(db_path=args.db, dry_run=args.dry_run) | |
| + | _print_report(result) | |
| + | ||
| + | ||
| + | if __name__ == "__main__": | |
| + | main() |
| @@ -48,6 +48,7 @@ PIPELINE = [ | ||
| ("votes_senate", "ingest.pollers.votes_senate", []), | ||
| ("votes_house", "ingest.pollers.votes_house", []), | ||
| ("votes_enrich", "ingest.pollers.votes_enrich", []), | ||
| + | ("link_votes", "ingest.link_votes", []), | |
| ("crec", "ingest.pollers.crec", []), | ||
| ("meetings", "ingest.pollers.meetings", []), | ||
| ("lobbying", "ingest.pollers.lobbying", []), |