● Zion Boggan
repos/Docket/commits/8899de6
zionboggan.com ↗
Code Commits Tags

Polish public launch: README, screenshots, infra scrub

- Rewrote README with logo, 2x2 screenshot grid, neutrality statement,
  data sources table, architecture diagram, and full feature/roadmap sections
- Added docs/screenshots/ (home dark, home light, spending, search)
  and docs/logo.svg for use in README and future docs
- Removed or genericized internal infrastructure references across all
  tracked docs: CT247/CT215, Proxmox pct-exec SSH commands, raw LAN IPs,
  homelab-specific paths (/shared/projects, CIFS notes), and model-name
  references (mistral-nemo, Ollama/3060) in public-facing docs
- Replaced em-dashes in all user-visible app text, page metadata,
  and component copy with plain hyphens; null-display em-dashes kept
- Fixed systemd unit Documentation= to point to public GitHub URL
8899de6   Zion Boggan committed on Jun 27, 2026 (1 month ago)
BILL_SUMMARY_ARCHIVE_NOTES.md +150 -0
@@ -0,0 +1,150 @@
+## Bill Summaries & PDF Archive - Implementation Notes
+
+### What was built
+
+Two new ingest modules added to `services/ingest/ingest/`:
+
+| Module | Purpose |
+|---|---|
+| `bill_summaries.py` | Backfills official CRS summaries onto `bills` table |
+| `archive_bills.py` | Downloads GovInfo bill PDFs and uploads to R2 |
+
+Both are registered in `run.py` pipeline (after `categorize` / `calendar`).
+
+---
+
+### Schema changes (additive, ALTER TABLE safe)
+
+Four new columns on `bills`:
+
+```sql
+ALTER TABLE bills ADD COLUMN summary TEXT;
+ALTER TABLE bills ADD COLUMN summary_source TEXT;
+ALTER TABLE bills ADD COLUMN pdf_url TEXT;
+ALTER TABLE bills ADD COLUMN archive_pdf_url TEXT;
+```
+
+Each module calls `_ensure_columns()` on startup (catches OperationalError if already exists).
+
+---
+
+### bill_summaries.py - how it works
+
+**Source:** GovInfo BILLSTATUS bulk XML (keyless, no API key needed).
+
+URL pattern:
+```
+https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/BILLSTATUS-{congress}{type}{number}.xml
+```
+
+The BILLSTATUS XML already contains `<summaries><summary><text>` (CRS text). The module:
+1. Queries DB for `govinfo_billstatus` bills missing `summary` (current congress)
+2. Re-fetches their BILLSTATUS XML
+3. Picks the item with the latest `<updateDate>` from `<summaries>`
+4. HTML-unescapes + strips tags from `<text>` (text is HTML-encoded in the source)
+5. Stores verbatim (public-domain CRS text) with `summary_source = "CRS via GovInfo"`
+
+**Counts (test run, 119th congress, hr+s types, limit 30):**
+- 30 summaries stored in ~20s
+- 17,149 congress-119 bills identified as needing summaries at test time
+
+**Sample output (H.R. 160 - Restoring Faith in Elections Act):**
+> "Restoring Faith in Elections Act - This bill establishes certain standards for voting, including voting by mail, in federal elections. It also establishes certain requirements for voter registration and maintenance of official lists of eligible voters..."
+
+**Default run settings:** `--congress 119 --limit 500 --delay 0.1` (500 summaries per pipeline run at ~0.1s/request = ~60s per run).
+
+---
+
+### archive_bills.py - how it works
+
+**Source:** GovInfo BILLS collection PDFs (keyless).
+
+URL pattern:
+```
+https://www.govinfo.gov/content/pkg/{pkg_id}/pdf/{pkg_id}.pdf
+```
+
+The `pkg_id` comes from `bill_versions.govinfo_package_id` (e.g. `BILLS-119hr160ih`).
+
+**Version priority** (most authoritative wins per bill):
+`enr > eh > es > ph > ps > rh > rs > ih > is > (other)`
+
+**R2 key:** `bills/{congress}/{pkg_id}.pdf`
+**Public URL:** `https://archive.usdocket.org/bills/{congress}/{pkg_id}.pdf`
+
+The module:
+1. Queries DB for bills with `bill_versions` but no `archive_pdf_url`
+2. Selects best version per bill (priority order above)
+3. Checks `r2.exists()` - skips if already in bucket
+4. Downloads PDF (verified `Content-Type: application/pdf` + minimum size check)
+5. Uploads via `r2.put_object()` with `Content-Type: application/pdf`
+6. Stores `pdf_url` (official GovInfo) + `archive_pdf_url` (our R2 copy)
+
+**Counts (test run, 119th congress, hr type, limit 5×2):**
+- 10 PDFs archived across 2 runs
+- 0 failures, 0 re-uploads (DB-level idempotency via `archive_pdf_url IS NULL` filter)
+
+**Verified working archive URL:**
+```
+https://archive.usdocket.org/bills/119/BILLS-119hr160ih.pdf
+```
+→ HTTP 200, `Content-Type: application/pdf`, 325,931 bytes
+
+**Default run settings:** `--congress 119 --limit 200 --delay 0.5` (polite; ~100s per run).
+
+---
+
+### Idempotency
+
+- **Summaries:** `WHERE summary IS NULL OR summary = ''` in SQL - re-runs skip filled rows.
+- **Archive:** `WHERE archive_pdf_url IS NULL OR archive_pdf_url = ''` in SQL + `r2.exists()` guard - safe to re-run.
+
+---
+
+### Pipeline wiring (run.py)
+
+```python
+("bill_summaries", "ingest.bill_summaries", []),
+("archive_bills", "ingest.archive_bills", []),
+```
+
+Both run after `categorize` / `calendar` steps. Both accept `--once` CLI flag.
+
+---
+
+### CSP hosts to add (frontend, separate pass)
+
+The frontend will need to show summaries and embed/link PDFs. Two hosts to add to `next.config.js`:
+
+**For embedded PDFs (`<iframe>` or `<embed>` pointing to archive.usdocket.org):**
+
+In the CSP `frame-src` directive (currently absent - add it):
+```
+frame-src 'none' https://archive.usdocket.org
+```
+Or if PDFs are shown via `<object>`:
+```
+object-src 'none' → object-src https://archive.usdocket.org
+```
+
+**For direct PDF links (no embed - just `<a href>`), no CSP change needed.**
+
+Current `next.config.js` has `"object-src 'none'"` and no `frame-src`. If the frontend embeds PDFs via `<iframe>` or `<object>`, those directives need `archive.usdocket.org` added. If PDFs are linked (new tab), nothing to change. Do not edit `next.config.js` in this pass - flag it for the frontend developer.
+
+---
+
+### Manual run commands
+
+**Summaries:**
+```bash
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.bill_summaries --once
+```
+
+**Archive PDFs:**
+```bash
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.archive_bills --once
+```
CYCLOPS_REPORT.md +1 -1
@@ -1,6 +1,6 @@
# Cyclops / Flywheel Security Assessment - Docket
-**Target:** http://REDACTED-IP:3000 (pre-launch build)
+**Target:** http://localhost:3000 (pre-launch build)
**Date:** 2026-06-26
**Scope:** Full web + `/api/*` assessment (authorized owner test)
**Method:** White-box (source at `/shared/projects/docket`) + live black-box PoC via Flywheel MCP scanner and raw HTTP. Every finding below was verified against the running target - no assertion without a request/response.
LATENCY_NOTES.md +4 -4
@@ -1,7 +1,7 @@
# Docket - Navigation Latency Audit & Fix Notes
Session: 2026-06-27. All timings local (`curl -w "%{time_total}"` against
-`http://localhost:3000` on CT247). Cloudflare Tunnel overhead is on top of these.
+`http://localhost:3000` on the server). Cloudflare Tunnel overhead is on top of these.
---
@@ -13,7 +13,7 @@ Every `/api/*` GET response had zero `Cache-Control` header before this session.
Cloudflare's default for headerless JSON responses is "bypass cache" - meaning
every client-side API call (register-hero 15-second poll, MembersBrowser
"show historical" paginated fetches, MemberVoteYear lazy year panels) made a
-full round-trip over the CF Tunnel to the CT247 origin.
+full round-trip over the CF Tunnel to the the server origin.
Fix: added `Cache-Control: public, s-maxage=300, stale-while-revalidate=600`
to all `/api/*` routes via `next.config.js` headers(). Routes that already set
@@ -84,7 +84,7 @@ QA after rebuild: **CLEAN** - 17 routes, no overflow, no hydration errors.
---
-## Before/After Timings (local, CT247 localhost)
+## Before/After Timings (local, the server localhost)
These are origin times (no Cloudflare). The "win" from Cloudflare edge caching
shows up for end-users, not on localhost.
@@ -166,7 +166,7 @@ curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/rulesets" \
## Summary for the Overseer
-Code changes are deployed on CT247 at `/opt/docket`. QA passes.
+Code changes are deployed on the server at `/opt/docket`. QA passes.
The single remaining action for the overseer is the Cloudflare cache rule above,
which unlocks edge caching of HTML pages and makes the existing `s-maxage=300`
headers actually take effect for end-users.
PERF2_NOTES.md +1 -1
@@ -14,7 +14,7 @@ Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate
```
Cloudflare honors that and returns `cf-cache-status: BYPASS`. Every visit then
-crossed the home Cloudflare Tunnel to the CT247 origin (~0.5-0.8s) instead of
+crossed the home Cloudflare Tunnel to the the server origin (~0.5-0.8s) instead of
serving from the CF edge (~50-150ms). The `/api/*` routes were already
cacheable (`public, s-maxage=300`).
PERF_NOTES.md +1 -1
@@ -2,7 +2,7 @@
Audit and fix session conducted 2026-06-26/27. All timings are server-side
response time measured with `curl -o /dev/null -w "%{time_total}"` against
-CT247's local Next.js listener (`http://localhost:3000`). "First hit" is a
+the server's local Next.js listener (`http://localhost:3000`). "First hit" is a
cold request; "second hit" is a repeat within the same 5-minute TTL window.
---
PHOTOS_NOTES.md +2 -2
@@ -87,8 +87,8 @@ python -m ingest.pollers.member_photos --once
## Test command
```bash
-# On CT247 (live DB):
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc \
+# On the server (live DB):
+# Run on deployment host: \
'cd /shared/projects/docket/services/ingest && \
DOCKET_DB_PATH=/opt/docket/data/docket.db \
/opt/docket/services/ingest/.venv/bin/python \
PLAN.md +5 -5
@@ -162,14 +162,14 @@ We **inherit TreeTrace's system architecture** (dark-first, forensic/record aest
---
-## 7. Tech Stack & Deployment (homelab-aware)
+## 7. Tech Stack & Deployment
- **Frontend:** Next.js (multi-page functional site, matches TreeTrace pattern), Tailwind with the token set above, dark-first. Mobile-first PWA so push works without app-store friction at MVP.
- **Backend/ingest:** Python or Node workers polling the four APIs on cron; a change-detection table keyed by `updateDate`; Postgres for bills/actions/awards; a normalized "event" table that powers feed + alerts.
-- **Summaries:** extraction-template pipeline. Can run **locally on the 3060 (Ollama)** for cost-free draft extraction, with a Polarization-Index QA gate before publish - but **prefer deterministic field extraction over generation** wherever possible.
-- **Push:** ntfy (already in the homelab) for MVP; web-push for PWA.
-- **Hosting:** Cloudflare Pages for the static/edge front (same as zionboggan.com / treetrace.dev), workers/CT for the ingest service. Could live on a CT in the cluster.
-- **Costs:** all primary data is free; main cost is compute for ingest + summary QA. Fits the "no paid services until first payout" rule.
+- **Summaries:** extraction-template pipeline. Can run locally with an open-weight LLM for cost-free draft extraction, with a Polarization-Index QA gate before publish - but **prefer deterministic field extraction over generation** wherever possible.
+- **Push:** ntfy for MVP; web-push for PWA.
+- **Hosting:** Cloudflare Pages for the static/edge front, self-hosted workers for the ingest service.
+- **Costs:** all primary data is free; main cost is compute for ingest + summary QA.
---
README.md +223 -21
@@ -1,30 +1,232 @@
-# Docket
+<img src="docs/logo.svg" alt="Docket" width="220" height="50">
-**The unfiltered public record of what government does - every bill, law, amendment, and vote, including what gets slipped in - made legible to any citizen.**
+**The unfiltered public record of what government does.**
-Docket is a neutral, non-partisan transparency resource. It collects, aggregates, and makes searchable the *complete* official record of US legislative activity - federal and granular state-level - and surfaces the parts that are technically public but buried where no ordinary person can find them: amendments and riders inserted by either side, voice votes, committee markups, conference-report changes, procedural maneuvers.
+[![License](https://img.shields.io/badge/License-Apache_2.0-E0A53B.svg)](LICENSE)
+[![Next.js](https://img.shields.io/badge/Next.js-14-black.svg)](https://nextjs.org)
+[![Python](https://img.shields.io/badge/Python-3.11+-3776AB.svg)](https://python.org)
+[![Status](https://img.shields.io/badge/Status-Active_Development-E0A53B.svg)](#)
-It serves two needs at once:
-1. **Awareness** - a calm, neutral feed so you catch what's happening (*"this was happening and I had no idea"*).
-2. **Lookup** - a resource where anyone, including an 80-year-old, can go dig up exactly what their representatives did on any bill, in any state, on a granular level - in plain English, with zero spin, every fact sourced to an official record.
+---
-Spending is one lens, not the point. The point is **full-spectrum, unfiltered visibility** into what government and its representatives actually do.
+Docket tracks federal and state legislative activity and makes it searchable for any citizen - bills, votes, amendments, riders, spending awards, executive actions, and the closed-door procedural moves that rarely make the news.
-## The one rule that defines everything
-**Zero political lean.** If Docket reads as left- or right-leaning, it has failed. Neutrality is not a tone we aim for - it is enforced in code (see [`docs/NEUTRALITY.md`](docs/NEUTRALITY.md)).
+Every fact links directly to an official government record. No editorializing. No partisan framing. Just the record.
-## Status
-Planning → build (started 2026-06-26). MVP = federal + state.
+---
-## Repo map
-- [`PLAN.md`](PLAN.md) - full product & brand plan (the source of truth for *what* and *why*)
-- [`docs/BRAND.md`](docs/BRAND.md) - design system, tokens, components (the *look*)
-- [`docs/NEUTRALITY.md`](docs/NEUTRALITY.md) - the enforceable neutrality doctrine (the *guardrails*)
-- [`docs/DATA_SOURCES.md`](docs/DATA_SOURCES.md) - API contracts for every data source
-- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) - system design, services, data model
+## Neutrality Statement
-## Stack (target)
-Next.js (PWA, dark-first) · Tailwind w/ Docket tokens · Postgres · Python/Node ingest workers · ntfy + web-push · Cloudflare front + homelab CT backend.
+Docket is deliberately and structurally non-partisan. It presents the official public record as-is. To enforce that:
-## Decisions (locked 2026-06-26)
-Name **Docket** · accent **brass `#E0A53B`** · scope **federal + state** · summaries **structured-fields-first**.
+- **Government sources only.** Every displayed fact links to an official `.gov` or state legislature URL. No think tanks, no news sources, no secondhand data.
+- **No adjectives in generated text.** Numbers, names, dates, and status only. "Reduced by $X" - yes. "Slashed" or "expanded" - no. This is enforced in code via a banned-words lint that fails the build.
+- **Ranked by official action, not by virality.** Feed ranking uses date of legislative action, dollar magnitude, and affected-population estimates from CBO or agency data - never news coverage, social shares, or editorial judgment.
+- **No comment sections, ever.** Comments are the single largest bias-injection surface. The only user feedback allowed is a yes/no on summary *clarity*, never on a bill's merit.
+- **LLM summaries are field-extraction templates, not free-form generation.** Prose summaries pull structured fields from official bill text. High-salience bills are tested against a Polarization Index before they go live.
+
+See [`docs/NEUTRALITY.md`](docs/NEUTRALITY.md) for the full enforceable doctrine, including CI gates.
+
+---
+
+## Screenshots
+
+<table>
+<tr>
+<td width="50%">
+
+**Activity feed - dark mode**
+
+<img src="docs/screenshots/home-dark.png" alt="Docket home - dark mode" width="100%">
+
+</td>
+<td width="50%">
+
+**Activity feed - light mode**
+
+<img src="docs/screenshots/home-light.png" alt="Docket home - light mode" width="100%">
+
+</td>
+</tr>
+<tr>
+<td width="50%">
+
+**Federal spending tracker**
+
+<img src="docs/screenshots/spending.png" alt="Federal spending view" width="100%">
+
+</td>
+<td width="50%">
+
+**Cross-type search**
+
+<img src="docs/screenshots/search.png" alt="Search across bills, votes, spending" width="100%">
+
+</td>
+</tr>
+</table>
+
+---
+
+## Features
+
+- **Bills and laws** - every bill introduced, its full text, sponsor, committee path, and enacted law
+- **Amendments and riders** - text-diff view showing what was added or removed between bill versions (Introduced to Enrolled), flagging provisions inserted in appropriations or conference reports
+- **Votes - recorded and unrecorded** - roll calls with per-member positions, and surface notices when something passed by voice vote or unanimous consent (no individual record exists - that absence is itself information)
+- **Executive actions** - Executive Orders, proposed rules, and final rules from the Federal Register
+- **Federal spending** - contracts, grants, agency outlays, and appropriation apportionments from USAspending.gov and Treasury Fiscal Data
+- **State coverage** - 50 states plus DC and Puerto Rico via OpenStates and LegiScan, with an honest per-state confidence map
+- **Lobbying disclosures** - LD-1/LD-2 filings from LDA.gov showing who lobbied which bill
+- **Congressional calendar and schedule** - committee markups, hearings, and floor session timelines
+- **/how-we-pick** and **/who-pays** - public transparency pages explaining the ranking algorithm and project funding
+
+---
+
+## Data Sources
+
+All primary sources are free and official. No secondhand sources appear in user-facing data.
+
+| Source | Data | Auth |
+|--------|------|------|
+| [Congress.gov API](https://api.congress.gov) | Bills, amendments, members, votes | Free api.data.gov key |
+| [GovInfo API](https://api.govinfo.gov) | Full bill text, Congressional Record, public laws | Same key |
+| [USAspending.gov API](https://api.usaspending.gov) | Federal awards, agency outlays | None |
+| [Treasury Fiscal Data](https://fiscaldata.treasury.gov) | Debt, deficit, daily cash flows | None |
+| [House Clerk XML](https://clerk.house.gov/evs/) | Roll-call votes, posted within hours | None |
+| [Senate roll-call XML](https://www.senate.gov/legislative/LIS/roll_call_lists/) | Senate roll calls | None |
+| [Federal Register API](https://www.federalregister.gov/api/v1) | Executive Orders, rules, notices | None |
+| [LDA.gov API](https://lda.gov/api) | Lobbying disclosures (LD-1/LD-2) | Free key |
+| [OpenStates v3](https://v3.openstates.org) | 50-state bill and vote data | API key (paid tier for production) |
+| [LegiScan](https://legiscan.com) | State coverage supplement | API key |
+| [CBO](https://cbo.gov/cost-estimates) | Cost estimates per bill | None (scrape) |
+
+When an aggregator normalizes data (OpenStates, LegiScan), the user-facing source link always points to the underlying official government record.
+
+See [`docs/DATA_SOURCES.md`](docs/DATA_SOURCES.md) for the full contracts, rate limits, and change-detection strategy.
+
+---
+
+## How It Works
+
+```
+Government APIs --> Python ingest workers --> SQLite
+(Congress.gov, (services/ingest/) (docket.db)
+ GovInfo, |
+ USAspending, v
+ Federal Register, Next.js API routes
+ OpenStates, ...) (app/api/*)
+ |
+ v
+ Browser (dark + light)
+```
+
+The ingest layer polls each source on a configurable schedule, detects changes by watermark and content-hash, and writes normalized rows to SQLite. The Next.js frontend reads its own API routes backed by `better-sqlite3` - no separate API server needed for single-box deployments. Every row carries an allowlisted `sourceUrl` pointing to an official government endpoint; the build rejects any fact without one.
+
+Detailed architecture: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
+API contract: [`docs/API.md`](docs/API.md)
+Ingest strategy: [`docs/INGESTION_STRATEGY.md`](docs/INGESTION_STRATEGY.md)
+
+---
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js 20+
+- Python 3.11+
+- A free [api.data.gov](https://api.data.gov) key (covers Congress.gov and GovInfo)
+
+### 1. Clone and install
+
+```bash
+git clone https://github.com/zionboggan/docket.git
+cd docket
+npm install
+```
+
+### 2. Set up environment variables
+
+```bash
+cp .env.example .env
+# Fill in DOCKET_CONGRESS_KEY (api.data.gov key)
+# Optional: DOCKET_OPENSTATES_KEY, DOCKET_LEGISCAN_KEY, DOCKET_LEGISCAN_KEY
+```
+
+### 3. Run ingest (first-time bootstrap)
+
+```bash
+cd services/ingest
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+
+# Seed with federal data (no keys needed for GovInfo bulk path)
+DOCKET_DB_PATH=../../docket.db python -m ingest.run --source bills_bulk --once
+DOCKET_DB_PATH=../../docket.db python -m ingest.run --source votes_house --once
+```
+
+### 4. Run the frontend
+
+```bash
+# from repo root
+DOCKET_DB_PATH=./docket.db npm run dev
+```
+
+Open [http://localhost:3000](http://localhost:3000).
+
+### 5. Run the neutrality lint
+
+```bash
+npm run neutrality
+```
+
+This checks all generated copy, summaries, and push text against the banned-words list and confirms every fact has an allowlisted source URL.
+
+---
+
+## Tech Stack
+
+| Layer | Technology |
+|-------|-----------|
+| Frontend | Next.js 14 (App Router), React 18, Tailwind CSS |
+| Database | SQLite via better-sqlite3 |
+| Ingest | Python 3.11+, asyncio, ARQ (Redis-backed workers) |
+| Fonts | Archivo (display), JetBrains Mono (data labels), Fraunces (prose) |
+| Hosting | Cloudflare Pages (frontend edge) + self-hosted ingest workers |
+| Push | ntfy (MVP), Web Push API |
+
+---
+
+## Roadmap
+
+- [x] Federal bills, votes, members
+- [x] Federal spending (USAspending + Treasury)
+- [x] Executive actions (Federal Register)
+- [x] Lobbying disclosures (LDA.gov)
+- [x] State-level bills and votes (50 states + DC + PR)
+- [x] Per-state data-confidence map
+- [x] Congressional calendar and schedule
+- [x] Bill version text-diff (rider/amendment detection)
+- [x] Dark and light themes, mobile-first
+- [x] Neutrality lint in CI
+- [ ] Push alerts for topic follows (policy area, agency, or member)
+- [ ] Per-member voting record page
+- [ ] CBO cost-estimate integration per bill
+- [ ] Personalized weekly digest
+- [ ] Historical trend views
+- [ ] Share-as-image (factual card, no comment import)
+
+---
+
+## Contributing
+
+Contributions that strengthen neutrality, improve data coverage, or fix bugs are welcome.
+
+Before opening a PR, run `npm run neutrality` and confirm the lint passes. Any copy that adds qualitative modifiers, assigns blame, or characterizes a bill's merit will not be merged.
+
+See the neutrality doctrine at [`docs/NEUTRALITY.md`](docs/NEUTRALITY.md).
+
+---
+
+## License
+
+Apache 2.0 - see [LICENSE](LICENSE).
SECURITY_AUDIT.md +5 -5
@@ -1,6 +1,6 @@
# Docket - Security Audit Report
**Date:** 2026-06-26
-**Scope:** `/shared/projects/docket` codebase + live instance `http://localhost:3000`
+**Scope:** `docket` codebase + live instance `http://localhost:3000`
**Architecture:** Next.js 14.2.15 (App Router) + better-sqlite3, read-only public data viewer, no auth, no writes
---
@@ -8,7 +8,7 @@
## Methodology
- Full source read: `lib/db.ts`, `lib/feed.ts`, `middleware.ts`, `next.config.js`, all `app/api/**` routes, `components/source-link.tsx`, `lib/neutrality.ts`
-- Live probes via `pct exec 247 -- curl ...`
+- Live probes via `curl` against the running instance
- DB schema inspection (row counts, indexes)
- `npm audit --omit=dev`
@@ -185,11 +185,11 @@ Not a vulnerability - noted for operational awareness.
---
-### F8 - INFO: `docket.db` and `docket.db-journal` exist in the project share at `/shared/projects/docket/`
+### F8 - INFO: `docket.db` and `docket.db-journal` may exist in the project directory
-**Observed:** `ls /shared/projects/docket/` (CT215) shows `docket.db` and `docket.db-journal` alongside the source code.
+**Observed:** `docket.db` and `docket.db-journal` can appear alongside the source code during development.
-These are not in the CT247 `/opt/docket/public/` directory and are not web-accessible. However, the share mounts suggest these files may be accessible to other containers that mount `/shared/projects`. Confirm the `.db` files in the share are not a copy of production data - or if they are, ensure the share ACLs restrict access appropriately.
+These are not in the deployment web root and are not web-accessible. Confirm the `.db` files are not checked into version control and are not a copy of production data. SQLite database files are excluded via `.gitignore`.
---
STATE_INGEST_FIX_NOTES.md +2 -3
@@ -102,9 +102,8 @@ future targeted pass within the monthly API budget.
## OpenStates status
-DNS resolution for `v3.openstates.org` fails from CT247 (network configuration issue,
-not a code bug). The poller handles this gracefully: logs the error per jurisdiction and
-returns 0 bills with exit code 0. Once DNS is resolved on CT247, the poller will work
+DNS resolution for `v3.openstates.org` requires proper network access from the ingest host. The poller handles DNS failures gracefully: logs the error per jurisdiction and
+returns 0 bills with exit code 0. Once network access is confirmed, the poller will work
as written. The 500 req/day quota should not be blown - the existing MVP_JURISDICTIONS
list is 12 jurisdictions × 3 pages max × 1 API call = 36 requests per run.
VOTE_ARCHIVE_NOTES.md +182 -0
@@ -0,0 +1,182 @@
+# Vote Archive Notes
+
+## Issue #5 - Fix source_urls and build rolling R2 archive
+
+### Task 1: Source URL Fixes
+
+#### Problem
+
+Senate and House vote source_urls were broken or suboptimal:
+
+**House** - `clerk.house.gov/Votes/{year}/roll{NNN}` → HTTP 404
+The House Clerk reorganized their site. The `/Votes/` page URL no longer exists.
+
+**Senate** - URLs pointed to raw XML (`.xml` extension), which works but delivers
+raw machine-readable content rather than the human-readable vote detail page.
+
+#### Correct URL Formats
+
+**House** (verified 200):
+```
+https://clerk.house.gov/evs/{year}/roll{NNN:03d}.xml
+```
+This is the authoritative machine-readable record and also the canonical public URL
+the House Clerk uses for the data.
+
+**Senate display page** (verified 200):
+```
+https://www.senate.gov/legislative/LIS/roll_call_votes/vote{c}{s}/vote_{c}_{s}_{n:05d}.htm
+```
+senate.gov links its roll-call index to `.htm` pages (human-readable layout).
+
+**Senate XML data** (verified 200, used for fetching and archiving):
+```
+https://www.senate.gov/legislative/LIS/roll_call_votes/vote{c}{s}/vote_{c}_{s}_{n:05d}.xml
+```
+Same path, `.xml` extension - delivers raw XML content.
+
+**Old CFM page** (broken, returns 301 → unavailable):
+```
+https://www.senate.gov/legislative/LIS/roll_call_vote_cfm.cfm?congress=...
+```
+
+#### Backfill Applied
+
+One-time UPDATE via `ingest.archive_votes.fix_source_urls()` (idempotent):
+
+| Chamber | URLs fixed |
+|---------|-----------|
+| House | 22,603 (changed `/Votes/` → `/evs/` + appended `.xml`) |
+| Senate | 12,916 (changed `.xml` extension → `.htm`) |
+
+SQL applied:
+```sql
+-- House
+UPDATE votes
+SET source_url = 'https://clerk.house.gov/evs/'
+ || substr(source_url, length('https://clerk.house.gov/Votes/') + 1)
+ || '.xml'
+WHERE chamber = 'house'
+ AND source_url LIKE 'https://clerk.house.gov/Votes/%';
+
+-- Senate
+UPDATE votes
+SET source_url = substr(source_url, 1, length(source_url) - 3) || 'htm'
+WHERE chamber = 'senate'
+ AND source_url LIKE '%.xml';
+```
+
+#### Code Changes
+
+- `services/ingest/ingest/pollers/votes_senate.py`
+ - Added `_source_url()` returning `.htm`; `_vote_url()` returns `.xml` (for fetching)
+ - `poll()` now stores `.htm` as `source_url` and writes `congress`/`session`/`roll_number`
+ - Updated `CURRENT_SESSION` to 2 (119th Congress 2nd session, Jan 2026)
+
+- `services/ingest/ingest/pollers/voteview.py`
+ - `_house_source_url()` changed from `/Votes/{year}/roll{NNN}` to `/evs/{year}/roll{NNN:03d}.xml`
+ - `_senate_source_url()` changed from `.xml` to `.htm`
+
+- `votes_house.py` - already correct (`/evs/` format), no change needed.
+
+---
+
+### Task 2: Rolling R2 Archive
+
+#### Module
+
+`services/ingest/ingest/archive_votes.py`
+
+Snapshots the official roll-call XML from `.gov` sources and uploads to R2
+(bucket `docket-archive`, public at `https://archive.usdocket.org`).
+
+#### R2 Key Scheme
+
+```
+votes/senate/{congress}/{session}/{per_session_roll:05d}.xml
+votes/house/{year}/{roll_number:03d}.xml
+```
+
+#### Senate Per-Session Roll Number Note
+
+senate.gov resets its roll-call numbers at the start of each new session.
+Voteview (which provides bulk historical data) numbers Senate votes cumulatively
+across the whole congress - so session 2 roll numbers continue from where
+session 1 ended.
+
+`_derive_senate_metadata()` corrects this:
+1. **Priority 1**: parse from `source_url` - the URL encodes the correct
+ per-session number as generated by our pollers.
+2. **Priority 2**: for session 1, the DB `roll_number` IS the per-session roll.
+ For session 2+, query the max roll from the previous session and subtract.
+
+#### Content Validation
+
+Responses are validated with `_is_xml_content()` before upload. senate.gov and
+clerk.house.gov redirect unavailable votes to HTML error pages (HTTP 301 → 200
+with HTML). Votes with HTML responses are skipped cleanly.
+
+#### Idempotent / Rolling
+
+- `exists(r2_key)` is checked before each fetch - already-archived votes are
+ skipped (only the `archive_url` DB column update is written if the key exists
+ but the DB field is missing).
+- Re-running archives new votes that have been ingested since the last run.
+- `fix_source_urls()` is idempotent (UPDATE WHERE LIKE - no-ops after first run).
+- `purge_bad_archives()` verifies each archived URL and resets `archive_url = NULL`
+ if the content is not valid XML (one-time cleanup; safe to re-run).
+
+#### Pipeline Integration
+
+Added to `ingest/run.py` after `link_votes`:
+
+```python
+("archive_votes", "ingest.archive_votes", []),
+```
+
+#### CLI Usage
+
+```bash
+# Full run (fix source_urls + purge bad + archive 500 votes)
+python -m ingest.archive_votes --once
+
+# Quick test
+python -m ingest.archive_votes --once --limit 20 --chamber house
+
+# Rolling run (no source fix, no purge, faster)
+python -m ingest.archive_votes --once --skip-source-fix --no-purge --limit 500
+
+# Full run via run.py
+DOCKET_DB_PATH=/opt/docket/data/docket.db python -m ingest.run --source archive_votes
+```
+
+#### Verified Archives
+
+Sample archived URLs confirmed 200 + valid XML:
+
+- `https://archive.usdocket.org/votes/house/2026/223.xml` - 200, 82,755 bytes, `application/xml`
+- `https://archive.usdocket.org/votes/house/2026/224.xml` - 200, valid XML
+- `https://archive.usdocket.org/votes/house/2026/225.xml` - 200, valid XML
+
+Sample senate archive (session 1, correct per-session roll numbers via source_url parsing):
+After running with the corrected `_derive_senate_metadata()`, senate session 1 votes
+archive correctly at `https://archive.usdocket.org/votes/senate/119/1/{roll:05d}.xml`.
+
+#### DB Schema Change
+
+```sql
+ALTER TABLE votes ADD COLUMN archive_url TEXT;
+```
+
+Applied idempotently by `ensure_archive_schema()` on first run.
+
+---
+
+### What Each Vote Row Has After This Work
+
+| Field | Value |
+|-------------|-------|
+| `source_url` | Official .gov URL (human-readable): `.htm` for Senate, `.xml` for House |
+| `archive_url` | Our preserved copy: `https://archive.usdocket.org/votes/{chamber}/...xml` |
+
+The frontend can show both: "Official source" (source_url) + "Archived copy" (archive_url).
app/api/bills/[id]/diff/route.ts +1 -1
@@ -19,7 +19,7 @@ export async function GET(
// `note` carries honesty text when a change entered via voice/UC/conference.
if (!diff.changes.length) {
diff.note =
- "No section-level diff on file. Some changes enter via voice vote or unanimous consent - no individual record exists for those.";
+ "No section-level diff on file. Some changes enter via voice vote or unanimous consent - no individual record exists for those.";
}
return NextResponse.json(diff);
}
app/api/laws/route.ts +38 -0
@@ -0,0 +1,38 @@
+import { NextResponse, type NextRequest } from "next/server";
+import { parsePaging } from "@/lib/db";
+import {
+ getPublicLaws,
+ getPublicLawsCount,
+ getPrivateLawsCount,
+ type LawListItem,
+ type LawSort,
+} from "@/lib/feed";
+
+export const dynamic = "force-dynamic";
+
+type LawsResponse = {
+ items: LawListItem[];
+ total: number;
+ limit: number;
+ offset: number;
+ /** Count of enacted Private Laws (a separate, smaller class). */
+ privateCount: number;
+};
+
+/** GET /api/laws - enacted Public Laws, most-recent-first. ?q=&sort=recent|oldest|number. */
+export async function GET(req: NextRequest) {
+ const sp = req.nextUrl.searchParams;
+ const { limit, offset } = parsePaging(sp);
+ const q = sp.get("q") || undefined;
+ const sortParam = sp.get("sort");
+ const sort: LawSort =
+ sortParam === "oldest" ? "oldest" :
+ sortParam === "number" ? "number" :
+ "recent";
+
+ const items = getPublicLaws({ q, sort }, limit, offset);
+ const total = getPublicLawsCount(q);
+ const privateCount = getPrivateLawsCount();
+ const body: LawsResponse = { items, total, limit, offset, privateCount };
+ return NextResponse.json(body);
+}
app/api/states/route.ts +1 -1
@@ -13,7 +13,7 @@ export async function GET() {
return NextResponse.json({
items: [],
total: 0,
- note: "State coverage polling is not yet active. When it is, each state reports its own coverage level here - gaps are recorded, not hidden.",
+ note: "State coverage polling is not yet active. When it is, each state reports its own coverage level here - gaps are recorded, not hidden.",
});
}
return NextResponse.json({ items, total: items.length });
app/bill/[id]/page.tsx +6 -6
@@ -22,7 +22,7 @@ export function generateMetadata({
const numeric = parseInt(params.id.replace(/^bill:/, ""), 10);
const bill = Number.isFinite(numeric) ? getBillDetail(numeric) : null;
if (!bill) return { title: "Bill not found" };
- return { title: `${bill.number} - ${bill.policyArea}`, description: bill.title };
+ return { title: `${bill.number} - ${bill.policyArea}`, description: bill.title };
}
export default function BillPage({
@@ -288,7 +288,7 @@ export default function BillPage({
className="font-prose italic"
style={{ fontSize: "0.9375rem", color: "var(--dk-ash)" }}
>
- Plain-language summary not yet available - read the official text
+ Plain-language summary not yet available - read the official text
below.
</p>
</div>
@@ -321,7 +321,7 @@ export default function BillPage({
style={{ fontSize: "0.9375rem", color: "var(--dk-ash)" }}
>
No section-level diff on file. Some changes enter via voice vote
- or unanimous consent - no individual record exists for those.
+ or unanimous consent - no individual record exists for those.
</p>
</div>
) : (
@@ -428,7 +428,7 @@ export default function BillPage({
className="font-display font-bold text-paper"
style={{ fontSize: "1.0625rem" }}
>
- {bill.votes.length} vote(s) recorded.
+ {bill.votes.length} vote{bill.votes.length === 1 ? "" : "s"} recorded.
</p>
) : (
<p
@@ -546,7 +546,7 @@ export default function BillPage({
</p>
<CiteThis
kind="bill"
- title={`${bill.number} - ${bill.title}`}
+ title={`${bill.number} - ${bill.title}`}
permalink={`https://usdocket.org/bill/${params.id}`}
sourceUrl={bill.sourceUrl}
/>
@@ -568,7 +568,7 @@ function SectionLabel({
{eyebrow ? (
<p
className="label"
- style={{ color: "var(--dk-signal)", marginBottom: "10px", fontSize: "11px", letterSpacing: "0.14em" }}
+ style={{ color: "var(--dk-signal)", marginBottom: "10px", fontSize: "12px", letterSpacing: "0.14em" }}
>
{eyebrow}
</p>
app/bills/page.tsx +2 -2
@@ -169,7 +169,7 @@ export default function BillsPage() {
}}
>
<p className="label mb-8" style={{ color: "var(--dk-ash-2)" }}>
- Bills sourced from congress.gov. Order is by last action date - no
+ Bills sourced from congress.gov. Order is by last action date -no
editorial weighting. No comment surface appears here, by policy.
</p>
<p className="label mb-4" style={{ color: "var(--dk-ash)" }}>
@@ -177,7 +177,7 @@ export default function BillsPage() {
</p>
<CiteThis
kind="page"
- title="Bills - Browse all bills"
+ title="Bills -Browse all bills"
permalink="https://usdocket.org/bills"
sourceUrl="https://www.congress.gov/browse/bills"
/>
app/calendar/page.tsx +3 -3
@@ -1,7 +1,7 @@
import type { Metadata } from "next";
import Link from "next/link";
import { Suspense } from "react";
-import { Card, PageHeader } from "@/components/ui";
+import { PageHeader } from "@/components/ui";
import { CalendarBrowser } from "@/components/calendar-browser";
export const revalidate = 300;
@@ -41,7 +41,7 @@ export default function CalendarPage() {
<PageHeader
eyebrow="Calendar · when it happens"
title="What takes effect when"
- lead="A month at a glance. Each day lists the laws that take effect, bills that became law, votes, and floor action. Pick a day to read its events - each sealed to its official source."
+ lead="A month at a glance. Each day lists the laws that take effect, bills that became law, votes, and floor action. Pick a day to read its events, each sealed to its official source."
/>
<Suspense fallback={<CalendarSkeleton />}>
@@ -63,7 +63,7 @@ function CalendarSkeleton() {
<div className="mt-8 space-y-6">
<div className="h-12 w-full animate-pulse rounded-sm bg-graphite" />
<div className="grid gap-6 lg:grid-cols-[1.5fr_1fr]">
- <Card className="hidden h-[28rem] animate-pulse bg-graphite md:block" />
+ <div className="hidden h-[28rem] animate-pulse rounded-sm bg-graphite md:block" />
<div className="h-[28rem] animate-pulse rounded-sm bg-graphite" />
</div>
</div>
app/categories/page.tsx +2 -2
@@ -8,7 +8,7 @@ export const revalidate = 300;
export const metadata: Metadata = {
title: "Topics",
description:
- "Browse the record by topic - bills and awards grouped by policy area. Counts only, no editorial weighting.",
+ "Browse the record by topic -bills and awards grouped by policy area. Counts only, no editorial weighting.",
};
export default function CategoriesPage() {
@@ -31,7 +31,7 @@ export default function CategoriesPage() {
<PageHeader
eyebrow="Topics · browse the record"
title="By policy area"
- lead="Every bill is tagged with one policy area; federal awards carry the same tags. Pick a topic to read the bills filed under it. Counts reflect the record only - no ranking by importance."
+ lead="Every bill is tagged with one policy area; federal awards carry the same tags. Pick a topic to read the bills filed under it. Counts reflect the record only -no ranking by importance."
/>
<div className="mt-8">
app/executive/page.tsx +1 -1
@@ -113,7 +113,7 @@ export default function ExecutivePage() {
<p className="label mb-4" style={{ color: "var(--dk-ash)" }}>Cite this page</p>
<CiteThis
kind="page"
- title="Federal Register - Executive actions"
+ title="Federal Register - Executive actions"
permalink="https://usdocket.org/executive"
sourceUrl="https://www.federalregister.gov/"
/>
app/globals.css +10 -1
@@ -43,6 +43,11 @@
rgba(224, 165, 59, 0) 70%
);
+ /* Site header - theme-aware so the bar + its text both read in BOTH modes.
+ Dark: translucent near-black bar, light bone text. */
+ --dk-header-bg: rgba(11, 12, 14, 0.88);
+ --dk-header-border: rgba(155, 160, 166, 0.18);
+
--dk-body-size: 18px;
--dk-leading: 1.6;
@@ -85,6 +90,10 @@
rgba(143, 89, 2, 0) 70%
);
+ /* Site header - light/paper bar so the (now dark) text reads. */
+ --dk-header-bg: rgba(255, 255, 255, 0.85);
+ --dk-header-border: #d8d2c4;
+
/* Match dark mode body size - day version, same rhythm */
--dk-body-size: 18px;
--dk-leading: 1.6;
@@ -248,7 +257,7 @@
font-family: var(--font-mono), ui-monospace, monospace;
text-transform: uppercase;
letter-spacing: 0.08em;
- font-size: 0.625rem; /* 10px */
+ font-size: 0.75rem; /* 12px - elderly readability bump (was 10px) */
padding: 0.1875rem 0.5rem;
border: 1px solid var(--dk-line);
border-radius: 2px;
app/how-we-pick/page.tsx +3 -3
@@ -116,8 +116,8 @@ export default function HowWePickPage() {
<Card className="p-5">
<ul className="space-y-2 font-mono text-[12px] text-ash">
<li>· Party affiliation and vote splits are factual data and may be shown.</li>
- <li>· Tones are desaturated and of equal weight - no saturated red vs blue.</li>
- <li>· Default sort of member or vote lists is by vote, then alphabetical - not grouped by party.</li>
+ <li>· Tones are desaturated and of equal weight -no saturated red vs blue.</li>
+ <li>· Default sort of member or vote lists is by vote, then alphabetical -not grouped by party.</li>
<li>· Charts use a neutral yea / nay encoding where the question is not party.</li>
</ul>
</Card>
@@ -154,6 +154,6 @@ const RULES = [
},
{
title: "Extraction templates for summaries",
- body: "Prose summaries, when they land, are field extraction into a fixed template - not open-ended generation. Each passes the banned-words lint and a polarization check.",
+ body: "Prose summaries, when they land, are field extraction into a fixed template -not open-ended generation. Each passes the banned-words lint and a polarization check.",
},
];
app/laws/page.tsx +179 -0
@@ -0,0 +1,179 @@
+import type { Metadata } from "next";
+import Link from "next/link";
+import { Suspense } from "react";
+import { getPublicLawsCount, getPrivateLawsCount } from "@/lib/feed";
+import { ordinal } from "@/lib/format";
+import { CiteThis } from "@/components/cite-this";
+import { LawsBrowser } from "@/components/laws-browser";
+
+export const revalidate = 300;
+
+export const metadata: Metadata = {
+ title: "Laws",
+ description:
+ "Enacted Public Laws of the United States. Each law links to the bill it came from and the official record.",
+};
+
+export default function LawsPage() {
+ const publicCount = getPublicLawsCount();
+ const privateCount = getPrivateLawsCount();
+
+ return (
+ <div
+ className="mx-auto w-full max-w-[860px] px-6 sm:px-8"
+ style={{
+ paddingTop: "clamp(32px, 6vw, 56px)",
+ paddingBottom: "clamp(48px, 8vw, 80px)",
+ }}
+ >
+ {/* Breadcrumb + Back */}
+ <nav aria-label="Breadcrumb" className="mb-10">
+ <ol className="flex flex-wrap items-center gap-2" style={{ marginBottom: "12px" }}>
+ <li>
+ <Link
+ href="/"
+ className="focus-ring label no-underline hover:underline"
+ style={{ color: "var(--dk-signal)", fontSize: "12px", letterSpacing: "0.12em" }}
+ >
+ Home
+ </Link>
+ </li>
+ <li aria-hidden="true" className="label" style={{ color: "var(--dk-ash-2)", fontSize: "12px" }}>
+ /
+ </li>
+ <li
+ className="label"
+ style={{ color: "var(--dk-ash)", fontSize: "12px", letterSpacing: "0.12em" }}
+ aria-current="page"
+ >
+ Laws
+ </li>
+ </ol>
+ <Link
+ href="/bills"
+ className="focus-ring tap no-underline hover:underline"
+ style={{
+ display: "inline-flex",
+ alignItems: "center",
+ gap: "8px",
+ fontFamily: "var(--font-mono), ui-monospace, monospace",
+ fontSize: "13px",
+ letterSpacing: "0.08em",
+ color: "var(--dk-signal)",
+ }}
+ >
+ ← All bills
+ </Link>
+ </nav>
+
+ {/* Page header */}
+ <header
+ style={{
+ borderBottom: "1px solid var(--dk-line)",
+ paddingBottom: "clamp(28px, 5vw, 48px)",
+ marginBottom: "clamp(28px, 5vw, 48px)",
+ }}
+ >
+ <p
+ className="label"
+ style={{
+ color: "var(--dk-signal)",
+ marginBottom: "20px",
+ display: "flex",
+ alignItems: "center",
+ gap: "8px",
+ fontSize: "12px",
+ letterSpacing: "0.14em",
+ }}
+ >
+ <span
+ aria-hidden="true"
+ style={{
+ width: "5px",
+ height: "5px",
+ borderRadius: "50%",
+ background: "var(--dk-signal)",
+ display: "inline-block",
+ flexShrink: 0,
+ }}
+ />
+ Enacted
+ </p>
+ <h1
+ className="font-display text-paper"
+ style={{
+ fontSize: "clamp(2.2rem, 7vw, 3.75rem)",
+ fontWeight: 900,
+ lineHeight: 1.0,
+ letterSpacing: "-0.01em",
+ wordSpacing: "0.06em",
+ textTransform: "uppercase",
+ maxWidth: "18ch",
+ marginBottom: "24px",
+ }}
+ >
+ Public Laws
+ </h1>
+ <p
+ className="font-prose"
+ style={{
+ fontStyle: "italic",
+ fontSize: "clamp(1rem, 1.8vw, 1.125rem)",
+ lineHeight: 1.55,
+ color: "var(--dk-ash)",
+ maxWidth: "56ch",
+ }}
+ >
+ A Public Law is a bill that has been enacted. Once enacted, it becomes a
+ fixed part of the record. This list grows as more bills become law -
+ currently {publicCount.toLocaleString()} from the {ordinal(119)} Congress
+ {privateCount > 0
+ ? `, alongside ${privateCount.toLocaleString()} Private Law${privateCount === 1 ? "" : "s"} (which apply to specific individuals or entities).`
+ : "."}
+ </p>
+ </header>
+
+ <Suspense fallback={<LawsSkeleton />}>
+ <LawsBrowser />
+ </Suspense>
+
+ {/* Footer / cite */}
+ <div
+ style={{
+ marginTop: "clamp(40px, 7vw, 72px)",
+ borderTop: "1px solid var(--dk-line)",
+ paddingTop: "clamp(24px, 4vw, 40px)",
+ }}
+ >
+ <p className="label mb-8" style={{ color: "var(--dk-ash-2)" }}>
+ Public Laws are identified by their status on congress.gov. Order is by
+ law number - no editorial weighting. No comment surface appears here, by
+ policy.
+ </p>
+ <p className="label mb-4" style={{ color: "var(--dk-ash)" }}>
+ Cite this page
+ </p>
+ <CiteThis
+ kind="page"
+ title="Laws - Enacted Public Laws"
+ permalink="https://usdocket.org/laws"
+ sourceUrl="https://www.congress.gov/public-laws/119th-congress"
+ />
+ </div>
+ </div>
+ );
+}
+
+function LawsSkeleton() {
+ return (
+ <div className="mt-8">
+ <div className="h-14 w-full animate-pulse rounded-sm bg-graphite" />
+ <div className="mt-8 mb-4 h-4 w-40 animate-pulse rounded-sm bg-graphite" />
+ <div className="space-y-3">
+ {Array.from({ length: 12 }).map((_, i) => (
+ <div key={i} className="h-28 w-full animate-pulse rounded-sm bg-graphite" />
+ ))}
+ </div>
+ </div>
+ );
+}
app/layout.tsx +1 -1
@@ -34,7 +34,7 @@ const fraunces = Fraunces({
export const metadata: Metadata = {
title: {
- default: "Docket - what the government did this week",
+ default: "Docket - what the government did this week",
template: "%s · Docket",
},
description:
app/members/[id]/page.tsx +16 -3
@@ -7,7 +7,7 @@ import { SourceLink } from "@/components/source-link";
import { PartyTag } from "@/components/party-tag";
import { PositionTag } from "@/components/position-tag";
import { MemberAvatar } from "@/components/member-avatar";
-import { StatusPill, classifyStatus } from "@/components/status-pill";
+import { StatusPill, classifyStatus, type StatusVariant } from "@/components/status-pill";
import {
getMember,
type MemberVote,
@@ -19,6 +19,17 @@ import { CiteThis } from "@/components/cite-this";
export const revalidate = 300;
+/** Short, non-clipping status labels for the sponsored-bills pills - the raw
+ * status can be a full sentence ("Read twice and referred to the Committee on
+ * Banking, Housing…") which overruns the nowrap pill. Mirrors /bills. */
+const STATUS_SHORT: Record<StatusVariant, string> = {
+ passed: "Passed",
+ failed: "Failed",
+ held: "In committee",
+ pending: "Introduced",
+ neutral: "Introduced",
+};
+
export function generateMetadata({
params,
}: {
@@ -227,7 +238,9 @@ export default function MemberDetailPage({
{b.title}
</p>
<div className="mt-2 flex items-center gap-2">
- <StatusPill variant={classifyStatus(b.status)}>{b.status || "Introduced"}</StatusPill>
+ <StatusPill variant={classifyStatus(b.status)}>
+ {STATUS_SHORT[classifyStatus(b.status)]}
+ </StatusPill>
<span className="label text-ash-2">{formatDate(b.updatedAt)}</span>
</div>
</div>
@@ -304,7 +317,7 @@ export default function MemberDetailPage({
<p className="label mb-4 text-ash">Cite this record</p>
<CiteThis
kind="member"
- title={`${member.name} (${member.partyCode}-${member.state}) - Congressional Record`}
+ title={`${member.name} (${member.partyCode}-${member.state}) - Congressional Record`}
permalink={`https://usdocket.org${member.href}`}
sourceUrl={member.sourceUrl}
/>
app/members/page.tsx +31 -8
@@ -77,15 +77,29 @@ export default function MembersPage() {
<PageHeader
eyebrow="Congress roster"
title="Representatives"
- lead="Search and filter the sitting House and Senate. Open a member to see their recorded votes, sponsored bills, and amendments - what your representative did, on the record."
+ lead="Search and filter the sitting House and Senate. Open a member to see their recorded votes, sponsored bills, and amendments - what your representative did, on the record."
/>
<div>
- {/* Summary by chamber, ordered by count then alphabetical (not by party). */}
+ {/* Headline = currently serving (what people expect); the all-time
+ record total is shown as an accurate sub-note. Ordered by chamber
+ size, not by party. */}
<div className="grid gap-4 sm:grid-cols-3">
- <SummaryCard label="Total" count={counts.total} />
- <SummaryCard label="Senate" count={counts.senate} />
- <SummaryCard label="House" count={counts.house} />
+ <SummaryCard
+ label="Currently serving"
+ count={counts.currentTotal}
+ sub={`${counts.total.toLocaleString()} on record, all Congresses`}
+ />
+ <SummaryCard
+ label="Senate"
+ count={counts.currentSenate}
+ sub={`${counts.senate.toLocaleString()} on record`}
+ />
+ <SummaryCard
+ label="House"
+ count={counts.currentHouse}
+ sub={`${counts.house.toLocaleString()} on record`}
+ />
</div>
<div className="mt-10">
@@ -108,7 +122,7 @@ export default function MembersPage() {
<p className="label mb-4" style={{ color: "var(--dk-ash)" }}>Cite this page</p>
<CiteThis
kind="page"
- title="Members of Congress - Roster"
+ title="Members of Congress - Roster"
permalink="https://usdocket.org/members"
sourceUrl="https://bioguide.congress.gov/"
/>
@@ -117,14 +131,23 @@ export default function MembersPage() {
);
}
-function SummaryCard({ label, count }: { label: string; count: number }) {
+function SummaryCard({
+ label,
+ count,
+ sub,
+}: {
+ label: string;
+ count: number;
+ sub?: string;
+}) {
return (
<Card className="p-5">
<p className="label mb-2">{label}</p>
<div className="flex items-baseline gap-2">
- <span className="display text-3xl text-paper">{count}</span>
+ <span className="display text-3xl text-paper">{count.toLocaleString()}</span>
<span className="label text-ash-2">members</span>
</div>
+ {sub ? <p className="label mt-2 text-ash-2">{sub}</p> : null}
</Card>
);
}
app/page.tsx +8 -8
@@ -2,7 +2,7 @@ import type { Metadata } from "next";
import Link from "next/link";
export const metadata: Metadata = {
- title: "Docket - The public record of the U.S. government",
+ title: "Docket - The public record of the U.S. government",
description:
"Docket is a neutral, sourced record of official federal government activity: bills, votes, spending, and more. Every fact links to its official .gov source.",
};
@@ -239,7 +239,7 @@ export default function HomePage() {
{/* Card kicker */}
<p
className="label"
- style={{ color: "var(--dk-signal)", marginBottom: "10px", fontSize: "11px", letterSpacing: "0.14em" }}
+ style={{ color: "var(--dk-signal)", marginBottom: "10px", fontSize: "12px", letterSpacing: "0.14em" }}
>
{s.kicker}
</p>
@@ -338,7 +338,7 @@ export default function HomePage() {
}}
>
{/* Kicker */}
- <p className="label" style={{ color: "var(--dk-signal)", marginBottom: "20px", fontSize: "11px", letterSpacing: "0.14em" }}>
+ <p className="label" style={{ color: "var(--dk-signal)", marginBottom: "20px", fontSize: "12px", letterSpacing: "0.14em" }}>
Citations
</p>
@@ -368,7 +368,7 @@ export default function HomePage() {
On any bill, vote, or member page, scroll to the bottom and use the{" "}
<strong style={{ color: "var(--dk-paper)" }}>Cite this record</strong>{" "}
panel. It fills in the author, date, title, permalink, and the official
- government source automatically - in APA, MLA, or Chicago format.
+ government source automatically - in APA, MLA, or Chicago format.
</p>
<div
@@ -382,9 +382,9 @@ export default function HomePage() {
fontStyle: "italic",
}}
>
- <span style={{ color: "var(--dk-ash-2)", fontStyle: "normal", fontSize: "11px", letterSpacing: "0.08em", textTransform: "uppercase" }}>Example APA · </span>
+ <span style={{ color: "var(--dk-ash-2)", fontStyle: "normal", fontSize: "12px", letterSpacing: "0.08em", textTransform: "uppercase" }}>Example APA · </span>
<span style={{ fontStyle: "normal" }}>
- Docket. (2025). H.R. 22 - [Title]. usdocket.org. Retrieved from https://congress.gov/...
+ Docket. (2025). H.R. 22 - [Title]. usdocket.org. Retrieved from https://congress.gov/...
</span>
</div>
</div>
@@ -407,7 +407,7 @@ export default function HomePage() {
padding: "clamp(24px, 4vw, 36px)",
}}
>
- <p className="label" style={{ color: "var(--dk-ash-2)", marginBottom: "16px", fontSize: "11px", letterSpacing: "0.14em" }}>
+ <p className="label" style={{ color: "var(--dk-ash-2)", marginBottom: "16px", fontSize: "12px", letterSpacing: "0.14em" }}>
About this site
</p>
<p
@@ -421,7 +421,7 @@ export default function HomePage() {
>
Docket is neutral by design. It does not editorialize, rank by political
popularity, or surface opinion. The feed is ordered by action date and
- dollar magnitude only - the same factors as the official record. Party
+ dollar magnitude only - the same factors as the official record. Party
colors appear only in desaturated, equal-weight tones. No comment
section. No ads. All source links open the official .gov record directly.
</p>
app/register/page.tsx +4 -4
@@ -17,7 +17,7 @@ import { formatUsd, formatDate, formatEt } from "@/lib/format";
export const metadata: Metadata = {
title: "The Register",
description:
- "A live, stamped ledger of official US government actions - bills, votes, and dollars. Every fact sourced.",
+ "A live, stamped ledger of official US government actions - bills, votes, and dollars. Every fact sourced.",
};
// Revalidate every 5 minutes - staleness is acceptable for this read-only
@@ -66,7 +66,7 @@ export default async function RegisterPage() {
<p className="prose-lead mb-8 max-w-2xl text-ash">
Every bill, vote, and federal award is listed below, drawn from
Congress.gov, the Federal Register, and USAspending.gov. Ranked by
- date - never by clicks or editorial judgment. Every entry links to
+ date - never by clicks or editorial judgment. Every entry links to
its official source.
</p>
@@ -137,7 +137,7 @@ export default async function RegisterPage() {
</p>
<p className="prose-lead mt-2 text-ash">
{awardStats.count} federal awards from USAspending.gov, summed across
- the record. Figures only - no characterization.
+ the record. Figures only - no characterization.
</p>
<Link
href="/spending"
@@ -190,7 +190,7 @@ export default async function RegisterPage() {
))}
</ul>
<p className="label mt-4 text-ash-2">
- Bills and awards grouped by policy area. Counts only - no weighting.
+ Bills and awards grouped by policy area. Counts only - no weighting.
</p>
</Card>
</div>
app/schedule/page.tsx +1 -1
@@ -40,7 +40,7 @@ export default function SchedulePage() {
</Link>
<PageHeader
- eyebrow="Schedule · what Congress is meeting about"
+ eyebrow={"Schedule · what Congress is meeting about"}
title="Congressional meetings"
lead="Committee hearings, markups, business meetings, and floor action. Browse upcoming sessions or look back at past meetings. Each entry links to its official schedule."
/>
app/spending/page.tsx +19 -9
@@ -19,6 +19,13 @@ export const revalidate = 300;
/** Impound status → muted semantic tone (released = sage, delayed = hold,
* withheld = clay). Never vivid; a withheld status is recorded as fact. */
+/** Normalize source "--" double-hyphens to a typographic en-dash for display
+ * (e.g. "Corps of Engineers--Civil Works"). Cosmetic only; the cited record
+ * is unchanged. */
+function dedash(s: string): string {
+ return s ? s.replace(/--/g, "-") : s;
+}
+
function apportionmentColor(status: string): string {
const s = (status || "").toLowerCase();
if (s.startsWith("withhold")) return "var(--dk-nay)";
@@ -107,7 +114,7 @@ export default function SpendingPage() {
<PageHeader
eyebrow="Spending register"
title="Dollars, by official record"
- lead="Awards from USAspending.gov; debt, outlays, and cash from the Daily Treasury Statement. Figures only - no characterization."
+ lead="Awards from USAspending.gov; debt, outlays, and cash from the Daily Treasury Statement. Figures only - no characterization."
/>
{/* ── Section navigation - fix #10: impound tracker reachable immediately ── */}
@@ -117,7 +124,7 @@ export default function SpendingPage() {
>
<p
className="label"
- style={{ color: "var(--dk-ash-2)", fontSize: "11px", letterSpacing: "0.12em", marginBottom: "10px" }}
+ style={{ color: "var(--dk-ash-2)", fontSize: "12px", letterSpacing: "0.12em", marginBottom: "10px" }}
>
Jump to
</p>
@@ -223,7 +230,7 @@ export default function SpendingPage() {
<p className="label mb-4" style={{ color: "var(--dk-ash)" }}>Cite this page</p>
<CiteThis
kind="page"
- title="Spending - Federal awards and apportionments"
+ title="Spending - Federal awards and apportionments"
permalink="https://usdocket.org/spending"
sourceUrl="https://www.usaspending.gov/"
/>
@@ -240,19 +247,22 @@ function ApportionmentsTable({
color: (s: string) => string;
}) {
const columns: Column[] = [
+ // Explicit widths so the flexible agency column absorbs the remainder and
+ // the right-side source column is wide enough that its host wraps cleanly
+ // (separator never stranded).
{ key: "agency", header: "Agency", cardTitle: true },
- { key: "account", header: "Account" },
- { key: "appropriated", header: "Appropriated", align: "right", hideOnCard: true },
- { key: "apportioned", header: "Apportioned", align: "right", hideOnCard: true },
- { key: "status", header: "Status" },
- { key: "source", header: "Source", align: "right" },
+ { key: "account", header: "Account", width: "7.5rem" },
+ { key: "appropriated", header: "Appropriated", align: "right", width: "8rem", hideOnCard: true },
+ { key: "apportioned", header: "Apportioned", align: "right", width: "8rem", hideOnCard: true },
+ { key: "status", header: "Status", width: "6.5rem" },
+ { key: "source", header: "Source", align: "right", width: "11rem" },
];
const rows: Row[] = apportionments.map((a) => {
const c = color(a.status);
return {
key: a.id,
cells: {
- agency: <span className="text-paper">{a.agency}</span>,
+ agency: <span className="text-paper">{dedash(a.agency)}</span>,
account: <span className="text-ash">{a.account}</span>,
appropriated: (
<span className="tabular-nums text-paper">
app/states/page.tsx +2 -2
@@ -150,7 +150,7 @@ export default function StatesPage() {
<PageHeader
eyebrow="State register"
title="50 states + DC"
- lead="Federal delegation and sponsored bills are on record for every state, DC, and U.S. territory. State-level legislative coverage is rolling out - gaps are recorded here, not hidden."
+ lead="Federal delegation and sponsored bills are on record for every state, DC, and U.S. territory. State-level legislative coverage is rolling out; gaps are recorded here, not hidden."
/>
{/* Coverage summary - 50 states only; DC and territories listed separately below. */}
@@ -175,7 +175,7 @@ export default function StatesPage() {
Coverage by state
</h2>
<p className="prose-lead mt-3 max-w-2xl text-ash">
- How much of each state&apos;s legislature is on record here - across bills,
+ How much of each state&apos;s legislature is on record here - across bills,
votes, and sponsors. Federal delegation and bill sponsorships (from
congress.gov) are on record regardless. Open a state for its congressional
delegation.
app/votes/[id]/page.tsx +6 -6
@@ -124,7 +124,7 @@ export default function VoteDetailPage({
{formatBillNumber(vote.billNumber)}
</Link>
{vote.billTitle ? (
- <span className="text-ash"> - {vote.billTitle}</span>
+ <span className="text-ash"> - {vote.billTitle}</span>
) : null}
</p>
) : (
@@ -205,7 +205,7 @@ export default function VoteDetailPage({
/>
{vote.notVoting ? (
<p className="label mt-4 text-ash-2">
- {vote.notVoting} member(s) not voting.
+ {vote.notVoting} member{vote.notVoting === 1 ? "" : "s"} not voting.
</p>
) : null}
<div className="mt-4">
@@ -233,7 +233,7 @@ export default function VoteDetailPage({
/>
</div>
<p className="label mt-4 text-ash-2">
- Default order: by position, then alphabetical - not grouped by party.
+ Default order: by position, then alphabetical - not grouped by party.
</p>
</Card>
</div>
@@ -258,7 +258,7 @@ export default function VoteDetailPage({
<p className="label mt-4 text-ash-2">
Party shown in desaturated, equal-weight tones. Source: {""}
<span style={{ color: positionColor("Yea") }}>Yea</span>,{" "}
- <span style={{ color: positionColor("Nay") }}>Nay</span> - never vivid red or blue.
+ <span style={{ color: positionColor("Nay") }}>Nay</span> - never vivid red or blue.
</p>
</div>
</>
@@ -268,7 +268,7 @@ export default function VoteDetailPage({
<SectionHeader eyebrow="No individual record" title="Passed without a roll call" />
<Card className="p-6">
<p className="prose-lead text-paper">
- No individual record exists - this action passed by voice vote or
+ No individual record exists - this action passed by voice vote or
unanimous consent. The chamber did not record how each member voted.
</p>
<p className="mt-4 font-mono text-[13px] text-ash">
@@ -286,7 +286,7 @@ export default function VoteDetailPage({
<p className="label mb-4 text-ash">Cite this record</p>
<CiteThis
kind="vote"
- title={`${vote.chamber} Roll-Call Vote - ${subject.primary} (${formatDate(vote.date)})`}
+ title={`${vote.chamber} Roll-Call Vote - ${subject.primary} (${formatDate(vote.date)})`}
permalink={`https://usdocket.org/votes/${voteUrlId}`}
sourceUrl={vote.sourceUrl}
/>
app/votes/page.tsx +3 -3
@@ -65,7 +65,7 @@ export default function VotesPage() {
<PageHeader
eyebrow="Roll-call register"
title="Votes, on the record"
- lead="Each row is a recorded vote in the House or Senate - what it was about, the tally, and a link to the official roll call. Open any vote to see how every member voted."
+ lead="Each row is a recorded vote in the House or Senate -what it was about, the tally, and a link to the official roll call. Open any vote to see how every member voted."
/>
<Suspense fallback={<VotesSkeleton />}>
@@ -82,14 +82,14 @@ export default function VotesPage() {
>
<p className="label mb-8" style={{ color: "var(--dk-ash-2)" }}>
Roll-call votes from clerk.house.gov and senate.gov XML feeds. Order is
- by date - no editorial weighting.
+ by date -no editorial weighting.
</p>
<p className="label mb-4" style={{ color: "var(--dk-ash)" }}>
Cite this page
</p>
<CiteThis
kind="page"
- title="Votes - Roll-call register"
+ title="Votes -Roll-call register"
permalink="https://usdocket.org/votes"
sourceUrl="https://clerk.house.gov/evs/ https://www.senate.gov/legislative/votes_new.htm"
/>
app/who-pays/page.tsx +8 -8
@@ -5,7 +5,7 @@ import { Card, PageHeader, SectionHeader } from "@/components/ui";
export const metadata: Metadata = {
title: "Who pays",
description:
- "Ownership and funding of Docket. A trust surface - public, factual, no adjectives.",
+ "Ownership and funding of Docket. A trust surface - public, factual, no adjectives.",
};
export default function WhoPaysPage() {
@@ -65,13 +65,13 @@ export default function WhoPaysPage() {
/>
<Card className="p-5">
<ul className="grid gap-2 font-mono text-[12px] text-ash sm:grid-cols-2">
- <li>· Congress.gov - bills, actions, members, votes</li>
- <li>· GovInfo - full bill text, Congressional Record, public laws</li>
- <li>· USAspending.gov - awards and outlays</li>
- <li>· Treasury Fiscal Data - debt, deficit, cash flows</li>
- <li>· House Clerk and Senate - roll-call votes (XML)</li>
- <li>· CBO - cost estimates</li>
- <li>· OpenStates - 50-state + DC/PR coverage (paid tier)</li>
+ <li>· Congress.gov -bills, actions, members, votes</li>
+ <li>· GovInfo -full bill text, Congressional Record, public laws</li>
+ <li>· USAspending.gov -awards and outlays</li>
+ <li>· Treasury Fiscal Data -debt, deficit, cash flows</li>
+ <li>· House Clerk and Senate -roll-call votes (XML)</li>
+ <li>· CBO -cost estimates</li>
+ <li>· OpenStates -50-state + DC/PR coverage (paid tier)</li>
</ul>
</Card>
</div>
components/bills-browser.tsx +2 -2
@@ -104,7 +104,7 @@ function BillCards({ bills }: { bills: BillListItem[] }) {
<Link
href={`/bills?policyArea=${encodeURIComponent(b.policyArea)}`}
className="focus-ring label hover:underline"
- style={{ color: "var(--dk-signal)", fontSize: "11px", letterSpacing: "0.1em" }}
+ style={{ color: "var(--dk-signal)", fontSize: "12px", letterSpacing: "0.1em" }}
>
{b.policyArea}
</Link>
@@ -294,7 +294,7 @@ export function BillsBrowser({
<section style={{ marginTop: "32px" }} aria-label="Bills list">
<p className="label" style={{ color: "var(--dk-ash)", marginBottom: "24px" }}>
{total.toLocaleString()} bill{total === 1 ? "" : "s"}
- {isFiltered ? " match" : ""} · showing {bills.length}
+ {isFiltered ? " match" : ""} · showing {Math.min(shown, bills.length).toLocaleString()}
</p>
{bills.length === 0 ? (
components/calendar-browser.tsx +609 -0
@@ -0,0 +1,609 @@
+"use client";
+
+import { useEffect, useState } from "react";
+import { useSearchParams } from "next/navigation";
+import Link from "next/link";
+import { Card, SectionHeader } from "@/components/ui";
+import { SourceLink } from "@/components/source-link";
+import type {
+ CalendarEvent,
+ CalendarKind,
+ CalendarDayKind,
+} from "@/lib/feed";
+import { clsx } from "@/lib/cx";
+
+/* Kind → plain-language + compact mark + muted token. The mark text is always
+ * shown (never color alone) so the grid stays legible in Plain mode and for
+ * color-blind readers (docs/ACCESSIBILITY.md #7; docs/NEUTRALITY.md §Viz). */
+type KindMeta = { value: CalendarKind; plain: string; mark: string; color: string };
+
+const KINDS: KindMeta[] = [
+ { value: "effective", plain: "Takes effect", mark: "Effective", color: "var(--dk-hold)" },
+ { value: "effective_parsed", plain: "Takes effect", mark: "Effective", color: "var(--dk-hold)" },
+ { value: "enacted", plain: "Became law", mark: "Law", color: "var(--dk-yea)" },
+ { value: "vote", plain: "Vote", mark: "Vote", color: "var(--dk-party-d)" },
+ { value: "floor", plain: "Floor action", mark: "Floor", color: "var(--dk-party-r)" },
+ { value: "committee", plain: "Committee", mark: "Committee", color: "var(--dk-party-i)" },
+ { value: "introduced", plain: "Introduced", mark: "Introduced", color: "var(--dk-ash)" },
+ { value: "apportionment", plain: "Apportionment", mark: "Appr.", color: "var(--dk-hold)" },
+];
+
+const KIND_BY_VALUE = new Map<CalendarKind, KindMeta>(KINDS.map((k) => [k.value, k]));
+
+const WEEKDAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
+const MONTH_NAMES = [
+ "January", "February", "March", "April", "May", "June",
+ "July", "August", "September", "October", "November", "December",
+];
+
+function pad(n: number): string {
+ return n < 10 ? `0${n}` : String(n);
+}
+
+function isoToday(): string {
+ const d = new Date();
+ return `${d.getUTCFullYear()}-${pad(d.getUTCMonth() + 1)}-${pad(d.getUTCDate())}`;
+}
+
+/** Parse YYYY-MM into {year,month}; fall back to the current month. */
+function parseMonth(raw: string | undefined): { year: number; month: number } {
+ const m = /^(\d{4})-(\d{1,2})$/.exec(raw ?? "");
+ if (m) {
+ const year = parseInt(m[1], 10);
+ const month = parseInt(m[2], 10);
+ if (year >= 1900 && year <= 9999 && month >= 1 && month <= 12) {
+ return { year, month };
+ }
+ }
+ const d = new Date();
+ return { year: d.getUTCFullYear(), month: d.getUTCMonth() + 1 };
+}
+
+function shiftMonth(year: number, month: number, delta: number): { year: number; month: number } {
+ const zero = month - 1 + delta;
+ const d = new Date(Date.UTC(year, zero, 1));
+ return { year: d.getUTCFullYear(), month: d.getUTCMonth() + 1 };
+}
+
+export function CalendarBrowser() {
+ const sp = useSearchParams();
+ const monthParam = sp.get("month") || undefined;
+ const dayParam = sp.get("day") || undefined;
+ const kindFilter = sp.get("kind") || undefined;
+
+ const { year, month } = parseMonth(monthParam);
+ const todayIso = isoToday();
+
+ const selectedDay =
+ dayParam && /^\d{4}-\d{2}-\d{2}$/.test(dayParam) ? dayParam : undefined;
+
+ const daysInMonth = new Date(Date.UTC(year, month, 0)).getUTCDate();
+ const firstWeekday = new Date(Date.UTC(year, month - 1, 1)).getUTCDay();
+ const from = `${year}-${pad(month)}-01`;
+ const to = `${year}-${pad(month)}-${pad(daysInMonth)}`;
+
+ const [daySummary, setDaySummary] = useState<CalendarDayKind[]>([]);
+ const [dayEvents, setDayEvents] = useState<CalendarEvent[]>([]);
+ const [loading, setLoading] = useState(true);
+
+ // Month grid: per-day kind counts (condensed). Re-fetch on month change.
+ useEffect(() => {
+ setLoading(true);
+ fetch(`/api/calendar?from=${from}&to=${to}&summary=1`)
+ .then((r) => r.json())
+ .then((data) => setDaySummary(data.items ?? []))
+ .catch(() => setDaySummary([]))
+ .finally(() => setLoading(false));
+ }, [from, to]);
+
+ // Full events for the selected day only (detail panel).
+ useEffect(() => {
+ if (!selectedDay) {
+ setDayEvents([]);
+ return;
+ }
+ const params = new URLSearchParams();
+ params.set("from", selectedDay);
+ params.set("to", selectedDay);
+ if (kindFilter) params.set("kind", kindFilter);
+ params.set("limit", "200");
+ fetch(`/api/calendar?${params.toString()}`)
+ .then((r) => r.json())
+ .then((data) => setDayEvents(data.items ?? []))
+ .catch(() => setDayEvents([]));
+ }, [selectedDay, kindFilter]);
+
+ // Filter the grid summary by kind client-side (same as the old SSR page).
+ const filteredSummary = kindFilter
+ ? daySummary.filter((s) => s.kind === kindFilter)
+ : daySummary;
+ const byDate = new Map<string, CalendarDayKind[]>();
+ for (const s of filteredSummary) {
+ const arr = byDate.get(s.date) ?? [];
+ arr.push(s);
+ byDate.set(s.date, arr);
+ }
+
+ const kindCounts = countKindsFromSummary(filteredSummary);
+
+ const prev = shiftMonth(year, month, -1);
+ const next = shiftMonth(year, month, +1);
+ const prevHref = hrefFor(prev, selectedDay, kindFilter);
+ const nextHref = hrefFor(next, selectedDay, kindFilter);
+ const todayHref = hrefFor({ year: todayYear(), month: todayMonth() }, todayIso, kindFilter);
+
+ // Build the grid cells (leading blanks, days, trailing blanks to fill rows).
+ const cells: Array<number | null> = [];
+ for (let i = 0; i < firstWeekday; i++) cells.push(null);
+ for (let d = 1; d <= daysInMonth; d++) cells.push(d);
+ while (cells.length % 7 !== 0) cells.push(null);
+
+ return (
+ <>
+ {/* Month navigation */}
+ <div className="mt-8 flex flex-wrap items-center justify-between gap-3">
+ <div className="flex items-center gap-1">
+ <Link
+ href={prevHref}
+ className="focus-ring label inline-flex min-h-[44px] items-center rounded-[3px] border border-line px-3 text-ash hover:text-paper"
+ aria-label={`Previous month, ${MONTH_NAMES[prev.month - 1]} ${prev.year}`}
+ >
+ ← {MONTH_NAMES[prev.month - 1].slice(0, 3)}
+ </Link>
+ <p
+ className="display px-2 text-paper"
+ aria-live="polite"
+ style={{ fontSize: "clamp(1.25rem, 5vw, 1.875rem)" }}
+ >
+ {MONTH_NAMES[month - 1]} {year}
+ </p>
+ <Link
+ href={nextHref}
+ className="focus-ring label inline-flex min-h-[44px] items-center rounded-[3px] border border-line px-3 text-ash hover:text-paper"
+ aria-label={`Next month, ${MONTH_NAMES[next.month - 1]} ${next.year}`}
+ >
+ {MONTH_NAMES[next.month - 1].slice(0, 3)} →
+ </Link>
+ </div>
+ <Link
+ href={todayHref}
+ className="focus-ring label inline-flex min-h-[44px] items-center rounded-[3px] border border-line px-3 text-ash underline underline-offset-4 hover:text-paper"
+ >
+ Today
+ </Link>
+ </div>
+
+ {/* Kind filter - plain-word chips; selection lives in the URL. */}
+ <div className="mt-5 flex flex-wrap items-center gap-1.5" role="group" aria-label="Filter by kind">
+ <Link
+ href={hrefFor({ year, month }, selectedDay, undefined)}
+ className={clsx(
+ "focus-ring label inline-flex min-h-[44px] items-center rounded-[3px] border px-2.5",
+ !kindFilter ? "border-paper text-paper" : "border-line text-ash hover:text-paper"
+ )}
+ aria-pressed={!kindFilter}
+ >
+ All
+ </Link>
+ {KINDS.filter((k, i) => KINDS.findIndex((x) => x.mark === k.mark) === i).map((k) => {
+ const active = kindFilter === k.value;
+ return (
+ <Link
+ key={k.value}
+ href={hrefFor({ year, month }, selectedDay, active ? undefined : k.value)}
+ className={clsx(
+ "focus-ring label inline-flex min-h-[44px] items-center gap-1.5 rounded-[3px] border px-2.5",
+ active ? "border-paper text-paper" : "border-line text-ash hover:text-paper"
+ )}
+ aria-pressed={active}
+ >
+ <span
+ aria-hidden="true"
+ className="inline-block h-2 w-2 rounded-full"
+ style={{ backgroundColor: k.color }}
+ />
+ {k.plain}
+ </Link>
+ );
+ })}
+ </div>
+
+ <div className="mt-6 grid gap-6 lg:grid-cols-[1.5fr_1fr]">
+ {/* LEFT: month grid (>=md) · agenda list (<md). */}
+ <div>
+ {/* DESKTOP MONTH GRID */}
+ <Card className="hidden overflow-hidden md:block">
+ <table className="w-full table-fixed border-collapse text-left">
+ <caption className="sr-only">
+ Calendar grid for {MONTH_NAMES[month - 1]} {year}. Each cell is a
+ day; select a day to read its events. Rows are weeks, columns
+ are weekdays starting Sunday.
+ </caption>
+ <thead>
+ <tr className="border-b border-line">
+ {WEEKDAYS.map((w) => (
+ <th key={w} scope="col" className="label px-2 py-2.5 font-normal">
+ {w}
+ </th>
+ ))}
+ </tr>
+ </thead>
+ <tbody>
+ {chunk(cells, 7).map((row, ri) => (
+ <tr key={ri} className="border-b border-line last:border-0">
+ {row.map((day, ci) => {
+ if (day == null) {
+ return (
+ <td key={ci} aria-hidden="true" className="bg-ink/40 p-0">
+ <div className="min-h-[5.5rem]" />
+ </td>
+ );
+ }
+ const dateStr = `${year}-${pad(month)}-${pad(day)}`;
+ const evs = byDate.get(dateStr) ?? [];
+ const evCount = dayEventCount(evs);
+ const isToday = dateStr === todayIso;
+ const isSelected = dateStr === selectedDay;
+ const marks = distinctMarks(evs);
+ const marksLabel = marks.map((m) => m.plain).join(", ");
+ return (
+ <td key={ci} className="border-r border-line p-0 align-top last:border-r-0">
+ <Link
+ href={hrefFor({ year, month }, dateStr, kindFilter)}
+ aria-label={`${day} ${MONTH_NAMES[month - 1]} ${year}${evCount ? `, ${evCount} event${evCount === 1 ? "" : "s"}${marksLabel ? `: ${marksLabel}` : ""}` : ", no events"}`}
+ aria-current={isSelected ? "true" : undefined}
+ className={clsx(
+ "focus-ring flex h-full min-h-[5.5rem] flex-col justify-between p-2 no-underline",
+ isSelected ? "bg-graphite" : "hover:bg-graphite/50"
+ )}
+ >
+ <span
+ className={clsx(
+ "flex h-7 w-7 items-center justify-center rounded-full font-mono text-[13px] tabular-nums",
+ isToday
+ ? "bg-signal text-ink"
+ : evCount
+ ? "text-paper"
+ : "text-ash-2"
+ )}
+ >
+ {day}
+ </span>
+ {evCount > 0 ? (
+ <span className="mt-1.5 flex items-center gap-1.5" aria-hidden="true">
+ <span className="flex flex-wrap gap-1">
+ {marks.slice(0, 4).map((m, i) => (
+ <span
+ key={i}
+ className="inline-block h-2 w-2 shrink-0 rounded-full"
+ style={{ backgroundColor: m.color }}
+ title={m.plain}
+ />
+ ))}
+ </span>
+ <span className="label !text-[10px] tabular-nums text-ash-2">
+ {evCount}
+ </span>
+ </span>
+ ) : null}
+ </Link>
+ </td>
+ );
+ })}
+ </tr>
+ ))}
+ </tbody>
+ </table>
+ <p className="label border-t border-line px-3 py-2 text-ash-2">
+ {loading
+ ? "Loading events…"
+ : `${filteredSummary.reduce((s, e) => s + e.count, 0)} event${filteredSummary.reduce((s, e) => s + e.count, 0) === 1 ? "" : "s"} this month. Select a day to read them.`}
+ </p>
+ </Card>
+
+ {/* MOBILE AGENDA */}
+ <div className="md:hidden">
+ <MobileAgenda
+ daySummary={filteredSummary}
+ kindByValue={KIND_BY_VALUE}
+ selectedDay={selectedDay}
+ dayEvents={dayEvents}
+ year={year}
+ month={month}
+ kindFilter={kindFilter}
+ loading={loading}
+ />
+ </div>
+ </div>
+
+ {/* SIDE: legend + selected day */}
+ <div className="space-y-6">
+ <Card className="p-5">
+ <p className="label mb-3">Legend</p>
+ <ul className="space-y-2">
+ {KINDS.filter((k, i) => KINDS.findIndex((x) => x.mark === k.mark) === i).map((k) => (
+ <li key={k.value} className="flex items-center gap-2.5">
+ <span
+ aria-hidden="true"
+ className="inline-block h-2.5 w-2.5 shrink-0 rounded-full"
+ style={{ backgroundColor: k.color }}
+ />
+ <span className="font-mono text-[12px] text-paper">{k.plain}</span>
+ <span className="label text-ash-2">{kindCounts.get(k.value) ?? 0}</span>
+ </li>
+ ))}
+ </ul>
+ <p className="label mt-4 text-ash-2">
+ Colors are a cue only. Each mark carries its name, so the grid
+ reads clearly in Plain mode too.
+ </p>
+ </Card>
+
+ <div>
+ <SectionHeader
+ eyebrow={selectedDay ? "Selected day" : "Pick a day"}
+ title={selectedDay ? formatLong(selectedDay) : "Day events"}
+ />
+ {!selectedDay ? (
+ <Card className="p-5">
+ <p className="font-mono text-[13px] text-ash-2">
+ Select a day in the grid to read its events. Each event links
+ to its official source.
+ </p>
+ </Card>
+ ) : dayEvents.length === 0 ? (
+ <Card className="p-5">
+ <p className="font-mono text-[13px] text-ash-2">
+ No events on record for {formatLong(selectedDay)}
+ {kindFilter ? " for this filter" : ""}.
+ </p>
+ </Card>
+ ) : (
+ <Card className="divide-y divide-[color:var(--dk-line)]">
+ {dayEvents.map((e) => {
+ const k = KIND_BY_VALUE.get(e.kind);
+ return (
+ <div key={e.id} className="p-4">
+ <div className="flex flex-wrap items-center gap-2">
+ {k ? (
+ <span
+ className="pill"
+ style={{
+ color: k.color,
+ borderColor: `color-mix(in srgb, ${k.color} 45%, transparent)`,
+ }}
+ >
+ {k.plain}
+ </span>
+ ) : (
+ <span className="pill text-ash">{e.kind}</span>
+ )}
+ <span className="label text-ash-2">{e.refType}</span>
+ </div>
+ <p className="mt-2 font-mono text-[12px] leading-relaxed text-paper">
+ {e.href ? (
+ <Link
+ href={e.href}
+ className="focus-ring rounded-sm underline underline-offset-4 hover:text-paper"
+ >
+ {e.title}
+ </Link>
+ ) : (
+ e.title
+ )}
+ </p>
+ <div className="mt-2">
+ <SourceLink url={e.sourceUrl} />
+ </div>
+ </div>
+ );
+ })}
+ </Card>
+ )}
+ </div>
+ </div>
+ </div>
+ </>
+ );
+}
+
+/* ---------------- helpers ---------------- */
+
+function hrefFor(
+ ym: { year: number; month: number },
+ day: string | undefined,
+ kind: string | undefined
+): string {
+ const params = new URLSearchParams();
+ params.set("month", `${ym.year}-${pad(ym.month)}`);
+ if (day) params.set("day", day);
+ if (kind) params.set("kind", kind);
+ return `/calendar?${params.toString()}`;
+}
+
+function todayYear(): number {
+ return new Date().getUTCFullYear();
+}
+function todayMonth(): number {
+ return new Date().getUTCMonth() + 1;
+}
+
+function chunk<T>(arr: T[], size: number): T[][] {
+ const out: T[][] = [];
+ for (let i = 0; i < arr.length; i += size) out.push(arr.slice(i, i + size));
+ return out;
+}
+
+/** Distinct kind marks for a day cell (uses condensed summary). */
+function distinctMarks(evs: CalendarDayKind[]): KindMeta[] {
+ const seen = new Set<string>();
+ const out: KindMeta[] = [];
+ for (const e of evs) {
+ const m = KIND_BY_VALUE.get(e.kind);
+ if (m && !seen.has(m.mark)) {
+ seen.add(m.mark);
+ out.push(m);
+ }
+ }
+ return out;
+}
+
+/** Total event count for a day cell (sums counts across kinds). */
+function dayEventCount(evs: CalendarDayKind[]): number {
+ return evs.reduce((s, e) => s + e.count, 0);
+}
+
+/** Kind counts from the condensed day summary (used by the legend). */
+function countKindsFromSummary(summary: CalendarDayKind[]): Map<CalendarKind, number> {
+ const m = new Map<CalendarKind, number>();
+ for (const s of summary) m.set(s.kind, (m.get(s.kind) ?? 0) + s.count);
+ return m;
+}
+
+function formatLong(iso: string): string {
+ const d = new Date(iso + "T00:00:00Z");
+ if (Number.isNaN(d.getTime())) return iso;
+ return d.toLocaleDateString("en-US", {
+ timeZone: "UTC",
+ weekday: "long",
+ year: "numeric",
+ month: "long",
+ day: "numeric",
+ });
+}
+
+/**
+ * Mobile agenda - per-day event counts + kind dots, with full event details
+ * for the selected day. Uses the condensed day summary; tap a day to navigate
+ * (URL-driven, same as the desktop grid).
+ */
+function MobileAgenda({
+ daySummary,
+ kindByValue,
+ selectedDay,
+ dayEvents,
+ year,
+ month,
+ kindFilter,
+ loading,
+}: {
+ daySummary: CalendarDayKind[];
+ kindByValue: Map<CalendarKind, KindMeta>;
+ selectedDay: string | undefined;
+ dayEvents: CalendarEvent[];
+ year: number;
+ month: number;
+ kindFilter: string | undefined;
+ loading: boolean;
+}) {
+ const byDate = new Map<string, CalendarDayKind[]>();
+ for (const s of daySummary) {
+ const arr = byDate.get(s.date) ?? [];
+ arr.push(s);
+ byDate.set(s.date, arr);
+ }
+ const days = Array.from(byDate.keys()).sort();
+
+ if (loading) {
+ return (
+ <Card className="p-5">
+ <p className="font-mono text-[13px] text-ash-2">Loading events…</p>
+ </Card>
+ );
+ }
+
+ if (days.length === 0) {
+ return (
+ <Card className="p-5">
+ <p className="font-mono text-[13px] text-ash-2">
+ No events on record for this month. Month navigation above moves to
+ other months; each event will link to its official source.
+ </p>
+ </Card>
+ );
+ }
+
+ return (
+ <div className="space-y-2">
+ {days.map((d) => {
+ const evs = byDate.get(d)!;
+ const count = dayEventCount(evs);
+ const marks = distinctMarks(evs);
+ const isSelected = d === selectedDay;
+ return (
+ <section key={d}>
+ <Link
+ href={hrefFor({ year, month }, d, kindFilter)}
+ className={`focus-ring block rounded-[3px] border p-3 no-underline hover:bg-graphite/50 ${isSelected ? "border-paper bg-graphite" : "border-line"}`}
+ aria-current={isSelected ? "true" : undefined}
+ >
+ <div className="flex items-center justify-between gap-3">
+ <span className="font-mono text-[13px] text-paper">{formatLong(d)}</span>
+ <span className="label text-ash-2 shrink-0">{count} event{count === 1 ? "" : "s"}</span>
+ </div>
+ <div className="mt-1.5 flex flex-wrap gap-1">
+ {marks.map((m, i) => (
+ <span
+ key={i}
+ className="pill !px-1.5 !py-0"
+ style={{
+ color: m.color,
+ borderColor: `color-mix(in srgb, ${m.color} 45%, transparent)`,
+ }}
+ >
+ {m.plain}
+ </span>
+ ))}
+ </div>
+ </Link>
+ {isSelected && dayEvents.length > 0 && (
+ <div className="mt-2 space-y-2 pl-3">
+ {dayEvents.map((e) => {
+ const k = kindByValue.get(e.kind);
+ return (
+ <article
+ key={e.id}
+ className="rounded-[3px] border border-line bg-surface p-3"
+ >
+ <div className="flex flex-wrap items-center gap-2">
+ {k ? (
+ <span
+ className="pill !py-0.5"
+ style={{
+ color: k.color,
+ borderColor: `color-mix(in srgb, ${k.color} 45%, transparent)`,
+ }}
+ >
+ {k.plain}
+ </span>
+ ) : (
+ <span className="pill !py-0.5 text-ash">{e.kind}</span>
+ )}
+ <span className="label text-ash-2">{e.refType}</span>
+ </div>
+ <p className="mt-2 break-words font-mono text-[12px] leading-relaxed text-paper">
+ {e.href ? (
+ <Link
+ href={e.href}
+ className="focus-ring rounded-sm underline underline-offset-4 hover:text-paper"
+ >
+ {e.title}
+ </Link>
+ ) : (
+ e.title
+ )}
+ </p>
+ <div className="mt-2">
+ <SourceLink url={e.sourceUrl} />
+ </div>
+ </article>
+ );
+ })}
+ </div>
+ )}
+ </section>
+ );
+ })}
+ </div>
+ );
+}
components/cite-this.tsx +1 -1
@@ -208,7 +208,7 @@ export function CiteThis({
<div className="flex items-center justify-between border-t border-line px-4 py-3 sm:px-5">
<p className="label text-ash-2">
{/* Nudge for educators/students - neutral phrasing */}
- APA · MLA · Chicago - copy and paste into your document or citation manager.
+ APA · MLA · Chicago - copy and paste into your document or citation manager.
</p>
<button
type="button"
components/data-table.tsx +7 -6
@@ -73,7 +73,7 @@ export function DataTable({
scroll context so the table never pushes the page wider than the card.
table-fixed prevents auto-layout from expanding columns beyond the
container width; break-words on each td ensures long text wraps. */}
- <div className="hidden w-full overflow-x-auto md:block">
+ <div className="hidden w-full overflow-x-auto lg:block">
<table className="w-full table-fixed border-collapse text-left">
<caption className="sr-only">{caption}</caption>
<colgroup>
@@ -106,7 +106,7 @@ export function DataTable({
{visibleRows.map((r) => (
<tr
key={r.key}
- className="border-b border-line font-mono text-[12px] transition-colors last:border-0 hover:bg-graphite/60"
+ className="border-b border-line font-mono text-[13px] transition-colors last:border-0 hover:bg-graphite/60"
>
{columns.map((c) => (
<td
@@ -125,9 +125,10 @@ export function DataTable({
</table>
</div>
- {/* MOBILE (<md): stacked labeled cards. */}
+ {/* MOBILE + TABLET (<lg): stacked labeled cards. iPad-portrait (768) gets
+ the clean card layout instead of a crammed table. */}
<ul
- className="divide-y divide-[color:var(--dk-line)] md:hidden"
+ className="divide-y divide-[color:var(--dk-line)] lg:hidden"
aria-label={cardLabel ?? caption}
>
{visibleRows.map((r) => (
@@ -148,7 +149,7 @@ export function DataTable({
<dt className="label w-[5.5rem] shrink-0 pt-0.5 normal-case">
{c.header}
</dt>
- <dd className="min-w-0 flex-1 break-words text-left font-mono text-[12px] text-ash">
+ <dd className="min-w-0 flex-1 break-words text-left font-mono text-[13px] text-ash">
{val}
</dd>
</div>
@@ -167,7 +168,7 @@ export function DataTable({
<button
type="button"
onClick={() => setShown((n) => n + (pageSize ?? 0))}
- className="focus-ring label inline-flex min-h-[40px] items-center rounded-[3px] border border-line px-3 text-ash underline underline-offset-4 transition-colors hover:border-paper hover:text-paper"
+ className="focus-ring label inline-flex min-h-[44px] items-center rounded-[3px] border border-line px-3 text-ash underline underline-offset-4 transition-colors hover:border-paper hover:text-paper"
>
Show {Math.min(pageSize, remaining).toLocaleString()} more
</button>
components/executive-browser.tsx +6 -3
@@ -124,10 +124,13 @@ function ExecutiveTable({
typeLabels: Record<string, string>;
}) {
const columns: Column[] = [
- { key: "date", header: "Date" },
- { key: "type", header: "Type", hideOnCard: true },
+ // Fixed widths on date/type so the flexible title column soaks up the
+ // remainder (else table-fixed splits leftover space equally and date/type
+ // sit in empty space while source wraps). Source widened so the host fits.
+ { key: "date", header: "Date", width: "6.5rem" },
+ { key: "type", header: "Type", hideOnCard: true, width: "9rem" },
{ key: "title", header: "Title", cardTitle: true },
- { key: "source", header: "Source", align: "right" },
+ { key: "source", header: "Source", align: "right", width: "13rem" },
];
const data: Row[] = items.map((e) => ({
key: e.id,
components/laws-browser.tsx +220 -0
@@ -0,0 +1,220 @@
+"use client";
+
+import { useEffect, useState } from "react";
+import { useSearchParams } from "next/navigation";
+import Link from "next/link";
+import { SourceLink } from "@/components/source-link";
+import { ArchiveLink } from "@/components/record-archive";
+import { ordinal } from "@/lib/format";
+import {
+ SearchFilterBar,
+ FilterField,
+ FilterInput,
+ FilterSelect,
+ FilterSubmit,
+} from "@/components/search-filter-bar";
+import type { LawListItem } from "@/lib/feed";
+
+function LawsSkeleton() {
+ return (
+ <div className="mt-8 space-y-4">
+ <div className="h-14 w-full animate-pulse rounded-sm bg-graphite" />
+ <div className="mt-8 mb-4 h-4 w-40 animate-pulse rounded-sm bg-graphite" />
+ <div className="space-y-3">
+ {Array.from({ length: 12 }).map((_, i) => (
+ <div key={i} className="h-28 w-full animate-pulse rounded-sm bg-graphite" />
+ ))}
+ </div>
+ </div>
+ );
+}
+
+/** Trim a summary to a readable snippet. */
+function snippet(s: string, max = 240): string {
+ if (!s) return "";
+ const clean = s.trim();
+ return clean.length <= max ? clean : clean.slice(0, max - 1).trimEnd() + "…";
+}
+
+function LawCards({ laws }: { laws: LawListItem[] }) {
+ return (
+ <ul
+ style={{ display: "flex", flexDirection: "column", gap: "12px", listStyle: "none", padding: 0, margin: 0 }}
+ role="list"
+ >
+ {laws.map((law) => (
+ <li key={law.id}>
+ <div
+ style={{
+ borderRadius: "10px",
+ border: "1px solid var(--dk-line)",
+ background: "var(--dk-surface)",
+ padding: "clamp(20px, 3vw, 28px)",
+ }}
+ >
+ {/* PL number - JetBrains Mono, the anchor */}
+ <div
+ style={{
+ display: "flex",
+ flexWrap: "wrap",
+ alignItems: "baseline",
+ justifyContent: "space-between",
+ gap: "12px",
+ marginBottom: "14px",
+ }}
+ >
+ <p
+ className="font-mono text-paper"
+ style={{ fontSize: "clamp(1.0625rem, 2.4vw, 1.25rem)", fontWeight: 700, letterSpacing: "0.01em" }}
+ >
+ Public Law {law.plNumber || "-"}
+ </p>
+ <span
+ className="font-mono"
+ style={{ fontSize: "12px", color: "var(--dk-ash-2)", letterSpacing: "0.06em" }}
+ >
+ {ordinal(law.congress)} Congress
+ </span>
+ </div>
+
+ {/* Enacting bill */}
+ <p className="label" style={{ color: "var(--dk-ash-2)", marginBottom: "6px" }}>
+ Enacted from{" "}
+ <Link
+ href={law.billHref}
+ className="focus-ring font-mono hover:text-paper underline underline-offset-4"
+ style={{ color: "var(--dk-signal)" }}
+ >
+ {law.billNumber}
+ </Link>
+ </p>
+
+ {/* Title */}
+ <Link
+ href={law.billHref}
+ className="focus-ring font-prose text-paper"
+ style={{
+ display: "block",
+ fontSize: "clamp(1rem, 1.8vw, 1.125rem)",
+ lineHeight: 1.5,
+ textDecoration: "underline",
+ textDecorationColor: "var(--dk-line)",
+ textUnderlineOffset: "3px",
+ marginBottom: law.summary ? "12px" : "16px",
+ }}
+ >
+ {law.title}
+ </Link>
+
+ {/* Summary snippet */}
+ {law.summary ? (
+ <p
+ className="font-prose"
+ style={{ fontSize: "0.9375rem", lineHeight: 1.6, color: "var(--dk-ash)", marginBottom: "16px" }}
+ >
+ {snippet(law.summary)}
+ </p>
+ ) : null}
+
+ {/* Source row */}
+ <div style={{ display: "flex", flexWrap: "wrap", alignItems: "center", gap: "12px" }}>
+ <SourceLink url={law.sourceUrl} label="Official record" />
+ {law.archivePdfUrl ? (
+ <ArchiveLink url={law.archivePdfUrl} label="Archived text - Docket" />
+ ) : null}
+ </div>
+ </div>
+ </li>
+ ))}
+ </ul>
+ );
+}
+
+export function LawsBrowser() {
+ const sp = useSearchParams();
+ const q = sp.get("q") ?? "";
+ const sort = sp.get("sort") ?? "";
+
+ const [laws, setLaws] = useState<LawListItem[]>([]);
+ const [total, setTotal] = useState(0);
+ const [loading, setLoading] = useState(true);
+
+ useEffect(() => {
+ setLoading(true);
+ const params = new URLSearchParams();
+ if (q) params.set("q", q);
+ if (sort) params.set("sort", sort);
+ params.set("limit", "200");
+
+ fetch(`/api/laws?${params.toString()}`)
+ .then((r) => r.json())
+ .then((data) => {
+ setLaws(data.items ?? []);
+ setTotal(data.total ?? 0);
+ })
+ .catch(() => {
+ setLaws([]);
+ setTotal(0);
+ })
+ .finally(() => setLoading(false));
+ }, [q, sort]);
+
+ const isFiltered = !!(q || sort);
+
+ if (loading) {
+ return <LawsSkeleton />;
+ }
+
+ return (
+ <div>
+ <div className="mt-8">
+ <form method="get" action="/laws" className="contents">
+ <SearchFilterBar
+ count={total}
+ countLabel="law"
+ clearHref={isFiltered ? "/laws" : undefined}
+ >
+ <FilterField label="Search laws" span="2">
+ <FilterInput
+ name="q"
+ defaultValue={q}
+ placeholder="Law number, bill number, or title…"
+ aria-label="Search public laws"
+ />
+ </FilterField>
+ <FilterField label="Sort by">
+ <FilterSelect
+ name="sort"
+ defaultValue={sort}
+ aria-label="Sort laws"
+ options={[
+ { value: "", label: "Most recent" },
+ { value: "oldest", label: "Oldest first" },
+ { value: "number", label: "By law number" },
+ ]}
+ />
+ </FilterField>
+ <FilterSubmit />
+ </SearchFilterBar>
+ </form>
+ </div>
+
+ <section style={{ marginTop: "32px" }} aria-label="Public laws list">
+ <p className="label" style={{ color: "var(--dk-ash)", marginBottom: "24px" }}>
+ {total.toLocaleString()} public law{total === 1 ? "" : "s"}
+ {isFiltered ? " match" : ""} · showing {laws.length}
+ </p>
+
+ {laws.length === 0 ? (
+ <div className="rounded-[6px] border border-line bg-surface p-8" role="status">
+ <p className="font-prose" style={{ fontSize: "1.0625rem", color: "var(--dk-ash)" }}>
+ No public laws match this search in the current record.
+ </p>
+ </div>
+ ) : (
+ <LawCards laws={laws} />
+ )}
+ </section>
+ </div>
+ );
+}
components/meetings-browser.tsx +25 -7
@@ -60,19 +60,36 @@ function formatDayHeader(iso: string): string {
});
}
+/** Display the meeting time accurately. The ingest already stores a 12-hour
+ * ET clock string (e.g. "02:30 PM", "2:00 PM"). The previous version re-parsed
+ * the hour as 24-hour and re-derived AM/PM, so "2:00 PM" became "2:00 AM" - the
+ * bogus "2:00 AM / 2:15 AM" bug. Now we trust an existing AM/PM string (just
+ * normalizing the leading zero), and only convert a bare 24-hour value. */
function formatTime(m: Meeting): string {
const t = (m.time || "").trim();
if (!t) return "";
- // Normalize "14:00" / "14:00:00" → "2:00 PM"; pass through anything odd.
+ // Already a 12-hour clock with a meridiem - normalize and trust it.
+ const ampm = t.match(/^(\d{1,2}):(\d{2})\s*([AaPp])\.?[Mm]\.?$/);
+ if (ampm) {
+ const h = parseInt(ampm[1], 10);
+ const h12 = h === 0 ? 12 : h;
+ return `${h12}:${ampm[2]} ${ampm[3].toUpperCase()}M`;
+ }
+ // Bare 24-hour "HH:MM" / "HH:MM:SS" → convert to 12-hour.
const parts = t.split(":");
const hh = parseInt(parts[0], 10);
const mm = parseInt(parts[1], 10);
- if (!Number.isFinite(hh) || !Number.isFinite(mm)) return t;
+ if (!Number.isFinite(hh) || !Number.isFinite(mm)) return "";
const period = hh < 12 ? "AM" : "PM";
const h12 = ((hh + 11) % 12) + 1;
return `${h12}:${String(mm).padStart(2, "0")} ${period}`;
}
+/** Normalize "--" double-hyphens to a typographic en-dash for display. */
+function dedash(s: string): string {
+ return s ? s.replace(/--/g, "-") : s;
+}
+
export function MeetingsBrowser({
seed,
committees,
@@ -229,7 +246,7 @@ export function MeetingsBrowser({
<Card className="p-8 text-center">
<p className="display text-2xl text-paper">No meetings on record yet.</p>
<p className="prose-lead mx-auto mt-3 max-w-md text-ash">
- Congressional committee and floor meetings will appear here - when
+ Congressional committee and floor meetings will appear here - when
committees hold them, each sealed to its official schedule.
</p>
</Card>
@@ -304,13 +321,14 @@ function MeetingCard({ m }: { m: Meeting }) {
) : null}
</div>
<p className="font-mono text-[13px] text-paper">
- {m.committee || "Committee not on record"}
+ {m.committee ? dedash(m.committee) : "Committee not on record"}
</p>
{m.subject ? (
- <p className="font-mono text-[12px] leading-relaxed text-ash">{m.subject}</p>
+ <p className="font-mono text-[13px] leading-relaxed text-ash">{dedash(m.subject)}</p>
) : null}
- {m.title ? (
- <p className="font-mono text-[12px] leading-relaxed text-ash-2">{m.title}</p>
+ {/* Suppress the title line when it just repeats the subject (duplicate text). */}
+ {m.title && m.title.trim() !== m.subject.trim() ? (
+ <p className="font-mono text-[13px] leading-relaxed text-ash-2">{dedash(m.title)}</p>
) : null}
{m.location ? (
<p className="label text-ash-2">Location · {m.location}</p>
components/member-vote-year.tsx +1 -1
@@ -95,7 +95,7 @@ export function MemberVoteYear({
)}
{error && (
<p className="px-4 py-3 font-mono text-[13px] text-ash-2">
- Failed to load - open again to retry.
+ Failed to load - open again to retry.
</p>
)}
{votes && <VoteTable votes={votes} />}
components/owl.tsx +141 -0
@@ -0,0 +1,141 @@
+/**
+ * Archie - the Docket archivist owl. A friendly, neutral record-keeper:
+ * reading glasses, perched on a ledger. Flat geometric vector in the brand
+ * palette (dark body, paper eyes, brass glasses). Self-contained colors so it
+ * reads on both the dark and light (parchment) page backgrounds.
+ *
+ * Used sparingly and quietly - Docket is a trust product, the owl is a warm
+ * accent, never a loud cartoon. See docs/BRAND.md.
+ *
+ * OwlBust - head/shoulders, for tight spots (~48px up)
+ * OwlFull - full mascot perched on a ledger
+ * OwlAlt - searching pose (empty magnifier), for "no results" empty states
+ */
+
+type OwlProps = { size?: number; className?: string; title?: string };
+
+export function OwlBust({ size = 56, className, title = "Docket archivist owl" }: OwlProps) {
+ return (
+ <svg
+ width={size}
+ height={size}
+ viewBox="0 0 120 120"
+ className={className}
+ role="img"
+ aria-label={title}
+ xmlns="http://www.w3.org/2000/svg"
+ >
+ <path d="M34 43 Q36 19 50 35 Q43 38 38 43 Z" fill="#2A2E35" />
+ <path d="M86 43 Q84 19 70 35 Q77 38 82 43 Z" fill="#2A2E35" />
+ <ellipse cx="60" cy="62" rx="43" ry="43" fill="#2A2E35" />
+ <ellipse cx="60" cy="87" rx="25" ry="21" fill="#363B44" />
+ <circle cx="44" cy="58" r="22" fill="#3A4049" />
+ <circle cx="76" cy="58" r="22" fill="#3A4049" />
+ <path d="M22 50 L13 46" stroke="#E0A53B" strokeWidth="3" strokeLinecap="round" fill="none" />
+ <path d="M98 50 L107 46" stroke="#E0A53B" strokeWidth="3" strokeLinecap="round" fill="none" />
+ <circle cx="44" cy="57" r="15" fill="#F2EFE9" />
+ <circle cx="76" cy="57" r="15" fill="#F2EFE9" />
+ <circle cx="46" cy="59" r="7.5" fill="#0B0C0E" />
+ <circle cx="74" cy="59" r="7.5" fill="#0B0C0E" />
+ <circle cx="48.5" cy="56.5" r="2.2" fill="#F2EFE9" />
+ <circle cx="76.5" cy="56.5" r="2.2" fill="#F2EFE9" />
+ <circle cx="44" cy="57" r="17" fill="none" stroke="#E0A53B" strokeWidth="3.5" />
+ <circle cx="76" cy="57" r="17" fill="none" stroke="#E0A53B" strokeWidth="3.5" />
+ <path d="M58 56 Q60 51 62 56" fill="none" stroke="#E0A53B" strokeWidth="3" strokeLinecap="round" />
+ <path d="M60 64 L54.5 70 L60 76 L65.5 70 Z" fill="#C9A66B" />
+ <path d="M34 38 Q40 33 47 35" stroke="#F2EFE9" strokeWidth="2.4" strokeLinecap="round" fill="none" opacity="0.85" />
+ <path d="M86 38 Q80 33 73 35" stroke="#F2EFE9" strokeWidth="2.4" strokeLinecap="round" fill="none" opacity="0.85" />
+ </svg>
+ );
+}
+
+export function OwlFull({ size = 200, className, title = "Docket archivist owl" }: OwlProps) {
+ return (
+ <svg
+ width={size}
+ height={size}
+ viewBox="0 0 240 240"
+ className={className}
+ role="img"
+ aria-label={title}
+ xmlns="http://www.w3.org/2000/svg"
+ >
+ <path d="M86 58 Q90 22 118 50 Q104 53 95 60 Z" fill="#2A2E35" />
+ <path d="M154 58 Q150 22 122 50 Q136 53 145 60 Z" fill="#2A2E35" />
+ <ellipse cx="120" cy="118" rx="62" ry="80" fill="#2A2E35" />
+ <path d="M64 96 Q50 140 66 188 Q60 144 74 104 Z" fill="#22262D" />
+ <path d="M176 96 Q190 140 174 188 Q180 144 166 104 Z" fill="#22262D" />
+ <ellipse cx="120" cy="140" rx="40" ry="50" fill="#363B44" />
+ <path d="M108 150 Q120 156 132 150" stroke="#2E333B" strokeWidth="2.4" fill="none" opacity="0.7" />
+ <path d="M108 164 Q120 170 132 164" stroke="#2E333B" strokeWidth="2.4" fill="none" opacity="0.7" />
+ <circle cx="100" cy="92" r="35" fill="#3A4049" />
+ <circle cx="140" cy="92" r="35" fill="#3A4049" />
+ <path d="M74 80 L60 74" stroke="#E0A53B" strokeWidth="4.2" strokeLinecap="round" fill="none" />
+ <path d="M166 80 L180 74" stroke="#E0A53B" strokeWidth="4.2" strokeLinecap="round" fill="none" />
+ <circle cx="99" cy="90" r="22" fill="#F2EFE9" />
+ <circle cx="141" cy="90" r="22" fill="#F2EFE9" />
+ <circle cx="102" cy="93" r="11" fill="#0B0C0E" />
+ <circle cx="138" cy="93" r="11" fill="#0B0C0E" />
+ <circle cx="106" cy="88.5" r="3.2" fill="#F2EFE9" />
+ <circle cx="142" cy="88.5" r="3.2" fill="#F2EFE9" />
+ <circle cx="99" cy="90" r="25" fill="none" stroke="#E0A53B" strokeWidth="5" />
+ <circle cx="141" cy="90" r="25" fill="none" stroke="#E0A53B" strokeWidth="5" />
+ <path d="M115 89 Q120 81 125 89" fill="none" stroke="#E0A53B" strokeWidth="4.2" strokeLinecap="round" />
+ <path d="M120 100 L111 110 L120 121 L129 110 Z" fill="#C9A66B" />
+ <path d="M77 62 Q90 54 105 58" stroke="#F2EFE9" strokeWidth="3.6" strokeLinecap="round" fill="none" opacity="0.85" />
+ <path d="M163 62 Q150 54 135 58" stroke="#F2EFE9" strokeWidth="3.6" strokeLinecap="round" fill="none" opacity="0.85" />
+ <g stroke="#C9A66B" strokeWidth="4" strokeLinecap="round" fill="none">
+ <path d="M104 182 L100 192 M104 182 L106 193 M104 182 L112 191" />
+ <path d="M136 182 L128 191 M136 182 L134 193 M136 182 L140 192" />
+ </g>
+ <rect x="62" y="190" width="116" height="30" rx="5" fill="#3A4049" />
+ <rect x="62" y="190" width="12" height="30" rx="5" fill="#E0A53B" />
+ <line x1="84" y1="212" x2="170" y2="212" stroke="#F2EFE9" strokeWidth="2.6" strokeLinecap="round" opacity="0.5" />
+ <line x1="84" y1="216.5" x2="170" y2="216.5" stroke="#F2EFE9" strokeWidth="2" strokeLinecap="round" opacity="0.3" />
+ </svg>
+ );
+}
+
+export function OwlAlt({ size = 200, className, title = "Docket owl searching the record" }: OwlProps) {
+ return (
+ <svg
+ width={size}
+ height={size}
+ viewBox="0 0 240 240"
+ className={className}
+ role="img"
+ aria-label={title}
+ xmlns="http://www.w3.org/2000/svg"
+ >
+ <circle cx="178" cy="70" r="30" fill="#121316" stroke="#E0A53B" strokeWidth="7" />
+ <line x1="200" y1="92" x2="218" y2="112" stroke="#E0A53B" strokeWidth="9" strokeLinecap="round" />
+ <path d="M72 70 Q76 36 102 62 Q89 65 81 72 Z" fill="#2A2E35" />
+ <path d="M136 70 Q132 36 106 62 Q119 65 127 72 Z" fill="#2A2E35" />
+ <ellipse cx="104" cy="128" rx="58" ry="74" fill="#2A2E35" />
+ <ellipse cx="104" cy="150" rx="38" ry="46" fill="#363B44" />
+ <path d="M150 132 Q176 110 186 92 Q172 118 150 120 Z" fill="#22262D" />
+ <path d="M52 108 Q40 150 56 196 Q50 154 62 116 Z" fill="#22262D" />
+ <circle cx="86" cy="104" r="33" fill="#3A4049" />
+ <circle cx="122" cy="104" r="33" fill="#3A4049" />
+ <path d="M62 93 L49 88" stroke="#E0A53B" strokeWidth="4" strokeLinecap="round" fill="none" />
+ <path d="M146 93 L159 88" stroke="#E0A53B" strokeWidth="4" strokeLinecap="round" fill="none" />
+ <circle cx="85" cy="102" r="21" fill="#F2EFE9" />
+ <circle cx="123" cy="102" r="21" fill="#F2EFE9" />
+ <circle cx="88" cy="105" r="10.5" fill="#0B0C0E" />
+ <circle cx="120" cy="105" r="10.5" fill="#0B0C0E" />
+ <circle cx="91.5" cy="100.5" r="3" fill="#F2EFE9" />
+ <circle cx="123.5" cy="100.5" r="3" fill="#F2EFE9" />
+ <circle cx="85" cy="102" r="24" fill="none" stroke="#E0A53B" strokeWidth="4.8" />
+ <circle cx="123" cy="102" r="24" fill="none" stroke="#E0A53B" strokeWidth="4.8" />
+ <path d="M100 101 Q104 93 109 101" fill="none" stroke="#E0A53B" strokeWidth="4" strokeLinecap="round" />
+ <path d="M104 111 L95.5 121 L104 131 L112.5 121 Z" fill="#C9A66B" />
+ <path d="M64 76 Q76 68 90 72" stroke="#F2EFE9" strokeWidth="3.4" strokeLinecap="round" fill="none" opacity="0.85" />
+ <path d="M144 76 Q132 68 118 72" stroke="#F2EFE9" strokeWidth="3.4" strokeLinecap="round" fill="none" opacity="0.85" />
+ <g stroke="#C9A66B" strokeWidth="4" strokeLinecap="round" fill="none">
+ <path d="M88 198 L84 208 M88 198 L90 209 M88 198 L96 207" />
+ <path d="M120 198 L112 207 M120 198 L118 209 M120 198 L124 208" />
+ </g>
+ <line x1="60" y1="210" x2="148" y2="210" stroke="#F2EFE9" strokeWidth="3" strokeLinecap="round" opacity="0.4" />
+ </svg>
+ );
+}
components/record-archive.tsx +130 -0
@@ -0,0 +1,130 @@
+import { clsx } from "@/lib/cx";
+import { SourceLink } from "@/components/source-link";
+
+/**
+ * Link to Docket's own preserved copy of a record on archive.usdocket.org.
+ *
+ * This is intentionally NOT a SourceLink: the SourceLink component refuses any
+ * host outside the .gov allowlist (neutrality rule #1 - every *fact* cites an
+ * official government source). The archived copy is Docket's mirror, not the
+ * official source, so it gets its own subtle, clearly-labeled treatment and the
+ * official .gov link always appears alongside it.
+ */
+export function ArchiveLink({
+ url,
+ label = "Archived copy - Docket",
+ className,
+}: {
+ url: string;
+ label?: string;
+ className?: string;
+}) {
+ const safeHref = /^https?:\/\//i.test(url) ? url : "#";
+ return (
+ <a
+ href={safeHref}
+ target="_blank"
+ rel="noopener noreferrer nofollow"
+ className={clsx(
+ "focus-ring label inline-flex min-h-[40px] max-w-full flex-wrap items-center gap-x-1 gap-y-0.5 rounded-[3px] border border-line px-2 py-1 underline underline-offset-4 transition-colors text-ash hover:text-paper hover:border-paper",
+ className
+ )}
+ title="Docket's preserved copy on archive.usdocket.org"
+ >
+ <span aria-hidden="true">⎘</span>
+ <span>{label}</span>
+ </a>
+ );
+}
+
+/** One-line note explaining why Docket keeps its own copy. */
+export function PreservationNote({ className }: { className?: string }) {
+ return (
+ <p
+ className={clsx("font-prose", className)}
+ style={{ fontSize: "0.8125rem", lineHeight: 1.5, color: "var(--dk-ash-2)" }}
+ >
+ Docket preserves a copy so the record survives if the government page
+ changes or is taken down. The official source is always linked first.
+ </p>
+ );
+}
+
+/**
+ * On-page PDF viewer for a bill's text. Embeds Docket's archived copy when we
+ * have it (archive.usdocket.org, which permits framing) and otherwise falls
+ * back to the official govinfo.gov PDF. Renders nothing but the official link
+ * when no PDF exists at all - never an empty viewer.
+ */
+export function BillPdfViewer({
+ pdfUrl,
+ archivePdfUrl,
+ sourceUrl,
+ billNumber,
+}: {
+ pdfUrl: string;
+ archivePdfUrl: string;
+ sourceUrl: string;
+ billNumber: string;
+}) {
+ const hasArchive = !!archivePdfUrl;
+ const hasPdf = !!pdfUrl;
+ const embedSrc = archivePdfUrl || pdfUrl;
+ const officialUrl = pdfUrl || sourceUrl;
+
+ return (
+ <section className="mt-12" aria-label="Bill text">
+ <div style={{ marginBottom: "20px" }}>
+ <p
+ className="label"
+ style={{ color: "var(--dk-signal)", marginBottom: "10px", fontSize: "12px", letterSpacing: "0.14em" }}
+ >
+ Bill text
+ </p>
+ <h2
+ className="font-display text-paper"
+ style={{ fontSize: "clamp(1.2rem, 3.5vw, 1.5rem)", fontWeight: 800, letterSpacing: "-0.02em", textTransform: "uppercase", lineHeight: 1.05 }}
+ >
+ Read the official text
+ </h2>
+ </div>
+
+ {embedSrc ? (
+ <div className="rounded-[6px] border border-line bg-surface" style={{ overflow: "hidden" }}>
+ <div
+ className="flex flex-wrap items-center justify-between gap-2 border-b border-line"
+ style={{ padding: "12px 16px" }}
+ >
+ <p className="label" style={{ color: "var(--dk-ash)" }}>
+ <span className="font-mono">{billNumber}</span> - official PDF
+ </p>
+ <p className="label" style={{ color: "var(--dk-ash-2)" }}>
+ {hasArchive ? "Showing Docket's preserved copy" : "Showing govinfo.gov copy"}
+ </p>
+ </div>
+ <iframe
+ src={embedSrc}
+ title={`${billNumber} bill text (PDF)`}
+ style={{ width: "100%", height: "70vh", border: "0", display: "block", background: "var(--dk-ink)" }}
+ />
+ </div>
+ ) : null}
+
+ {/* Source links beneath the viewer */}
+ <div className="mt-4 flex flex-wrap items-center gap-3">
+ <SourceLink url={officialUrl} label="Official source" />
+ {hasArchive ? <ArchiveLink url={archivePdfUrl} /> : null}
+ </div>
+ {hasArchive ? <PreservationNote className="mt-3" /> : null}
+ {!embedSrc ? (
+ <p
+ className="mt-3 font-prose italic"
+ style={{ fontSize: "0.9375rem", color: "var(--dk-ash-2)" }}
+ >
+ No PDF of the bill text is on file yet. The official record is linked
+ above.
+ </p>
+ ) : null}
+ </section>
+ );
+}
components/register-hero.tsx +7 -6
@@ -56,7 +56,7 @@ export function RegisterHero({ seed }: { seed: FeedItem[] }) {
return (
<section
className="relative overflow-hidden border-y border-line"
- aria-label="The Register - live feed of official actions"
+ aria-label="The Register - live feed of official actions"
>
{/* rationed liquid-glass brass wash (hero only) */}
<div
@@ -121,7 +121,7 @@ export function RegisterHero({ seed }: { seed: FeedItem[] }) {
<div className="mt-3 flex flex-col gap-2 sm:flex-row sm:items-center sm:justify-between">
<p className="label text-ash-2">
- Ranked by action date, dollar magnitude, and people affected - never by coverage.
+ Ranked by action date, dollar magnitude, and people affected - never by coverage.
</p>
<Link
href="/how-we-pick"
@@ -141,7 +141,7 @@ function EmptyRegister() {
<p className="display text-2xl text-paper">The register is quiet.</p>
<p className="prose-lead mx-auto mt-3 max-w-md text-ash">
No official actions are on record yet. The ledger fills as bills are
- filed, votes are taken, and awards are made - each line sealed to its
+ filed, votes are taken, and awards are made - each line sealed to its
government source.
</p>
</div>
@@ -157,7 +157,6 @@ function RegisterRowInner({
}) {
const isPass = row.actionType === "PASSED";
const isFail = row.actionType === "FAILED";
- const result = row.amountText || "\u2014";
return (
<div className="reg-row grid grid-cols-1 gap-1 px-3 py-2.5 sm:grid-cols-[5.5rem_7.5rem_minmax(0,1fr)_6.5rem_minmax(0,5rem)] sm:gap-x-2 sm:items-center sm:px-4">
<span
@@ -191,7 +190,7 @@ function RegisterRowInner({
/>
</span>
<span
- className="font-mono text-[11px] uppercase tracking-wider"
+ className="font-mono text-[12px] uppercase tracking-wider"
style={{
color: isPass
? "var(--dk-yea)"
@@ -203,7 +202,9 @@ function RegisterRowInner({
{row.actionType}
</span>
<span className="font-mono text-[12px] tabular-nums text-paper sm:text-right">
- {result}
+ {/* Keep the em-dash placeholder only on the sm+ grid (column alignment);
+ on the stacked mobile card an empty result must not show a stray "-". */}
+ {row.amountText ? row.amountText : <span className="hidden sm:inline">{"-"}</span>}
</span>
</div>
);
components/rider-flag.tsx +1 -1
@@ -15,7 +15,7 @@ export function RiderFlag({ className }: { className?: string }) {
borderColor: "color-mix(in srgb, var(--dk-signal) 45%, transparent)",
backgroundColor: "var(--dk-signal-tint)",
}}
- title="Flagged as a rider - added outside the bill's named subject"
+ title="Flagged as a rider - added outside the bill's named subject"
>
<span aria-hidden="true">◆</span> Rider flag
</span>
components/show-more.tsx +59 -0
@@ -0,0 +1,59 @@
+"use client";
+
+import { useState, type ReactNode } from "react";
+
+/**
+ * Progressive-disclosure list wrapper. Renders the first `pageSize` of a
+ * pre-rendered node array, then a single accessible "Show N more" button that
+ * reveals the next page. Keeps long lists (search groups, bill cards) from
+ * dumping thousands of rows into one tens-of-thousands-of-pixels page while
+ * preserving the cached /api architecture (the full set is already client-side;
+ * only the rendered slice grows).
+ *
+ * The parent supplies the container (Card, ul, grid). This component renders the
+ * visible items as a fragment followed by the control, so it drops into any
+ * layout. When the items are <li>, pass `controlWrapper="li"` so the button row
+ * stays valid inside a <ul>.
+ */
+export function ShowMore({
+ items,
+ pageSize = 50,
+ controlWrapper = "div",
+ noun = "more",
+}: {
+ items: ReactNode[];
+ pageSize?: number;
+ controlWrapper?: "div" | "li";
+ noun?: string;
+}) {
+ const [shown, setShown] = useState(pageSize);
+ const visible = items.slice(0, shown);
+ const remaining = items.length - shown;
+
+ const control =
+ remaining > 0 ? (
+ <div className="flex items-center justify-between gap-3 px-4 py-3">
+ <span className="label text-ash-2">
+ Showing {shown.toLocaleString()} of {items.length.toLocaleString()}
+ </span>
+ <button
+ type="button"
+ onClick={() => setShown((n) => n + pageSize)}
+ className="focus-ring label inline-flex min-h-[40px] items-center rounded-[3px] border border-line px-3 text-ash underline underline-offset-4 transition-colors hover:border-paper hover:text-paper"
+ >
+ Show {Math.min(pageSize, remaining).toLocaleString()} {noun}
+ </button>
+ </div>
+ ) : null;
+
+ return (
+ <>
+ {visible}
+ {control
+ ? controlWrapper === "li"
+ ? <li>{control}</li>
+ : control
+ : null}
+ </>
+ );
+}
components/site-footer.tsx +1 -1
@@ -20,7 +20,7 @@ export function SiteFooter() {
</p>
<div className="mt-6 flex items-center gap-3">
<OwlBust size={40} className="shrink-0 opacity-90" title="Archie, the Docket archivist" />
- <span className="label text-ash-2">Docket - the public record</span>
+ <span className="label text-ash-2">Docket - the public record</span>
</div>
</div>
components/site-header.tsx +11 -7
@@ -52,9 +52,10 @@ export function SiteHeader() {
return (
<>
<header
- className="sticky top-0 z-40 border-b border-line"
+ className="sticky top-0 z-40"
style={{
- background: "rgba(11,12,14,0.88)",
+ background: "var(--dk-header-bg)",
+ borderBottom: "1px solid var(--dk-header-border)",
backdropFilter: "blur(14px) saturate(120%)",
WebkitBackdropFilter: "blur(14px) saturate(120%)",
height: "64px",
@@ -70,8 +71,8 @@ export function SiteHeader() {
<Link
href="/"
className="focus-ring shrink-0 no-underline"
- aria-label="Docket - home"
- style={{ display: "inline-flex", alignItems: "center", gap: "10px" }}
+ aria-label="Docket - home"
+ style={{ display: "inline-flex", alignItems: "center", gap: "10px", minHeight: "44px" }}
>
<CustodyMark size={28} />
<Wordmark className="font-display text-paper" />
@@ -93,12 +94,15 @@ export function SiteHeader() {
className="focus-ring no-underline"
style={{
fontFamily: "var(--font-mono), ui-monospace, monospace",
- fontSize: "11.5px",
+ fontSize: "12.5px",
fontWeight: 400,
letterSpacing: "0.13em",
textTransform: "uppercase",
color: active ? "var(--dk-paper)" : "var(--dk-ash)",
- padding: "8px 7px",
+ display: "inline-flex",
+ alignItems: "center",
+ minHeight: "44px",
+ padding: "0 7px",
whiteSpace: "nowrap",
borderRadius: "6px",
position: "relative",
@@ -148,7 +152,7 @@ export function SiteHeader() {
background: "transparent",
color: "var(--dk-ash)",
fontFamily: "var(--font-mono), ui-monospace, monospace",
- fontSize: "11.5px",
+ fontSize: "12.5px",
letterSpacing: "0.12em",
textTransform: "uppercase",
transition: "border-color 0.2s, color 0.2s",
components/source-link.tsx +11 -3
@@ -39,11 +39,19 @@ export function SourceLink({
)}
title={allowed ? `Official record · ${host}` : "Source host not allowlisted"}
>
- <span aria-hidden="true">↗</span>
- <span className="whitespace-nowrap">{label}</span>
+ {/* Arrow + label + separator stay together on one line; only the domain
+ wraps. This stops the "·" from stranding alone on its own line when a
+ long host (e.g. apportionment-public.max.gov) wraps in a narrow cell. */}
+ <span className="inline-flex items-center gap-x-1 whitespace-nowrap">
+ <span aria-hidden="true">↗</span>
+ <span>
+ {label}
+ {allowed && host ? <span className="text-ash-2"> ·</span> : null}
+ </span>
+ </span>
{allowed && host ? (
<span className="text-[11px] tracking-normal text-ash-2 [overflow-wrap:anywhere]">
- · {host}
+ {host}
</span>
) : null}
</a>
components/ui.tsx +5 -1
@@ -106,7 +106,7 @@ export function PageHeader({
<h1
className="font-display text-paper"
style={{
- fontSize: "clamp(2.2rem, 7vw, 3.75rem)",
+ fontSize: "clamp(2rem, 7vw, 3.75rem)",
fontWeight: 900,
lineHeight: 1.0,
letterSpacing: "-0.01em",
@@ -114,6 +114,10 @@ export function PageHeader({
textTransform: "uppercase",
maxWidth: "18ch",
marginBottom: lead ? "24px" : 0,
+ // A single long word (e.g. "REPRESENTATIVES") must never overflow at
+ // 390px - let it hyphenate/break rather than push the page wide.
+ overflowWrap: "anywhere",
+ hyphens: "auto",
}}
>
{title}
components/votes-browser.tsx +2 -2
@@ -212,7 +212,7 @@ export function VotesBrowser() {
<SectionHeader
eyebrow={`${total.toLocaleString()} vote${total === 1 ? "" : "s"}${isFiltered ? " match" : ""} · showing ${votes.length}`}
title="Recorded votes"
- description="Newest action first. A vote with no Yea/Nay tally passed by voice or unanimous consent - no individual record exists for those."
+ description="Newest action first. A vote with no Yea/Nay tally passed by voice or unanimous consent - no individual record exists for those."
/>
{votes.length === 0 ? (
<Card className="p-8 text-center">
@@ -226,7 +226,7 @@ export function VotesBrowser() {
<VotesTable votes={votes} />
)}
<p className="label mt-4 text-ash-2">
- Source: clerk.house.gov and senate.gov roll-call XML. Figures only -
+ Source: clerk.house.gov and senate.gov roll-call XML. Figures only -
no characterization.
</p>
</div>
docs/API.md +1 -1
@@ -1,6 +1,6 @@
# Docket - Internal API Contract
-Single-box deployment (CT247): Next.js app + SQLite (local, `/opt/docket/data/docket.db`) populated by the Python ingest. Frontend reads its OWN Next.js route handlers (`app/api/*`) via `better-sqlite3`. Backend agents write to the SQLite schema below; frontend consumes these JSON shapes. This is the contract - build to it on both sides.
+Single-box deployment: Next.js app + SQLite (local, `/opt/docket/data/docket.db`) populated by the Python ingest. Frontend reads its OWN Next.js route handlers (`app/api/*`) via `better-sqlite3`. Backend agents write to the SQLite schema below; frontend consumes these JSON shapes. This is the contract - build to it on both sides.
## Conventions
- All responses JSON. Every item carries `sourceUrl` (allowlisted official `.gov`/state-leg host; aggregator → underlying official record).
docs/DESIGN_V3.md +1 -1
@@ -4,7 +4,7 @@ The light V2 (Atkinson, then Archivo/Fraunces light + serif body) is REJECTED -
## GROUND TRUTH (study these, replicate them)
- Reference screenshots (captured from the live sites): `_ops/qa-view/REF-treetrace-desktop.png`, `_ops/qa-view/REF-zionboggan-desktop.png`.
-- Their live CSS (curl from CT247 to read the real values): treetrace.dev `/assets/css/{tokens,shell,glass}.css`; zionboggan.com `/assets/css/{main,flare,sections}.css`.
+- Their live CSS: treetrace.dev `/assets/css/{tokens,shell,glass}.css`; zionboggan.com `/assets/css/{main,flare,sections}.css`.
- What both share (the aesthetic to match):
- **DARK** near-black backgrounds (ink ~#0B0C0E, raised surface ~#121316/#1A1C1F).
- **Archivo 800-900, UPPERCASE, big, tight tracking** for display headlines (e.g. "RECORD THE PATH", "ZION BOGGAN").
docs/INGESTION_STRATEGY.md +6 -6
@@ -9,18 +9,18 @@ Roadmap for ingest v2. Today's pollers are naive (full pull each run). This is h
4. **SHA-256 content-hash gate** - hash raw payload before parse; skip parse/upsert if unchanged. `INSERT … ON CONFLICT(source,external_id) DO UPDATE … WHERE hash != EXCLUDED.hash`. Applies everywhere.
5. **RSS/Atom feed registry** - ~20 states publish official feeds (KS, WA, HI, TX, WI, WV…). `feedparser` + last-seen GUID = sub-minute, zero-budget change detection.
6. **Session-calendar priority polling** - `sessions` table (active/interim/dormant); poll active hourly, recess daily, dormant weekly. ~70% volume cut off-season.
-7. **ARQ + aiometer async workers** - Redis-backed asyncio queue (better fit than Celery here); `aiometer` for per-domain concurrency + rps caps; `tenacity` backoff on 429. Distribute across **CT215 + CT220** via worker hash-ring.
-8. **LLM selector self-healing (3060)** - on field-extraction failure, queue raw HTML + schema → local `mistral-nemo:12b` proposes CSS/XPath → validate against live HTML → write to `scraper_selectors` table (no redeploy). ~98% accuracy across layout changes (2025 research).
-9. **Embedding action-vocab normalization** - 50 states use different action strings → embed with `nomic-embed-text` (Ollama/3060) → cluster (HDBSCAN) → map to one canonical taxonomy (INTRODUCED/REFERRED/PASSED/SIGNED/VETOED…). One-time batch, re-run on new strings.
+7. **ARQ + aiometer async workers** - Redis-backed asyncio queue (better fit than Celery here); `aiometer` for per-domain concurrency + rps caps; `tenacity` backoff on 429. Distribute across worker processes via hash-ring.
+8. **LLM selector self-healing** - on field-extraction failure, queue raw HTML + schema → a local LLM proposes CSS/XPath → validate against live HTML → write to `scraper_selectors` table (no redeploy). ~98% accuracy across layout changes (2025 research).
+9. **Embedding action-vocab normalization** - 50 states use different action strings → embed with a local embedding model → cluster (HDBSCAN) → map to one canonical taxonomy (INTRODUCED/REFERRED/PASSED/SIGNED/VETOED…). One-time batch, re-run on new strings.
10. **Near-duplicate bill clustering** - embed bill summaries → store vectors in the existing **Qdrant** → cosine >0.92 flags model legislation (ALEC etc.) spreading across states. Unique Docket feature; zero marginal API cost.
## Recommended architecture
- **Source registry YAML** - each source: `type` (api|bulk|rss|scrape), `change_detection` (watermark|rss|hash|head), `poll_interval_active/dormant`, `rate_limit_rps`, `js_required`, `priority`. Scheduler is data-driven; new/changed sources need no code.
- **Three-tier ingest:** (1) bulk bootstrap once per session/congress; (2) delta API polling via watermarks (hourly active / daily dormant); (3) persistent RSS watchers emitting change events immediately.
-- **ARQ worker pool (CT215+CT220)**, partitioned by source hash-ring, Redis broker; workers write raw payloads → `raw_ingest` staging.
+- **ARQ worker pool**, partitioned by source hash-ring, Redis broker; workers write raw payloads → `raw_ingest` staging.
- **Hash-gated parse stage** - second worker computes `content_hash`, skips unchanged, else runs source parser → `*_staging` → idempotent upsert + emit `bill_versions` diff on text change.
- **Dead-letter + alerting** - failures → `scrape_failures` (raw error + HTML snapshot + retry count); Discord webhook after 3; 6-hour retry worker.
-- **LLM repair sidecar** - separate process on the 3060 (Ollama), consumes a Redis `selector_repair_queue`; main scrapers read selectors from DB each run.
+- **LLM repair sidecar** - separate process (local LLM), consumes a Redis `selector_repair_queue`; main scrapers read selectors from DB each run.
## 50-state scraping
- **Fork `openstates/openstates-scrapers`** (~200 scrapers on `scrapelib` + `spatula` + `pupa`) - battle-tested session mgmt/robots/retry. Don't start from scratch.
@@ -32,7 +32,7 @@ Roadmap for ingest v2. Today's pollers are naive (full pull each run). This is h
- **GovTrack/`unitedstates/congress` git mirrors** - bill versions as commits; `git log --diff-filter=M` gives changed-bills-since-last-sync without API.
- **Undocumented state JSON endpoints** - check the network tab before writing any scraper.
-## 3060 LLM: value vs trap
+## Local LLM: value vs trap
- **Use:** selector self-healing, action-vocab embedding/normalization, PDF/messy-HTML extraction (long tail), near-dup clustering. All batch/exception-path.
- **Trap:** LLM as primary parser for structured HTML; LLM in the real-time hot path (creates permanent backlog at 100k+ bills/session); trusting LLM output without schema validation; re-parsing the Congressional Record (use `unitedstates/congressional-record`).
docs/NEUTRALITY.md +1 -1
@@ -32,7 +32,7 @@ No free-text user comments on bills, votes, or spending - anywhere. The single
When prose summaries land (Phase 2+; MVP is structured-fields-only), they are produced by **field extraction into a fixed template**, not open-ended generation.
- Template: (1) official title + number, (2) sponsor name/party/state - factual, uncharacterized, (3) what the text authorizes (extracted clauses, plain language), (4) CBO score if any (dollar figure only), (5) current status.
- Every summary passes the banned-words lint + a **Polarization-Index check** before publish (generate D-leaning and R-leaning variants, assert the default sits statistically between them). High-salience bills (100+ media mentions) get human review before push.
-- LLM runs locally on the 3060 (Ollama) for drafting; extraction is preferred over generation wherever a field can be pulled deterministically.
+- LLM runs locally for drafting; extraction is preferred over generation wherever a field can be pulled deterministically.
---
docs/logo.svg +9 -0
@@ -0,0 +1,9 @@
+<svg viewBox="0 0 320 72" xmlns="http://www.w3.org/2000/svg">
+ <rect width="320" height="72" fill="#0B0C0E"/>
+ <g transform="translate(20,12) scale(0.48)">
+ <rect x="9" y="9" width="82" height="82" rx="19" fill="none" stroke="#F2EFE9" stroke-width="5"/>
+ <line x1="29" y1="65" x2="63" y2="65" stroke="#F2EFE9" stroke-width="5" stroke-linecap="round" opacity="0.5"/>
+ <path d="M31 53 L42 65 L69 32" fill="none" stroke="#E0A53B" stroke-width="8" stroke-linecap="round" stroke-linejoin="round"/>
+ </g>
+ <text x="76" y="46" font-family="Archivo, Arial, sans-serif" font-weight="800" font-size="30" letter-spacing="-0.6" fill="#F2EFE9">Docket</text>
+</svg>
docs/screenshots/home-dark.png +0 -0
Binary file not shown
docs/screenshots/home-light.png +0 -0
Binary file not shown
docs/screenshots/search.png +0 -0
Binary file not shown
docs/screenshots/spending.png +0 -0
Binary file not shown
lib/feed.ts +36 -10
@@ -150,13 +150,13 @@ export type MemberSummary = {
state: string;
district: number | null;
chamber: string; // house | senate
- photoUrl: string; // official headshot (unitedstates/images mirror) - "" when none
+ photoUrl: string; // official headshot (unitedstates/images mirror) - "" when none
sourceUrl: string;
href: string;
- /** True for current Congress (119th) members. The ingest assigns IDs 1-537
- * to the sitting House and Senate; historical members receive IDs above 2000.
- * This flag drives the roster's default view (current first, historical behind
- * a disclosure toggle). */
+ /** True for sitting 119th-Congress members. Sourced from the members.is_current
+ * column, set to 1 for every bioguide in legislators-current.json. This flag
+ * drives the roster's default view (current first, historical behind a
+ * disclosure toggle) and each state's current delegation. */
isCurrent: boolean;
};
@@ -167,7 +167,7 @@ export type VotePosition = {
state: string;
district: number | null;
position: string; // Yea / Nay / Present / Not Voting
- photoUrl: string; // official headshot - "" when none
+ photoUrl: string; // official headshot - "" when none
memberHref: string;
};
@@ -428,11 +428,16 @@ export function getFeedItems(perSource = 400): FeedItem[] {
)
);
- // Bills (Congress.gov) - most recently acted on (latest action / updated_at).
+ // Bills - most recently acted on (latest action / updated_at). Scoped to
+ // federal congressional bills (congress > 0): the register's other sources
+ // (awards, votes, executive actions) are all federal, so this keeps the
+ // ledger a consistent federal official-action feed. State-legislature bills
+ // remain browsable on /bills and the per-state pages.
items.push(
...safeAll<FeedItem>(
`SELECT id, number, title, status, updated_at, source_url
FROM bills
+ WHERE congress > 0
ORDER BY updated_at DESC
LIMIT :lim`,
{ lim: perSource },
@@ -1377,8 +1382,8 @@ function getAllMembersRaw(): MemberSummary[] {
}
/**
- * Current Congress members only (IDs 1-600, the 119th sitting House + Senate).
- * This is the default SSR seed for the members page - ~537 members instead of
+ * Current Congress members only (is_current = 1, the 119th sitting House +
+ * Senate). This is the default SSR seed for the members page - ~537 instead of
* 12.7k. Historical members are fetched lazily via /api/members when the user
* opens the disclosure toggle. Keeps the SSR HTML payload under 1MB.
*/
@@ -1393,7 +1398,14 @@ export function getMembersDefault(): MemberSummary[] {
* Cheap aggregate counts for the members page summary cards.
* Avoids serializing 12.7k members just to count them.
*/
-export function getMemberCounts(): { total: number; senate: number; house: number } {
+export function getMemberCounts(): {
+ total: number;
+ senate: number;
+ house: number;
+ currentTotal: number;
+ currentSenate: number;
+ currentHouse: number;
+} {
return cached("members:counts", () => {
const total = safeCount("members");
const senateRow = safeGet<{ c: number }>(
@@ -1402,10 +1414,24 @@ export function getMemberCounts(): { total: number; senate: number; house: numbe
const houseRow = safeGet<{ c: number }>(
`SELECT COUNT(*) AS c FROM members WHERE LOWER(chamber) = 'house'`
);
+ // Currently-serving counts (is_current = 1) for the headline cards - the
+ // all-time totals are kept for an accurate "on record" sub-note.
+ const curTotalRow = safeGet<{ c: number }>(
+ `SELECT COUNT(*) AS c FROM members WHERE is_current = 1`
+ );
+ const curSenateRow = safeGet<{ c: number }>(
+ `SELECT COUNT(*) AS c FROM members WHERE is_current = 1 AND LOWER(chamber) = 'senate'`
+ );
+ const curHouseRow = safeGet<{ c: number }>(
+ `SELECT COUNT(*) AS c FROM members WHERE is_current = 1 AND LOWER(chamber) = 'house'`
+ );
return {
total,
senate: senateRow?.c ?? 0,
house: houseRow?.c ?? 0,
+ currentTotal: curTotalRow?.c ?? 0,
+ currentSenate: curSenateRow?.c ?? 0,
+ currentHouse: curHouseRow?.c ?? 0,
};
});
}
lib/mock-data.ts +1 -1
@@ -338,7 +338,7 @@ export const AWARDS: Award[] = [
{
id: "w2",
agency: "Department of Health and Human Services",
- recipient: "State of California - Medicaid",
+ recipient: "State of California - Medicaid",
amountUsd: 612_000_000,
actionDate: "2026-06-26",
program: "Federal share, Medicaid",
lib/vote-subject.ts +3 -3
@@ -46,7 +46,7 @@ export function resolveVoteSubject(v: VoteSubjectInput): VoteSubject {
// 1. Linked bill (full bill record in DB)
if (v.billNumber) {
- const title = v.billTitle ? ` - ${v.billTitle}` : "";
+ const title = v.billTitle ? ` -${v.billTitle}` : "";
return {
primary: `${formatBillNumber(v.billNumber)}${title}`,
secondary: question || result,
@@ -57,7 +57,7 @@ export function resolveVoteSubject(v: VoteSubjectInput): VoteSubject {
// 2. Presidential nomination (Senate confirmation vote)
if (ref.toUpperCase().startsWith("PN")) {
return {
- primary: `Nomination - ${question || ref}`,
+ primary: `Nomination -${question || ref}`,
secondary: result,
kind: "nomination",
};
@@ -66,7 +66,7 @@ export function resolveVoteSubject(v: VoteSubjectInput): VoteSubject {
// 3. Treaty document ratification
if (ref.toUpperCase().startsWith("TREATYDOC")) {
return {
- primary: `Treaty document - ${question || ref}`,
+ primary: `Treaty document -${question || ref}`,
secondary: result,
kind: "treaty",
};
public/apple-touch-icon.png +0 -0
Binary file not shown
public/icon-512.png +0 -0
Binary file not shown
services/ingest/CATEGORIZE_NOTES.md +4 -5
@@ -116,13 +116,12 @@ Test classes:
```bash
# Full backfill (all bills + awards + categories)
-cd /shared/projects/docket/services/ingest
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- .venv/bin/python -m ingest.categorize --once
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.categorize --once
# Limit subjects fetch (for testing)
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- .venv/bin/python -m ingest.categorize --once --limit-subjects 50
+python -m ingest.categorize --once --limit-subjects 50
# Show category counts without re-running backfill
DOCKET_DB_PATH=/opt/docket/data/docket.db \
services/ingest/CLOSEDDOOR_NOTES.md +8 -14
@@ -1,5 +1,5 @@
# Closed-Door + Executive Layer - Build Notes
-_Built 2026-06-26. All pollers verified live on CT247._
+_Built 2026-06-26. All pollers verified live._
## What was built
@@ -20,7 +20,7 @@ write; `is_new_event()` / `mark_seen()` enforce idempotency.
---
-## Live run results (CT247 - 2026-06-26)
+## Live run results (2026-06-26)
| Table | Rows | Notes |
|-------|------|-------|
@@ -143,19 +143,13 @@ All validated by `source_validator.validate_source_url()` before every DB write.
## How to run
```bash
-# On CT247 (venv at /opt/docket/services/ingest/.venv)
-cd /shared/projects/docket/services/ingest
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.federal_register --once
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.pollers.federal_register --once
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.openomb --once --max 50
-
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.crec --once --days 14
-
-DOCKET_DB_PATH=/opt/docket/data/docket.db DOCKET_LDA_KEY=<your_key> \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.lobbying --once
+python -m ingest.pollers.openomb --once --max 50
+python -m ingest.pollers.crec --once --days 14
+DOCKET_LDA_KEY=<your_key> python -m ingest.pollers.lobbying --once
```
## LDA key
services/ingest/INGEST_NOTES.md +17 -20
@@ -30,40 +30,37 @@ tests/
## How to Run
-### Setup (one-time on CT241)
+### Setup (one-time)
```bash
-# Venv must be local - the CIFS share (/shared/projects) does not support
-# SQLite file locking or venv symlinks
-python3 -m venv /tmp/docket-venv
-/tmp/docket-venv/bin/pip install -r /shared/projects/docket/services/ingest/requirements.txt
-
-# SQLite DB also lives locally for the same reason
-mkdir -p /var/tmp/docket
+cd services/ingest
+python3 -m venv .venv
+.venv/bin/pip install -r requirements.txt
```
### Run all pollers once
```bash
-cd /shared/projects/docket/services/ingest
-/tmp/docket-venv/bin/python -m ingest.run --once
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.run --once
```
### Run one poller
```bash
-/tmp/docket-venv/bin/python -m ingest.run --source usaspending
-/tmp/docket-venv/bin/python -m ingest.run --source votes_senate
+python -m ingest.run --source usaspending
+python -m ingest.run --source votes_senate
```
### Run tests
```bash
-cd /shared/projects/docket/services/ingest
-/tmp/docket-venv/bin/python -m pytest tests/ -v
+cd services/ingest
+python -m pytest tests/ -v
```
-## Test Results (verified 2026-06-26 on CT241)
+## Test Results (verified 2026-06-26)
```
35 passed in 4.09s
@@ -130,12 +127,12 @@ Override with: `export DOCKET_DB_PATH=/path/to/docket.db`
endpoint returns null for Action Date on many records. For completeness, use the
`/bulk_download/` endpoint or per-award detail endpoint to get richer metadata.
-4. **SQLite on CIFS does not work** - `/shared/projects` on CT241 is a CIFS mount,
- which does not support SQLite's file-locking mechanism. DB defaults to local
- `/var/tmp/docket/docket.db`. For production, migrate to Postgres on a dedicated CT.
+4. **SQLite must be on local storage** - Network filesystems (CIFS/NFS) do not
+ support SQLite's file-locking mechanism. Keep the DB on local disk.
+ Set `DOCKET_DB_PATH` to a local path. For production scale, migrate to Postgres.
-5. **Venv on CIFS does not work** - same CIFS limitation; venv must live in
- `/tmp/docket-venv` or another local path on CT241.
+5. **Venv must be on local storage** - same filesystem requirement; the venv
+ should live on the deployment host's local disk.
## Neutrality Enforcement (architectural summary)
services/ingest/STATE_NOTES.md +10 -14
@@ -22,14 +22,12 @@ jurisdictions (50 states + DC + PR).
### Compute state coverage (works without any API keys)
```bash
-# On CT247 with the production DB
-cd /shared/projects/docket/services/ingest
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.state_coverage
+cd services/ingest
+export DOCKET_DB_PATH=/path/to/docket.db
+python -m ingest.state_coverage
# JSON output
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.state_coverage --json
+python -m ingest.state_coverage --json
```
Expected output (no keys, empty state DB):
@@ -46,11 +44,10 @@ missing (never seen) : 52
```bash
export DOCKET_OPENSTATES_KEY=<your_key>
# MVP subset (12 jurisdictions - conserves 500/day free quota)
-/opt/docket/services/ingest/.venv/bin/python -m ingest.run --source openstates
+python -m ingest.run --source openstates
# All 52 jurisdictions (needs paid tier)
-DOCKET_OPENSTATES_ALL=1 \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.run --source openstates
+DOCKET_OPENSTATES_ALL=1 python -m ingest.run --source openstates
```
### Run LegiScan poller (requires key)
@@ -58,18 +55,17 @@ DOCKET_OPENSTATES_ALL=1 \
```bash
export DOCKET_LEGISCAN_KEY=<your_key>
# MVP subset (12 states - ~200 API calls)
-/opt/docket/services/ingest/.venv/bin/python -m ingest.run --source legiscan
+python -m ingest.run --source legiscan
# All 52 jurisdictions (needs ~2k API calls - monitor monthly quota)
-DOCKET_LEGISCAN_ALL=1 \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.run --source legiscan
+DOCKET_LEGISCAN_ALL=1 python -m ingest.run --source legiscan
```
### Run tests
```bash
-cd /shared/projects/docket/services/ingest
-/opt/docket/services/ingest/.venv/bin/python -m pytest tests/ -v
+cd services/ingest
+python -m pytest tests/ -v
# Expected: 190 passed
```
services/ingest/VOTES_NOTES.md +1 -1
@@ -1,6 +1,6 @@
# Docket - Votes & Members Build Notes
-Verified 2026-06-26 on CT247 (venv `/opt/docket/services/ingest/.venv`, DB `/opt/docket/data/docket.db`).
+Verified 2026-06-26 (venv `/opt/docket/services/ingest/.venv`, DB `/opt/docket/data/docket.db`).
## Files Created / Modified
services/ingest/VOTEVIEW_NOTES.md +6 -4
@@ -137,22 +137,24 @@ incomplete run).
### Proof slice (congresses 118-119, 2023-present)
```bash
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc 'cd /shared/projects/docket/services/ingest && DOCKET_DB_PATH=/opt/docket/data/docket.db /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.voteview --once --start 118 --end 119'"
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.pollers.voteview --once --start 118 --end 119
```
### Full 2019-present (congresses 116-119)
```bash
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc 'cd /shared/projects/docket/services/ingest && DOCKET_DB_PATH=/opt/docket/data/docket.db /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.voteview --once --start 116 --end 119'"
+python -m ingest.pollers.voteview --once --start 116 --end 119
```
### Full historical backfill (1989-present, ~30 congresses, recommended first pass)
```bash
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc 'cd /shared/projects/docket/services/ingest && DOCKET_DB_PATH=/opt/docket/data/docket.db /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.voteview --once --start 101 --end 119'"
+python -m ingest.pollers.voteview --once --start 101 --end 119
```
### Full Voteview archive (1789-present, all ~120 congresses)
```bash
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc 'cd /shared/projects/docket/services/ingest && DOCKET_DB_PATH=/opt/docket/data/docket.db /opt/docket/services/ingest/.venv/bin/python -m ingest.pollers.voteview --once --start 1 --end 119'"
+python -m ingest.pollers.voteview --once --start 1 --end 119
```
Add `--report` to any command to print the CSV vs DB completeness table.
services/ingest/WATCHER_NOTES.md +8 -18
@@ -113,13 +113,13 @@ Use it to tell an external dashboard "openomb was last checked at T".
The `cycle_id` is a monotonically increasing integer that ties `change_events`
and `watcher_manifest` entries to the same logical run.
-## systemd install on CT247
+## systemd install
```bash
# 1. Copy unit files
-cp /shared/projects/docket/services/ingest/deploy/docket-watcher.service \
+cp services/ingest/deploy/docket-watcher.service \
/etc/systemd/system/
-cp /shared/projects/docket/services/ingest/deploy/docket-watcher.timer \
+cp services/ingest/deploy/docket-watcher.timer \
/etc/systemd/system/
# 2. Reload and enable
@@ -150,26 +150,16 @@ sqlite3 /opt/docket/data/docket.db \
## Testing a single cycle (without full pipeline)
```bash
-cd /shared/projects/docket/services/ingest
-DOCKET_DB_PATH=/opt/docket/data/docket.db \
- .venv/bin/python -m ingest.watcher --once --sources federal_register
-```
-
-Or via the exec command from the overseer notes:
-
-```bash
-ssh root@REDACTED-IP "pct exec 247 -- bash -lc \
- 'cd /shared/projects/docket/services/ingest && \
- DOCKET_DB_PATH=/opt/docket/data/docket.db \
- /opt/docket/services/ingest/.venv/bin/python -m ingest.watcher --once \
- --sources federal_register'"
+cd services/ingest
+export DOCKET_DB_PATH=/opt/docket/data/docket.db
+python -m ingest.watcher --once --sources federal_register
```
## Running the unit tests
```bash
-cd /shared/projects/docket/services/ingest
-.venv/bin/pytest tests/test_watcher.py -v
+cd services/ingest
+python -m pytest tests/test_watcher.py -v
```
## Adding a new full-set source
services/ingest/_check_summaries.py +15 -0
@@ -0,0 +1,15 @@
+import sqlite3
+c = sqlite3.connect('/opt/docket/data/docket.db')
+c.execute('PRAGMA busy_timeout=30000')
+c.row_factory = sqlite3.Row
+total = c.execute("SELECT COUNT(*) FROM bills WHERE source='legiscan'").fetchone()[0]
+withs = c.execute("SELECT COUNT(*) FROM bills WHERE source='legiscan' AND summary IS NOT NULL AND summary<>''").fetchone()[0]
+print("legiscan total:", total, "with_summary:", withs)
+rows = c.execute("""SELECT jurisdiction, COUNT(*) n FROM bills
+ WHERE source='legiscan' AND summary IS NOT NULL AND summary<>''
+ GROUP BY jurisdiction ORDER BY n DESC LIMIT 20""").fetchall()
+print("by state:", [(r['jurisdiction'], r['n']) for r in rows])
+samp = c.execute("""SELECT jurisdiction, number, substr(summary,1,120) s, summary_source
+ FROM bills WHERE source='legiscan' AND summary IS NOT NULL AND summary<>'' LIMIT 5""").fetchall()
+for r in samp:
+ print("SAMPLE", r['jurisdiction'], r['number'], '|', r['summary_source'], '|', r['s'])
services/ingest/backfill_is_current.py +114 -0
@@ -0,0 +1,114 @@
+"""
+One-time backfill for members.is_current.
+
+Authoritatively marks the sitting 119th Congress: every bioguide present in the
+unitedstates/congress-legislators `legislators-current.json` dataset (the same
+source the members poller ingests from) is is_current = 1; everyone else is 0.
+
+Run on CT247 against the deployed DB:
+
+ DOCKET_DB_PATH=/opt/docket/data/docket.db \
+ python3 -m services.ingest.backfill_is_current
+
+or, from inside services/ingest/:
+
+ DOCKET_DB_PATH=/opt/docket/data/docket.db python3 backfill_is_current.py
+
+Idempotent: safe to re-run. Adds the column if it is missing, then performs a
+single transaction (two bulk UPDATEs) so it stays light against the live ingest
+(the connection carries a 60s busy_timeout).
+"""
+
+import logging
+import os
+import sys
+
+import requests
+
+# Support both "python3 -m services.ingest.backfill_is_current" and direct run.
+try:
+ from ingest.db import get_connection
+ from ingest.pollers.members import LEGISLATORS_JSON_URL, _parse_legislators
+except ModuleNotFoundError: # direct run from a different cwd
+ sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
+ from ingest.db import get_connection
+ from ingest.pollers.members import LEGISLATORS_JSON_URL, _parse_legislators
+
+logger = logging.getLogger("backfill_is_current")
+
+
+def fetch_current_bioguides() -> list[str]:
+ resp = requests.get(LEGISLATORS_JSON_URL, timeout=60)
+ resp.raise_for_status()
+ data = resp.json()
+ if not isinstance(data, list):
+ raise ValueError(f"unexpected JSON shape: {type(data)}")
+ members = _parse_legislators(data)
+ return [m["bioguide_id"] for m in members if m.get("bioguide_id")]
+
+
+def main() -> int:
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+ )
+
+ bioguides = fetch_current_bioguides()
+ logger.info("fetched %d current bioguides from legislators-current.json", len(bioguides))
+ if not bioguides:
+ logger.error("no current bioguides fetched; aborting (DB left unchanged)")
+ return 1
+
+ conn = get_connection()
+
+ # Ensure the column exists (idempotent).
+ try:
+ conn.execute(
+ "ALTER TABLE members ADD COLUMN is_current INTEGER NOT NULL DEFAULT 0"
+ )
+ except Exception:
+ pass # already present
+
+ total = conn.execute("SELECT COUNT(*) AS c FROM members").fetchone()["c"]
+
+ # Single transaction: reset all, then mark the current roster.
+ placeholders = ",".join("?" for _ in bioguides)
+ conn.execute("BEGIN")
+ conn.execute("UPDATE members SET is_current = 0")
+ cur = conn.execute(
+ f"UPDATE members SET is_current = 1 WHERE bioguide_id IN ({placeholders})",
+ bioguides,
+ )
+ marked = cur.rowcount
+ conn.commit()
+
+ matched = conn.execute(
+ "SELECT COUNT(*) AS c FROM members WHERE is_current = 1"
+ ).fetchone()["c"]
+
+ # How many current bioguides had no matching row (e.g. roster not yet ingested).
+ missing = len(set(bioguides)) - matched
+
+ logger.info(
+ "members total=%d | current-roster=%d | marked is_current=1 -> %d (rowcount=%d) | not-in-db=%d",
+ total,
+ len(bioguides),
+ matched,
+ marked,
+ missing,
+ )
+
+ # Per-state sanity check for a couple of states.
+ for st in ("WY", "CA", "TX"):
+ row = conn.execute(
+ "SELECT COUNT(*) AS c FROM members WHERE is_current = 1 AND UPPER(state) = ?",
+ (st,),
+ ).fetchone()
+ logger.info(" current members in %s: %d", st, row["c"])
+
+ conn.close()
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
services/ingest/deploy/docket-daily.service +25 -0
@@ -0,0 +1,25 @@
+[Unit]
+Description=Docket ingest - daily tier (refresh + roster/fiscal/lobbying + bounded enrichment + state coverage)
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+After=network-online.target docket-web.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=root
+WorkingDirectory=/opt/docket/services/ingest
+Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db
+EnvironmentFile=-/opt/docket/.env
+
+# Shared flock - see docket-refresh.service. Mutually exclusive with the other
+# two tiers; skips cleanly (exit 0) if a tier is already running.
+ExecStart=/bin/flock --nonblock --conflict-exit-code 0 /run/docket-ingest.lock /opt/docket/services/ingest/.venv/bin/python -m ingest.run --daily
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=docket-daily
+TimeoutStartSec=10800
+Nice=10
+
+[Install]
+WantedBy=multi-user.target
services/ingest/deploy/docket-daily.timer +12 -0
@@ -0,0 +1,12 @@
+[Unit]
+Description=Docket daily tier - once daily (04:30 America/Chicago)
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+
+[Timer]
+OnCalendar=*-*-* 04:30 America/Chicago
+Persistent=true
+RandomizedDelaySec=120
+Unit=docket-daily.service
+
+[Install]
+WantedBy=timers.target
services/ingest/deploy/docket-refresh.service +29 -0
@@ -0,0 +1,29 @@
+[Unit]
+Description=Docket ingest - refresh tier (light incremental federal "what's new")
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+After=network-online.target docket-web.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=root
+WorkingDirectory=/opt/docket/services/ingest
+Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db
+# Keys (optional - pollers skip gracefully when absent)
+EnvironmentFile=-/opt/docket/.env
+
+# LOCK-SAFE: a single shared flock across the refresh/daily/weekly tiers makes
+# them mutually exclusive. --nonblock + --conflict-exit-code 0 means: if another
+# tier already holds the lock, skip this tick cleanly (exit 0) instead of running
+# a second concurrent writer. run.py itself runs every step sequentially, so the
+# DB only ever sees one sequential writer - WAL-safe against the site's readers.
+ExecStart=/bin/flock --nonblock --conflict-exit-code 0 /run/docket-ingest.lock /opt/docket/services/ingest/.venv/bin/python -m ingest.run --refresh
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=docket-refresh
+TimeoutStartSec=3600
+Nice=10
+
+[Install]
+WantedBy=multi-user.target
services/ingest/deploy/docket-refresh.timer +14 -0
@@ -0,0 +1,14 @@
+[Unit]
+Description=Docket refresh tier - twice daily (06:00 & 18:00 America/Chicago)
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+
+[Timer]
+OnCalendar=*-*-* 06,18:00 America/Chicago
+# Catch up one missed fire across reboots/downtime
+Persistent=true
+# Small jitter to avoid firing on the exact second
+RandomizedDelaySec=120
+Unit=docket-refresh.service
+
+[Install]
+WantedBy=timers.target
services/ingest/deploy/docket-watcher.service +5 -5
@@ -1,6 +1,6 @@
[Unit]
-Description=Docket ingest watcher - single cycle (ADDED/UPDATED/REMOVED detection)
-Documentation=file:/shared/projects/docket/services/ingest/WATCHER_NOTES.md
+Description=Docket ingest watcher -single cycle (ADDED/UPDATED/REMOVED detection)
+Documentation=https://github.com/zionboggan/docket/blob/main/services/ingest/WATCHER_NOTES.md
# Triggered by docket-watcher.timer; designed for one-shot use per timer tick.
# For continuous mode, change ExecStart to: python -m ingest.watcher --loop --interval 1800
After=network-online.target
@@ -14,17 +14,17 @@ WorkingDirectory=/opt/docket/services/ingest
# Python venv + module invocation
ExecStart=/opt/docket/services/ingest/.venv/bin/python -m ingest.watcher --once
-# DB lives in a local (non-CIFS) path - SQLite requirement
+# DB lives in a local (non-CIFS) path -SQLite requirement
Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db
-# Keys (optional - pollers skip gracefully when absent)
+# Keys (optional -pollers skip gracefully when absent)
EnvironmentFile=-/opt/docket/.env
StandardOutput=journal
StandardError=journal
SyslogIdentifier=docket-watcher
-# Restart on failure (but not on clean exit - oneshot)
+# Restart on failure (but not on clean exit -oneshot)
Restart=on-failure
RestartSec=60s
services/ingest/deploy/docket-watcher.timer +3 -3
@@ -1,13 +1,13 @@
[Unit]
-Description=Docket ingest watcher - run every 30 minutes
-Documentation=file:/shared/projects/docket/services/ingest/WATCHER_NOTES.md
+Description=Docket ingest watcher -run every 30 minutes
+Documentation=https://github.com/zionboggan/docket/blob/main/services/ingest/WATCHER_NOTES.md
[Timer]
# Fire 30 s after boot (gives network and other services time to settle)
OnBootSec=30s
# Then every 30 minutes
OnUnitActiveSec=30min
-# Persist across reboots - catch up on any missed fires
+# Persist across reboots -catch up on any missed fires
Persistent=true
# Randomise start within a 2-minute window to avoid thundering-herd
RandomizedDelaySec=120
services/ingest/deploy/docket-weekly.service +27 -0
@@ -0,0 +1,27 @@
+[Unit]
+Description=Docket ingest - weekly tier (full diff/rider/amendments + state pollers + coverage)
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+After=network-online.target docket-web.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=root
+WorkingDirectory=/opt/docket/services/ingest
+Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db
+EnvironmentFile=-/opt/docket/.env
+
+# Shared flock - see docket-refresh.service. Mutually exclusive with the other
+# two tiers; skips cleanly (exit 0) if a tier is already running. This is the
+# heaviest tier (full diff --all --congress 119 + state pollers) so it gets the
+# longest TimeoutStartSec.
+ExecStart=/bin/flock --nonblock --conflict-exit-code 0 /run/docket-ingest.lock /opt/docket/services/ingest/.venv/bin/python -m ingest.run --weekly
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=docket-weekly
+TimeoutStartSec=21600
+Nice=15
+
+[Install]
+WantedBy=multi-user.target
services/ingest/deploy/docket-weekly.timer +12 -0
@@ -0,0 +1,12 @@
+[Unit]
+Description=Docket weekly tier - Sundays (03:00 America/Chicago)
+Documentation=https://github.com/zionboggan/docket/blob/main/docs/REFRESH.md
+
+[Timer]
+OnCalendar=Sun *-*-* 03:00 America/Chicago
+Persistent=true
+RandomizedDelaySec=300
+Unit=docket-weekly.service
+
+[Install]
+WantedBy=timers.target
services/ingest/ingest/archive_bills.py +338 -0
@@ -0,0 +1,338 @@
+"""
+Bill PDF archive - download official PDFs from GovInfo and upload to R2.
+
+For each bill that has a version in `bill_versions`, we download the official
+PDF from GovInfo and upload it to the docket-archive R2 bucket so it is
+preserved even if GovInfo reorganizes or removes the file.
+
+GovInfo PDF URL pattern (no API key required):
+ https://www.govinfo.gov/content/pkg/{pkg_id}/pdf/{pkg_id}.pdf
+ e.g. https://www.govinfo.gov/content/pkg/BILLS-119hr160ih/pdf/BILLS-119hr160ih.pdf
+
+R2 key: bills/{congress}/{pkg_id}.pdf
+Public URL: https://archive.usdocket.org/bills/{congress}/{pkg_id}.pdf
+
+The `bills` table gets two new columns (additive):
+ pdf_url TEXT - official GovInfo PDF URL
+ archive_pdf_url TEXT - our R2 preservation copy URL
+
+Version priority (best/most-final version wins per bill):
+ enr > eh > es > ph > ps > rh > rs > ih > is > (anything else)
+
+Usage
+-----
+ python -m ingest.archive_bills --once
+ python -m ingest.archive_bills --once --congress 119 --limit 200
+ python -m ingest.archive_bills --once --types hr s
+
+Idempotent: bills with archive_pdf_url already set are skipped; R2 exists()
+check prevents re-uploading objects already in the bucket.
+"""
+
+import argparse
+import logging
+import sqlite3
+import sys
+import time
+from typing import List, Optional
+
+import requests
+
+from ingest.db import get_connection, init_schema
+from ingest.r2 import exists, put_object, public_url
+
+logger = logging.getLogger(__name__)
+
+SOURCE = "govinfo_billstatus"
+GOVINFO_BASE = "https://www.govinfo.gov/content/pkg"
+
+BILL_TYPES = ["hr", "s", "hjres", "sjres", "hconres", "sconres", "hres", "sres"]
+
+# Priority order for version selection: lower index = preferred (most authoritative).
+_VERSION_PRIORITY = [
+ "enr", # enrolled bill (signed into law or sent to president)
+ "eh", # engrossed in house
+ "es", # engrossed in senate
+ "eah", # engrossed amendment house
+ "eas", # engrossed amendment senate
+ "ph", # passed house
+ "ps", # passed senate
+ "rh", # reported in house
+ "rs", # reported in senate
+ "pch", # placed on calendar house
+ "pcs", # placed on calendar senate
+ "ih", # introduced in house
+ "is", # introduced in senate
+]
+
+_VERSION_RANK = {code: i for i, code in enumerate(_VERSION_PRIORITY)}
+
+
+# ---------------------------------------------------------------------------
+# Schema
+# ---------------------------------------------------------------------------
+
+def _ensure_columns(conn: sqlite3.Connection) -> None:
+ """Add pdf_url / archive_pdf_url columns to bills if they don't exist yet."""
+ for col, typedef in [("pdf_url", "TEXT"), ("archive_pdf_url", "TEXT")]:
+ try:
+ conn.execute(f"ALTER TABLE bills ADD COLUMN {col} {typedef}")
+ conn.commit()
+ logger.info("Added column bills.%s", col)
+ except sqlite3.OperationalError:
+ pass # column already exists
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _parse_external_id(external_id: str):
+ """
+ Decompose an external_id like '119-hr-160' or '119-hconres-1' into
+ (congress: int, bill_type: str, number: str).
+ """
+ congress_str, rest = external_id.split("-", 1)
+ bill_type, number = rest.rsplit("-", 1)
+ return int(congress_str), bill_type, number
+
+
+def _best_version(versions: list) -> Optional[dict]:
+ """
+ Given a list of dicts with 'version_code' and 'govinfo_package_id',
+ return the one with the highest priority (most authoritative).
+ Falls back to the first item if no known codes match.
+ """
+ if not versions:
+ return None
+
+ def rank(v):
+ return _VERSION_RANK.get(v["version_code"], len(_VERSION_PRIORITY))
+
+ return min(versions, key=rank)
+
+
+def _govinfo_pdf_url(pkg_id: str) -> str:
+ return f"{GOVINFO_BASE}/{pkg_id}/pdf/{pkg_id}.pdf"
+
+
+def _r2_key(congress: int, pkg_id: str) -> str:
+ return f"bills/{congress}/{pkg_id}.pdf"
+
+
+# ---------------------------------------------------------------------------
+# Main poll
+# ---------------------------------------------------------------------------
+
+def poll(
+ congress: int = 119,
+ bill_types: Optional[List[str]] = None,
+ db_path: Optional[str] = None,
+ limit: Optional[int] = 200,
+ delay: float = 0.5,
+) -> int:
+ """
+ Archive bill PDFs for the given congress to R2.
+
+ For each bill in bill_versions (current congress), selects the most
+ authoritative version, downloads the PDF from GovInfo, and uploads to R2.
+ Skips bills that already have an archive_pdf_url.
+
+ Parameters
+ ----------
+ congress : Congress number (default 119).
+ bill_types : Bill types to process (default = all types in BILL_TYPES).
+ db_path : Override SQLite path.
+ limit : Max PDFs to archive per run (None = unlimited).
+ delay : Seconds between GovInfo PDF downloads.
+
+ Returns
+ -------
+ Number of PDFs newly archived.
+ """
+ if bill_types is None:
+ bill_types = BILL_TYPES
+
+ conn = get_connection(db_path) if db_path else get_connection()
+ init_schema(conn)
+ _ensure_columns(conn)
+
+ session = requests.Session()
+ session.headers["User-Agent"] = "Docket-ingest/1.0 (public transparency research)"
+
+ type_set = set(bill_types)
+
+ # Query bills for this congress that don't have an archive copy yet,
+ # joined to bill_versions so we can find their package IDs.
+ rows = conn.execute(
+ """
+ SELECT DISTINCT b.id, b.external_id,
+ bv.version_code, bv.govinfo_package_id
+ FROM bills b
+ JOIN bill_versions bv ON bv.bill_id = b.id
+ WHERE b.source = ?
+ AND b.congress = ?
+ AND (b.archive_pdf_url IS NULL OR b.archive_pdf_url = '')
+ ORDER BY b.id
+ """,
+ (SOURCE, congress),
+ ).fetchall()
+
+ # Group by bill_id to select the best version per bill
+ by_bill: dict = {}
+ for row in rows:
+ bid = row["id"]
+ if bid not in by_bill:
+ by_bill[bid] = {"id": bid, "external_id": row["external_id"], "versions": []}
+ by_bill[bid]["versions"].append({
+ "version_code": row["version_code"],
+ "govinfo_package_id": row["govinfo_package_id"],
+ })
+
+ bills_list = list(by_bill.values())
+ logger.info(
+ "archive_bills: %d bills to process (congress %d, types=%s, limit=%s)",
+ len(bills_list), congress, bill_types, limit,
+ )
+
+ archived = 0
+ skipped_type = 0
+ skipped_r2 = 0
+ failed = 0
+
+ for bill in bills_list:
+ if limit is not None and archived >= limit:
+ logger.info("archive_bills: limit %d reached, stopping.", limit)
+ break
+
+ ext_id = bill["external_id"]
+ bill_id = bill["id"]
+
+ try:
+ cong, bill_type, number = _parse_external_id(ext_id)
+ except Exception:
+ logger.debug("Cannot parse external_id %s, skipping.", ext_id)
+ continue
+
+ if bill_type not in type_set:
+ skipped_type += 1
+ continue
+
+ best = _best_version(bill["versions"])
+ if not best:
+ continue
+
+ pkg_id = best["govinfo_package_id"]
+ if not pkg_id:
+ continue
+
+ r2_key = _r2_key(cong, pkg_id)
+ gov_pdf_url = _govinfo_pdf_url(pkg_id)
+
+ # Idempotent: check R2 first
+ if exists(r2_key):
+ archive_url = public_url(r2_key)
+ conn.execute(
+ "UPDATE bills SET pdf_url=?, archive_pdf_url=? WHERE id=?",
+ (gov_pdf_url, archive_url, bill_id),
+ )
+ skipped_r2 += 1
+ continue
+
+ # Download PDF from GovInfo
+ logger.debug("Downloading PDF: %s", gov_pdf_url)
+ try:
+ r = session.get(gov_pdf_url, timeout=60, allow_redirects=False)
+ except requests.RequestException as exc:
+ logger.warning("PDF fetch failed for %s: %s", ext_id, exc)
+ failed += 1
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ if r.status_code != 200 or "application/pdf" not in r.headers.get("content-type", ""):
+ logger.debug(
+ "HTTP %d / Content-Type %r for %s - skipping",
+ r.status_code, r.headers.get("content-type", ""), ext_id,
+ )
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ pdf_bytes = r.content
+ if len(pdf_bytes) < 1024:
+ logger.debug("PDF too small (%d bytes) for %s, skipping", len(pdf_bytes), ext_id)
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ # Upload to R2
+ archive_url = put_object(r2_key, pdf_bytes, content_type="application/pdf")
+ if not archive_url:
+ logger.warning("R2 upload failed for %s (key=%s)", ext_id, r2_key)
+ failed += 1
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ # Store both URLs in bills table
+ conn.execute(
+ "UPDATE bills SET pdf_url=?, archive_pdf_url=? WHERE id=?",
+ (gov_pdf_url, archive_url, bill_id),
+ )
+ archived += 1
+
+ if archived % 25 == 0:
+ conn.commit()
+ logger.info("archive_bills: %d PDFs archived so far", archived)
+
+ if delay > 0:
+ time.sleep(delay)
+
+ conn.commit()
+ conn.close()
+
+ logger.info(
+ "archive_bills: done - %d archived, %d already in R2, %d skipped (type), %d failed",
+ archived, skipped_r2, skipped_type, failed,
+ )
+ return archived
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+ stream=sys.stdout,
+ )
+ ap = argparse.ArgumentParser(
+ description="Archive official bill PDFs from GovInfo to R2"
+ )
+ ap.add_argument("--once", action="store_true", help="Run once and exit")
+ ap.add_argument("--congress", type=int, default=119, help="Congress number (default 119)")
+ ap.add_argument(
+ "--types", nargs="+", default=None,
+ help="Bill types to process (hr s hjres …); default = all",
+ )
+ ap.add_argument(
+ "--limit", type=int, default=200,
+ help="Max PDFs to archive per run (default 200; 0 = unlimited)",
+ )
+ ap.add_argument(
+ "--delay", type=float, default=0.5,
+ help="Seconds between GovInfo PDF downloads (default 0.5)",
+ )
+ args = ap.parse_args()
+
+ limit_val = args.limit if args.limit and args.limit > 0 else None
+ n = poll(
+ congress=args.congress,
+ bill_types=args.types,
+ limit=limit_val,
+ delay=args.delay,
+ )
+ print(f"Done: {n} PDFs archived")
services/ingest/ingest/archive_votes.py +566 -0
@@ -0,0 +1,566 @@
+"""
+Rolling R2 archive for recorded congressional votes.
+
+Snapshots the authoritative roll-call XML from official .gov sources and
+uploads it to Cloudflare R2 (bucket: docket-archive, public at
+https://archive.usdocket.org) for long-term preservation.
+
+Also performs a one-time backfill of broken source_urls:
+ - House: clerk.house.gov/Votes/{year}/roll{NNN} → /evs/{year}/roll{NNN:03d}.xml (was 404)
+ - Senate: ...vote_{c}_{s}_{n:05d}.xml → ...vote_{c}_{s}_{n:05d}.htm (XML→readable page)
+
+R2 key scheme:
+ Senate: votes/senate/{congress}/{session}/{per_session_roll:05d}.xml
+ House: votes/house/{year}/{roll_number:03d}.xml
+
+SENATE SESSION NOTE
+-------------------
+senate.gov resets its roll-call numbers at the start of each new session.
+Voteview numbers Senate votes cumulatively across the congress (session 2
+votes continue from where session 1 ended). This module corrects the offset
+before constructing senate.gov URLs: for sessions > 1 it queries the DB for
+the max roll number from the previous session and subtracts that as the
+per-session offset.
+
+CONTENT VALIDATION
+------------------
+Responses are validated to contain XML (starts with "<?xml" or "<rollcall"
+or similar) before uploading. If senate.gov redirects to an error page, the
+HTML content is detected and the vote is skipped cleanly.
+
+archive_url is stored back on the votes row so the frontend can show
+"Archived copy" alongside "Official source."
+
+Usage:
+ python -m ingest.archive_votes --once
+ python -m ingest.archive_votes --once --limit 200
+ python -m ingest.archive_votes --once --chamber senate
+ python -m ingest.archive_votes --once --skip-source-fix # skip URL backfill
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import re
+import sqlite3
+import time
+from typing import Optional
+
+import requests
+
+from ingest.db import get_connection, init_schema
+from ingest.r2 import exists, put_object, public_url
+
+logger = logging.getLogger(__name__)
+
+# ── Constants ─────────────────────────────────────────────────────────────────
+
+SENATE_XML_BASE = "https://www.senate.gov/legislative/LIS/roll_call_votes"
+HOUSE_EVS_BASE = "https://clerk.house.gov/evs"
+
+# Delay between HTTP fetches to be polite to .gov servers.
+FETCH_DELAY_S = 0.4
+
+# Regex to parse senate vote XML filename: vote_{c}_{s}_{n:05d}.xml
+_SENATE_RE = re.compile(
+ r"/vote(\d+)(\d)/vote_(\d+)_(\d+)_(\d{5})\.(?:xml|htm)$"
+)
+
+
+# ── Schema ────────────────────────────────────────────────────────────────────
+
+def ensure_archive_schema(conn: sqlite3.Connection) -> None:
+ """Add archive_url column to votes if it does not exist yet."""
+ try:
+ conn.execute("ALTER TABLE votes ADD COLUMN archive_url TEXT")
+ conn.commit()
+ logger.info("archive_votes: added archive_url column to votes")
+ except Exception:
+ pass # already present
+
+
+# ── Source-URL backfill ───────────────────────────────────────────────────────
+
+def fix_source_urls(conn: sqlite3.Connection) -> tuple[int, int]:
+ """
+ One-time (idempotent) correction of broken source_url values.
+
+ House: clerk.house.gov/Votes/{year}/roll{NNN} → /evs/{year}/roll{NNN:03d}.xml
+ Senate: ...{n:05d}.xml → ...{n:05d}.htm
+
+ Returns (house_fixed, senate_fixed) counts.
+ """
+ # House: /Votes/ path returns 404; /evs/ + .xml returns 200.
+ # URL structure is identical except for the path prefix and the .xml suffix.
+ cur = conn.execute(
+ """
+ UPDATE votes
+ SET source_url = 'https://clerk.house.gov/evs/'
+ || substr(source_url, length('https://clerk.house.gov/Votes/') + 1)
+ || '.xml'
+ WHERE chamber = 'house'
+ AND source_url LIKE 'https://clerk.house.gov/Votes/%'
+ """
+ )
+ house_fixed = cur.rowcount
+
+ # Senate: swap .xml extension for .htm so users land on the HTML page.
+ cur = conn.execute(
+ """
+ UPDATE votes
+ SET source_url = substr(source_url, 1, length(source_url) - 3) || 'htm'
+ WHERE chamber = 'senate'
+ AND source_url LIKE '%.xml'
+ """
+ )
+ senate_fixed = cur.rowcount
+
+ conn.commit()
+ logger.info(
+ "archive_votes: source_url fix - house %d, senate %d",
+ house_fixed, senate_fixed,
+ )
+ return house_fixed, senate_fixed
+
+
+# ── R2 key helpers ────────────────────────────────────────────────────────────
+
+def _senate_r2_key(congress: int, session: int, per_session_roll: int) -> str:
+ return f"votes/senate/{congress}/{session}/{per_session_roll:05d}.xml"
+
+
+def _house_r2_key(year: int, roll_number: int) -> str:
+ return f"votes/house/{year}/{roll_number:03d}.xml"
+
+
+def _senate_xml_url(congress: int, session: int, per_session_roll: int) -> str:
+ return (
+ f"{SENATE_XML_BASE}/vote{congress}{session}/"
+ f"vote_{congress}_{session}_{per_session_roll:05d}.xml"
+ )
+
+
+def _house_xml_url(year: int, roll_number: int) -> str:
+ return f"{HOUSE_EVS_BASE}/{year}/roll{roll_number:03d}.xml"
+
+
+def _is_xml_content(data: bytes) -> bool:
+ """
+ Return True if *data* looks like XML roll-call content.
+
+ senate.gov and clerk.house.gov redirect unavailable votes to HTML error
+ pages; we detect and reject those to avoid archiving garbage.
+ """
+ if not data:
+ return False
+ head = data[:200].lstrip()
+ return (
+ head.startswith(b"<?xml")
+ or head.startswith(b"<rollcall")
+ or head.startswith(b"<vote_summary")
+ )
+
+
+def _parse_senate_source_url(source_url: str) -> Optional[tuple[int, int, int]]:
+ """
+ Extract (congress, session, per_session_roll) from a senate source_url.
+
+ The URL always encodes the actual per-session roll number issued by
+ senate.gov, so this is reliable even when the DB roll_number is the
+ cumulative Voteview number.
+
+ Handles both .htm and .xml forms produced by our pollers.
+ Returns None on parse failure.
+ """
+ m = _SENATE_RE.search(source_url)
+ if not m:
+ return None
+ try:
+ congress = int(m.group(3))
+ session = int(m.group(4))
+ roll_number = int(m.group(5))
+ return congress, session, roll_number
+ except (ValueError, IndexError):
+ return None
+
+
+def _derive_senate_metadata(
+ conn: sqlite3.Connection,
+ row: sqlite3.Row,
+) -> Optional[tuple[int, int, int]]:
+ """
+ Return (congress, session, per_session_roll) for a senate vote row.
+
+ senate.gov issues per-session roll numbers (resets to 1 each new session).
+ Voteview stores cumulative roll numbers across the congress, so for session
+ 2+ we must subtract the session-1 maximum to get the senate.gov number.
+
+ Strategy:
+ - votes_senate.py entries (voteview_rollnumber IS NULL): parse source_url -
+ the URL carries the correct per-session number for these rows.
+ - Voteview entries session 1 (voteview_rollnumber IS NOT NULL, session <= 1):
+ roll_number IS the per-session number; use DB columns.
+ - Voteview entries session 2+ (voteview_rollnumber IS NOT NULL, session > 1):
+ roll_number is cumulative; subtract prev-session max to get per-session roll.
+ """
+ congress = row["congress"]
+ session = row["session"] or 1
+ roll_number = row["roll_number"]
+ voteview_roll = row["voteview_rollnumber"]
+
+ # votes_senate.py entries: parse from source_url (correct per-session number).
+ if voteview_roll is None:
+ parsed = _parse_senate_source_url(row["source_url"] or "")
+ return parsed # None if unparseable
+
+ # Voteview entries: use DB columns.
+ if not (congress and roll_number):
+ return None
+
+ congress = int(congress)
+ session = int(session)
+ roll_number = int(roll_number)
+
+ if session <= 1:
+ # Session 1: Voteview rollnumber == senate.gov per-session roll.
+ return congress, session, roll_number
+
+ # Session 2+: Voteview uses cumulative numbering across the congress.
+ # Subtract the max roll from the preceding session to get the per-session roll.
+ prev_session_max = conn.execute(
+ """
+ SELECT MAX(roll_number)
+ FROM votes
+ WHERE chamber = 'senate'
+ AND congress = ?
+ AND session = ?
+ """,
+ (congress, session - 1),
+ ).fetchone()[0]
+
+ if not prev_session_max:
+ # Cannot compute offset; skip this vote.
+ return None
+
+ per_session_roll = roll_number - int(prev_session_max)
+ if per_session_roll <= 0:
+ return None
+
+ return congress, session, per_session_roll
+
+
+def _derive_house_year(row: sqlite3.Row) -> Optional[int]:
+ """Derive year from the year column or from date[:4]."""
+ if row["year"]:
+ return int(row["year"])
+ date_str = row["date"] or ""
+ try:
+ return int(date_str[:4])
+ except (ValueError, TypeError):
+ return None
+
+
+# ── Core archive loop ─────────────────────────────────────────────────────────
+
+def archive_votes(
+ conn: sqlite3.Connection,
+ chamber: Optional[str] = None,
+ limit: int = 500,
+) -> int:
+ """
+ Archive up to *limit* unarchived recorded votes to R2.
+
+ Returns the number of votes successfully archived this run.
+ """
+ where_chamber = "AND chamber = ?" if chamber else ""
+
+ params: list = []
+ if chamber:
+ params.append(chamber)
+ params.append(limit)
+
+ rows = conn.execute(
+ f"""
+ SELECT id, chamber, date, year, congress, session, roll_number,
+ source_url, voteview_rollnumber
+ FROM votes
+ WHERE recorded = 1
+ AND archive_url IS NULL
+ {where_chamber}
+ ORDER BY date DESC, id DESC
+ LIMIT ?
+ """,
+ params,
+ ).fetchall()
+
+ archived = 0
+
+ for row in rows:
+ vote_id = row["id"]
+ ch = row["chamber"]
+
+ if ch == "senate":
+ meta = _derive_senate_metadata(conn, row)
+ if not meta:
+ logger.debug(
+ "archive_votes: senate vote id=%d: cannot derive metadata, skipping",
+ vote_id,
+ )
+ continue
+ congress, session, per_session_roll = meta
+ r2_key = _senate_r2_key(congress, session, per_session_roll)
+ xml_url = _senate_xml_url(congress, session, per_session_roll)
+
+ elif ch == "house":
+ year = _derive_house_year(row)
+ if not year or not row["roll_number"]:
+ logger.debug(
+ "archive_votes: house vote id=%d: missing year/roll_number, skipping",
+ vote_id,
+ )
+ continue
+ roll_number = int(row["roll_number"])
+ r2_key = _house_r2_key(year, roll_number)
+ xml_url = _house_xml_url(year, roll_number)
+
+ else:
+ logger.debug("archive_votes: unknown chamber %r for id=%d, skipping", ch, vote_id)
+ continue
+
+ # If this key is already in R2, verify it's valid XML before recording.
+ # Bad R2 objects (HTML error pages from earlier runs) must be skipped
+ # so purge_bad_archives() can clear them and they can be re-archived.
+ if exists(r2_key):
+ pub = public_url(r2_key)
+ try:
+ r2_resp = requests.get(pub, timeout=20)
+ if r2_resp.ok and _is_xml_content(r2_resp.content):
+ conn.execute(
+ "UPDATE votes SET archive_url = ? WHERE id = ?",
+ (pub, vote_id),
+ )
+ conn.commit()
+ archived += 1
+ else:
+ # Bad content in R2; skip - purge_bad_archives() will clear it.
+ logger.debug(
+ "archive_votes: R2 key %s exists but content is not XML; "
+ "skipping (run purge to clear)",
+ r2_key,
+ )
+ except Exception:
+ pass
+ continue
+
+ # Fetch the XML from .gov.
+ try:
+ resp = requests.get(xml_url, timeout=30)
+ except requests.RequestException as exc:
+ logger.warning("archive_votes: fetch error %s: %s", xml_url, exc)
+ time.sleep(FETCH_DELAY_S)
+ continue
+
+ if resp.status_code == 404:
+ logger.debug("archive_votes: 404 for %s (skipping)", xml_url)
+ time.sleep(FETCH_DELAY_S * 0.5)
+ continue
+ if not resp.ok:
+ logger.warning(
+ "archive_votes: HTTP %d for %s", resp.status_code, xml_url
+ )
+ time.sleep(FETCH_DELAY_S)
+ continue
+
+ data = resp.content
+ if not data:
+ logger.debug("archive_votes: empty body for %s", xml_url)
+ time.sleep(FETCH_DELAY_S * 0.5)
+ continue
+
+ # Validate that we received actual XML, not an error HTML page.
+ # senate.gov and clerk.house.gov redirect unavailable votes to HTML.
+ if not _is_xml_content(data):
+ logger.debug(
+ "archive_votes: non-XML response for %s "
+ "(redirect to error page?), skipping",
+ xml_url,
+ )
+ time.sleep(FETCH_DELAY_S * 0.5)
+ continue
+
+ # Upload to R2.
+ pub = put_object(r2_key, data, content_type="application/xml")
+ if pub is None:
+ logger.warning(
+ "archive_votes: R2 upload failed or creds absent for %s", r2_key
+ )
+ # Do NOT mark archive_url: the upload failed (creds may simply be
+ # absent in test environments), so this vote stays unarchived and
+ # is retried on a later run.
+ time.sleep(FETCH_DELAY_S)
+ continue
+
+ conn.execute(
+ "UPDATE votes SET archive_url = ? WHERE id = ?",
+ (pub, vote_id),
+ )
+ conn.commit()
+ archived += 1
+
+ logger.debug("archive_votes: archived %s → %s", xml_url, pub)
+ time.sleep(FETCH_DELAY_S)
+
+ logger.info(
+ "archive_votes: archived %d/%d votes (chamber=%s, limit=%d)",
+ archived, len(rows), chamber or "all", limit,
+ )
+ return archived
+
+
+# ── Entry point ───────────────────────────────────────────────────────────────
+
+def purge_bad_archives(conn: sqlite3.Connection) -> int:
+ """
+ Reset archive_url for any vote whose archived content is not valid XML.
+
+ On the first run we may have uploaded HTML error pages (from senate.gov
+ redirects) before the XML-validation guard was in place. This function
+ fetches each archived URL, checks the content, and clears archive_url if
+ the content is not XML so the vote is re-queued on the next archive run.
+ """
+ rows = conn.execute(
+ "SELECT id, archive_url FROM votes WHERE archive_url IS NOT NULL"
+ ).fetchall()
+
+ purged = 0
+ for row in rows:
+ vote_id = row["id"]
+ archive_url = row["archive_url"]
+ try:
+ resp = requests.get(archive_url, timeout=20)
+ if not resp.ok or not _is_xml_content(resp.content):
+ conn.execute(
+ "UPDATE votes SET archive_url = NULL WHERE id = ?",
+ (vote_id,),
+ )
+ purged += 1
+ logger.debug(
+ "archive_votes: purged bad archive for id=%d (%s)",
+ vote_id, archive_url,
+ )
+ except Exception as exc:
+ logger.debug(
+ "archive_votes: purge check error for id=%d: %s", vote_id, exc
+ )
+ time.sleep(0.1)
+
+ if purged:
+ conn.commit()
+ logger.info("archive_votes: purged %d bad archive entries", purged)
+ return purged
+
+
+def run(
+ db_path: Optional[str] = None,
+ chamber: Optional[str] = None,
+ limit: int = 500,
+ skip_source_fix: bool = False,
+ purge: bool = True,
+) -> dict:
+ """
+ Full archive run:
+ 1. Ensure archive_url column exists.
+ 2. Fix broken source_urls (idempotent).
+ 3. Purge any previously-archived entries that contain HTML error pages.
+ 4. Archive up to *limit* unarchived recorded votes to R2.
+
+ Returns a dict with counts.
+ """
+ import sqlite3 as _sqlite3
+ db = db_path or __import__("ingest.db", fromlist=["DB_PATH"]).DB_PATH
+ conn = _sqlite3.connect(db, timeout=60)
+ conn.row_factory = _sqlite3.Row
+ conn.execute("PRAGMA foreign_keys=ON")
+ conn.execute("PRAGMA busy_timeout=60000")
+ init_schema(conn)
+ ensure_archive_schema(conn)
+
+ house_fixed = senate_fixed = 0
+ if not skip_source_fix:
+ house_fixed, senate_fixed = fix_source_urls(conn)
+
+ purged = 0
+ if purge:
+ purged = purge_bad_archives(conn)
+
+ archived = archive_votes(conn, chamber=chamber, limit=limit)
+ conn.close()
+
+ return {
+ "house_source_urls_fixed": house_fixed,
+ "senate_source_urls_fixed": senate_fixed,
+ "bad_archives_purged": purged,
+ "votes_archived": archived,
+ }
+
+
+# ── CLI ───────────────────────────────────────────────────────────────────────
+
+def main() -> None:
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+ )
+ parser = argparse.ArgumentParser(
+ description="Archive recorded congressional votes to R2."
+ )
+ parser.add_argument(
+ "--once", action="store_true",
+ help="Run once and exit (required for CLI use).",
+ )
+ parser.add_argument(
+ "--limit", type=int, default=500,
+ help="Max votes to archive per run (default 500).",
+ )
+ parser.add_argument(
+ "--chamber", choices=["senate", "house"],
+ help="Archive only one chamber.",
+ )
+ parser.add_argument(
+ "--db",
+ help="Override DB path (uses DOCKET_DB_PATH env or default if omitted).",
+ )
+ parser.add_argument(
+ "--skip-source-fix", action="store_true",
+ help="Skip the source_url backfill step.",
+ )
+ parser.add_argument(
+ "--no-purge", action="store_true",
+ help="Skip the bad-archive purge check (faster for routine runs).",
+ )
+ args = parser.parse_args()
+
+ if not args.once:
+ parser.print_help()
+ return
+
+ result = run(
+ db_path=args.db,
+ chamber=args.chamber,
+ limit=args.limit,
+ skip_source_fix=args.skip_source_fix,
+ purge=not args.no_purge,
+ )
+
+ print(
+ f"\narchive_votes complete:\n"
+ f" house source_urls fixed: {result['house_source_urls_fixed']}\n"
+ f" senate source_urls fixed: {result['senate_source_urls_fixed']}\n"
+ f" bad archives purged: {result['bad_archives_purged']}\n"
+ f" votes archived to R2: {result['votes_archived']}\n"
+ )
+
+
+if __name__ == "__main__":
+ main()
services/ingest/ingest/bill_summaries.py +318 -0
@@ -0,0 +1,318 @@
+"""
+Bill summaries ingest - CRS plain-language summaries from GovInfo BILLSTATUS XML.
+
+The BILLSTATUS XML already fetched by bills_bulk contains a <summaries> element
+with official CRS (Congressional Research Service) text. This module backfills
+the `summary` and `summary_source` columns on the `bills` table by re-fetching
+those XMLs for bills that don't have a summary yet.
+
+Source: GovInfo BILLSTATUS (bulkdata, keyless)
+ https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/BILLSTATUS-{congress}{type}{num}.xml
+
+Usage
+-----
+ python -m ingest.bill_summaries --once
+ python -m ingest.bill_summaries --once --congress 119 --limit 500
+ python -m ingest.bill_summaries --once --types hr s # specific bill types only
+
+The summary text is stored verbatim (US gov CRS text, public domain). HTML tags
+are stripped; HTML entities are decoded. Attribution: "CRS via GovInfo".
+"""
+
+import argparse
+import html
+import logging
+import re
+import sqlite3
+import sys
+import time
+import xml.etree.ElementTree as ET
+from typing import List, Optional
+
+import requests
+
+from ingest.db import get_connection, init_schema
+
+logger = logging.getLogger(__name__)
+
+SOURCE = "govinfo_billstatus"
+SUMMARY_SOURCE = "CRS via GovInfo"
+BULKDATA_BASE = "https://www.govinfo.gov/bulkdata/BILLSTATUS"
+
+BILL_TYPES = ["hr", "s", "hjres", "sjres", "hconres", "sconres", "hres", "sres"]
+
+_HTML_TAG_RE = re.compile(r"<[^>]+>")
+# Block-level tags become paragraph breaks; any other tag becomes a single
+# space, so adjacent words/sentences never run together (e.g. "voters.First").
+_BLOCK_TAG_RE = re.compile(r"</?(p|div|br|li|ul|ol|h[1-6]|tr|table)\b[^>]*>", re.I)
+_WS_RUN_RE = re.compile(r"[ \t]+")
+_BLANKS_RE = re.compile(r"\n{3,}")
+
+
+# ---------------------------------------------------------------------------
+# Schema
+# ---------------------------------------------------------------------------
+
+def _ensure_columns(conn: sqlite3.Connection) -> None:
+ """Add summary / summary_source columns to bills if they don't exist yet."""
+ for col, typedef in [("summary", "TEXT"), ("summary_source", "TEXT")]:
+ try:
+ conn.execute(f"ALTER TABLE bills ADD COLUMN {col} {typedef}")
+ conn.commit()
+ logger.info("Added column bills.%s", col)
+ except sqlite3.OperationalError:
+ pass # column already exists
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _parse_external_id(external_id: str):
+ """
+ Decompose an external_id like '119-hr-160' or '119-hconres-1' into
+ (congress: int, bill_type: str, number: str).
+ """
+ congress_str, rest = external_id.split("-", 1)
+ bill_type, number = rest.rsplit("-", 1)
+ return int(congress_str), bill_type, number
+
+
+def _billstatus_url(congress: int, bill_type: str, number: str) -> str:
+ return (
+ f"{BULKDATA_BASE}/{congress}/{bill_type}/"
+ f"BILLSTATUS-{congress}{bill_type}{number}.xml"
+ )
+
+
+def _strip_html(raw: str) -> str:
+ """Decode HTML entities and strip HTML tags from a CRS summary text field.
+
+ Block tags -> newline, inline tags -> space, so sentences and words never
+ concatenate ("voters.</p><p>First" -> "voters.\n\nFirst", not "voters.First").
+ """
+ unescaped = html.unescape(raw)
+ # Block boundaries become paragraph breaks.
+ text = _BLOCK_TAG_RE.sub("\n", unescaped)
+ # Any remaining tag becomes a space.
+ text = _HTML_TAG_RE.sub(" ", text)
+ # Collapse runs of spaces/tabs, then trim excess blank lines, per line.
+ text = _WS_RUN_RE.sub(" ", text)
+ text = "\n".join(line.strip() for line in text.split("\n"))
+ text = _BLANKS_RE.sub("\n\n", text)
+ return text.strip()
+
+
+def _extract_summary(xml_bytes: bytes) -> Optional[str]:
+ """
+ Parse a BILLSTATUS XML byte string and return the best available CRS
+ summary text (plain text, HTML-stripped), or None if not present.
+
+ Strategy: take the item whose <updateDate> is latest. Falls back to the
+ last <item> in document order if dates are missing.
+ """
+ try:
+ root = ET.fromstring(xml_bytes)
+ except ET.ParseError:
+ return None
+
+ bill_el = root.find("bill")
+ if bill_el is None:
+ if root.tag in ("bill", "Bill"):
+ bill_el = root
+ else:
+ return None
+
+ summaries_el = bill_el.find("summaries")
+ if summaries_el is None:
+ return None
+
+ items = summaries_el.findall("summary")
+ if not items:
+ return None
+
+ # Pick the item with the latest updateDate
+ best_item = None
+ best_date = ""
+ for item in items:
+ upd_el = item.find("updateDate")
+ upd = upd_el.text.strip() if upd_el is not None and upd_el.text else ""
+ if upd >= best_date:
+ best_date = upd
+ best_item = item
+
+ if best_item is None:
+ return None
+
+ text_el = best_item.find("text")
+ if text_el is None or not text_el.text:
+ return None
+
+ return _strip_html(text_el.text) or None
+
+
+# ---------------------------------------------------------------------------
+# Main poll
+# ---------------------------------------------------------------------------
+
+def poll(
+ congress: int = 119,
+ bill_types: Optional[List[str]] = None,
+ db_path: Optional[str] = None,
+ limit: Optional[int] = 500,
+ delay: float = 0.1,
+) -> int:
+ """
+ Backfill CRS summaries for bills that don't have one yet.
+
+ Fetches BILLSTATUS XML from GovInfo (no API key required) and stores the
+ best available summary text. Idempotent: skips bills that already have a
+ summary. Polite: `delay` seconds between requests.
+
+ Parameters
+ ----------
+ congress : Congress number to process (default 119).
+ bill_types : Bill types to process (default = all types in BILL_TYPES).
+ db_path : Override SQLite path.
+ limit : Maximum bills to process per run (None = unlimited).
+ delay : Seconds between GovInfo requests.
+
+ Returns
+ -------
+ Number of summaries newly stored.
+ """
+ if bill_types is None:
+ bill_types = BILL_TYPES
+
+ conn = get_connection(db_path) if db_path else get_connection()
+ init_schema(conn)
+ _ensure_columns(conn)
+
+ session = requests.Session()
+ session.headers["User-Agent"] = "Docket-ingest/1.0 (public transparency research)"
+
+ # Build a set of bill_types as uppercase for matching the DB external_id field.
+ # external_id pattern: "{congress}-{bill_type}-{number}"
+ type_set = set(bill_types)
+
+ # Query bills missing summaries for this congress + source
+ rows = conn.execute(
+ """
+ SELECT id, external_id
+ FROM bills
+ WHERE source = ?
+ AND congress = ?
+ AND (summary IS NULL OR summary = '')
+ ORDER BY id
+ """,
+ (SOURCE, congress),
+ ).fetchall()
+
+ logger.info(
+ "bill_summaries: %d congress-%d bills missing summaries (types=%s, limit=%s)",
+ len(rows), congress, bill_types, limit,
+ )
+
+ stored = 0
+ skipped_type = 0
+
+ for row in rows:
+ if limit is not None and stored >= limit:
+ logger.info("bill_summaries: limit %d reached, stopping.", limit)
+ break
+
+ ext_id = row["external_id"]
+ bill_id = row["id"]
+
+ try:
+ cong, bill_type, number = _parse_external_id(ext_id)
+ except Exception:
+ logger.debug("Cannot parse external_id %s, skipping.", ext_id)
+ continue
+
+ if bill_type not in type_set:
+ skipped_type += 1
+ continue
+
+ url = _billstatus_url(cong, bill_type, number)
+ logger.debug("Fetching BILLSTATUS: %s", url)
+
+ try:
+ r = session.get(url, timeout=30, allow_redirects=True)
+ except requests.RequestException as exc:
+ logger.warning("Fetch failed for %s: %s", ext_id, exc)
+ continue
+
+ if r.status_code != 200:
+ logger.debug("HTTP %d for %s", r.status_code, ext_id)
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ summary_text = _extract_summary(r.content)
+ if not summary_text:
+ logger.debug("No summary in BILLSTATUS for %s", ext_id)
+ if delay > 0:
+ time.sleep(delay)
+ continue
+
+ conn.execute(
+ "UPDATE bills SET summary=?, summary_source=? WHERE id=?",
+ (summary_text, SUMMARY_SOURCE, bill_id),
+ )
+ stored += 1
+
+ if stored % 50 == 0:
+ conn.commit()
+ logger.info("bill_summaries: %d summaries committed so far", stored)
+
+ if delay > 0:
+ time.sleep(delay)
+
+ conn.commit()
+ conn.close()
+
+ logger.info(
+ "bill_summaries: done - %d summaries stored, %d skipped (type filter)",
+ stored, skipped_type,
+ )
+ return stored
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+ stream=sys.stdout,
+ )
+ ap = argparse.ArgumentParser(
+ description="Backfill CRS summaries from GovInfo BILLSTATUS XML"
+ )
+ ap.add_argument("--once", action="store_true", help="Run once and exit")
+ ap.add_argument("--congress", type=int, default=119, help="Congress number (default 119)")
+ ap.add_argument(
+ "--types", nargs="+", default=None,
+ help="Bill types to process (hr s hjres …); default = all",
+ )
+ ap.add_argument(
+ "--limit", type=int, default=500,
+ help="Max summaries to fetch per run (default 500; 0 = unlimited)",
+ )
+ ap.add_argument(
+ "--delay", type=float, default=0.1,
+ help="Seconds between GovInfo requests (default 0.1)",
+ )
+ args = ap.parse_args()
+
+ limit_val = args.limit if args.limit and args.limit > 0 else None
+ n = poll(
+ congress=args.congress,
+ bill_types=args.types,
+ limit=limit_val,
+ delay=args.delay,
+ )
+ print(f"Done: {n} summaries stored")
services/ingest/ingest/calendar.py +3 -3
@@ -514,7 +514,7 @@ def _aggregate_votes(conn: sqlite3.Connection) -> int:
def _aggregate_bill_actions(conn: sqlite3.Connection) -> int:
"""
- introduced / committee / floor - key bill action dates from actions table.
+ introduced / committee / floor - key bill action dates from actions table.
Mapping:
IntroReferral → introduced
@@ -565,7 +565,7 @@ def _aggregate_bill_actions(conn: sqlite3.Connection) -> int:
def _aggregate_apportionments(conn: sqlite3.Connection) -> int:
"""
- apportionment - apportionment dates from the apportionments table.
+ apportionment - apportionment dates from the apportionments table.
"""
# Check the apportionments table exists
tbl = conn.execute(
@@ -612,7 +612,7 @@ def _aggregate_apportionments(conn: sqlite3.Connection) -> int:
def _aggregate_effective_parsed(conn: sqlite3.Connection) -> int:
"""
- effective_parsed - Effective dates parsed from bill/action text via regex.
+ effective_parsed - Effective dates parsed from bill/action text via regex.
Scans:
1. actions.text for all BecameLaw / President-signed rows (short text)
services/ingest/ingest/pollers/legiscan_dataset_summaries.py +267 -0
@@ -0,0 +1,267 @@
+"""
+LegiScan BULK dataset summary backfill for STATE bills.
+
+Populates bills.summary + bills.summary_source='LegiScan' for source='legiscan'
+rows that have no summary yet, using the LegiScan BULK dataset API instead of
+191k individual getBill calls (quota-safe).
+
+Method
+------
+ 1. getDatasetList (1 API call) -> all available session datasets.
+ 2. For each dataset (prioritised: current sessions first, then recent by size),
+ getDataset returns a base64 ZIP of every bill JSON in that session. Each
+ bill JSON has `bill_id`, `state` (abbrev), `description` (official synopsis)
+ and `title`.
+ 3. Match by external_id = legiscan-{state}-{bill_id}. Store `description`
+ (fall back to `title`), HTML-cleaned the same way bill_summaries.py does.
+ summary_source = 'LegiScan'.
+
+Idempotent: only updates rows WHERE summary IS NULL. Self-terminating: stops once
+every summary-missing bill has been matched (the "needset" empties), or when the
+dataset cap / time budget is hit.
+
+DB safety: parsing/network happen OUTSIDE any write transaction; updates commit in
+small batches (BATCH rows) with retry on "database is locked" (relies on the 60s
+busy_timeout). A concurrent federal job also writes the bills table, so we never
+hold a long write lock and never purge/rebuild anything.
+"""
+
+import base64
+import html
+import io
+import logging
+import os
+import re
+import sqlite3
+import sys
+import time
+import zipfile
+from collections import defaultdict
+
+import requests
+
+logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(message)s",
+ handlers=[
+ logging.StreamHandler(sys.stdout),
+ logging.FileHandler("/opt/docket/services/ingest/legiscan_summaries.log"),
+ ],
+)
+log = logging.getLogger("legiscan_bulk")
+
+BASE_URL = "https://api.legiscan.com/"
+SUMMARY_SOURCE = "LegiScan"
+BATCH = 3000
+LOCK_RETRIES = 8
+
+# Tunables (env-overridable)
+DATASET_CAP = int(os.environ.get("LS_DATASET_CAP", "997"))
+TIME_BUDGET_SEC = int(os.environ.get("LS_TIME_BUDGET", str(60 * 60))) # 60 min
+RATE_SLEEP = float(os.environ.get("LS_RATE_SLEEP", "0.25"))
+
+DB_PATH = os.environ.get("DOCKET_DB_PATH", "/opt/docket/data/docket.db")
+
+# ---- HTML cleaning (mirrors bill_summaries._strip_html) --------------------
+_HTML_TAG_RE = re.compile(r"<[^>]+>")
+_BLOCK_TAG_RE = re.compile(r"</?(p|div|br|li|ul|ol|h[1-6]|tr|table)\b[^>]*>", re.I)
+_WS_RUN_RE = re.compile(r"[ \t]+")
+_BLANKS_RE = re.compile(r"\n{3,}")
+
+
+def strip_html(raw: str) -> str:
+ if not raw:
+ return ""
+ unescaped = html.unescape(raw)
+ text = _BLOCK_TAG_RE.sub("\n", unescaped)
+ text = _HTML_TAG_RE.sub(" ", text)
+ text = _WS_RUN_RE.sub(" ", text)
+ text = "\n".join(line.strip() for line in text.split("\n"))
+ text = _BLANKS_RE.sub("\n\n", text)
+ return text.strip()
+
+
+def _load_key() -> str:
+ k = os.environ.get("DOCKET_LEGISCAN_KEY")
+ if k:
+ return k
+ for line in open("/opt/docket/.env"):
+ if line.startswith("DOCKET_LEGISCAN_KEY="):
+ return line.strip().split("=", 1)[1]
+ raise SystemExit("DOCKET_LEGISCAN_KEY not found")
+
+
+def _api(key, op, **params):
+ q = {"key": key, "op": op}
+ q.update(params)
+ last = None
+ for attempt in range(3):
+ try:
+ time.sleep(RATE_SLEEP)
+ r = requests.get(BASE_URL, params=q, timeout=180)
+ r.raise_for_status()
+ d = r.json()
+ if d.get("status") != "OK":
+ last = "status=%s" % d.get("status")
+ # LegiScan quota/auth errors surface here
+ log.warning("op=%s returned %s", op, last)
+ if str(d.get("alert", "")):
+ log.warning("alert: %s", d.get("alert"))
+ return None
+ return d
+ except Exception as exc: # noqa
+ last = exc
+ log.warning("op=%s attempt %d failed: %s", op, attempt + 1, exc)
+ time.sleep(2 + attempt * 3)
+ log.error("op=%s giving up: %s", op, last)
+ return None
+
+
+def load_needset(db_path):
+ """Return {bill_id_str: external_id} for legiscan bills missing a summary."""
+ conn = sqlite3.connect(db_path)
+ conn.execute("PRAGMA busy_timeout=60000")
+ rows = conn.execute(
+ "SELECT external_id FROM bills "
+ "WHERE source='legiscan' AND (summary IS NULL OR summary='')"
+ ).fetchall()
+ conn.close()
+ need = {}
+ for (ext,) in rows:
+ # external_id = legiscan-{STATE}-{bill_id}
+ bid = ext.rsplit("-", 1)[-1]
+ need[bid] = ext
+ return need
+
+
+def flush_batch(db_path, batch):
+ """Apply a batch of (summary, external_id) updates with lock-retry."""
+ if not batch:
+ return 0
+ for attempt in range(LOCK_RETRIES):
+ try:
+ conn = sqlite3.connect(db_path)
+ conn.execute("PRAGMA busy_timeout=60000")
+ conn.executemany(
+ "UPDATE bills SET summary=?, summary_source=? "
+ "WHERE external_id=? AND source='legiscan' "
+ "AND (summary IS NULL OR summary='')",
+ [(s, SUMMARY_SOURCE, ext) for (s, ext) in batch],
+ )
+ conn.commit()
+ conn.close()
+ return len(batch)
+ except sqlite3.OperationalError as exc:
+ try:
+ conn.close()
+ except Exception:
+ pass
+ if "locked" in str(exc).lower():
+ wait = 2 + attempt * 3
+ log.warning("DB locked, retry %d after %ds", attempt + 1, wait)
+ time.sleep(wait)
+ continue
+ raise
+ log.error("batch failed after %d lock retries; dropping batch of %d",
+ LOCK_RETRIES, len(batch))
+ return 0
+
+
+def main():
+ key = _load_key()
+ log.info("loading needset (legiscan bills missing summary)...")
+ need = load_needset(DB_PATH)
+ total_need = len(need)
+ log.info("needset: %d state bills missing a summary", total_need)
+ if total_need == 0:
+ log.info("nothing to do")
+ return
+
+ dl = _api(key, "getDatasetList")
+ if not dl:
+ log.error("getDatasetList failed (quota/key?). Aborting.")
+ return
+ datasets = dl["datasetlist"]
+ log.info("datasetlist: %d datasets", len(datasets))
+
+ # Priority: current sessions (prior=0) first, then newest year, then largest.
+ datasets.sort(key=lambda x: (x.get("prior", 1), -x.get("year_start", 0),
+ -x.get("dataset_size", 0)))
+
+ start = time.time()
+ updated = 0
+ per_state = defaultdict(int)
+ batch = []
+ datasets_done = 0
+
+ for ds in datasets:
+ if not need:
+ log.info("needset empty - all summaries matched. stopping.")
+ break
+ if datasets_done >= DATASET_CAP:
+ log.info("dataset cap %d reached. stopping.", DATASET_CAP)
+ break
+ if time.time() - start > TIME_BUDGET_SEC:
+ log.info("time budget reached. stopping.")
+ break
+
+ sid = ds["session_id"]
+ d = _api(key, "getDataset", id=sid, access_key=ds["access_key"])
+ datasets_done += 1
+ if not d:
+ continue
+ try:
+ zb = base64.b64decode(d["dataset"]["zip"])
+ zf = zipfile.ZipFile(io.BytesIO(zb))
+ except Exception as exc: # noqa
+ log.warning("session %s: bad zip: %s", sid, exc)
+ continue
+
+ matched_here = 0
+ for name in zf.namelist():
+ if "/bill/" not in name or not name.endswith(".json"):
+ continue
+ try:
+ import json
+ obj = json.loads(zf.read(name))
+ except Exception:
+ continue
+ b = obj.get("bill", obj)
+ bid = b.get("bill_id")
+ if bid is None:
+ continue
+ bid = str(bid)
+ ext = need.get(bid)
+ if not ext:
+ continue
+ text = strip_html(b.get("description") or "") or strip_html(b.get("title") or "")
+ if not text:
+ # nothing usable; drop from needset so we don't re-fetch forever
+ need.pop(bid, None)
+ continue
+ batch.append((text, ext))
+ st = b.get("state") or ext.split("-")[1]
+ per_state[st] += 1
+ need.pop(bid, None)
+ matched_here += 1
+ if len(batch) >= BATCH:
+ updated += flush_batch(DB_PATH, batch)
+ batch = []
+
+ log.info("session %s (%s): matched %d | total updated %d | remaining need %d | datasets %d",
+ sid, ds.get("year_start"), matched_here, updated + len(batch),
+ len(need), datasets_done)
+
+ if batch:
+ updated += flush_batch(DB_PATH, batch)
+ batch = []
+
+ elapsed = int(time.time() - start)
+ log.info("DONE: updated=%d datasets_processed=%d remaining_need=%d elapsed=%ds",
+ updated, datasets_done, len(need), elapsed)
+ log.info("per-state updated: %s",
+ dict(sorted(per_state.items(), key=lambda kv: -kv[1])))
+
+
+if __name__ == "__main__":
+ main()
services/ingest/ingest/r2.py +61 -0
@@ -0,0 +1,61 @@
+"""R2 archive uploader (Cloudflare). Outbound-only from the lab; no S3 creds.
+
+Uses the CF REST object API with the global key (from env CF_ACCOUNT_ID/CF_EMAIL/
+CF_GLOBAL_KEY - source /shared/projects/.env.secrets before running). Bucket
+`docket-archive` is public at https://archive.usdocket.org. Uploads are the
+preservation layer: snapshot official .gov records so they survive when the
+government reorganizes/removes them.
+"""
+import os
+import urllib.request
+import urllib.error
+
+ACCT = os.environ.get("CF_ACCOUNT_ID")
+EMAIL = os.environ.get("CF_EMAIL")
+KEY = os.environ.get("CF_GLOBAL_KEY")
+BUCKET = "docket-archive"
+PUBLIC_BASE = "https://archive.usdocket.org"
+
+
+def enabled() -> bool:
+ return bool(ACCT and EMAIL and KEY)
+
+
+def public_url(key: str) -> str:
+ return f"{PUBLIC_BASE}/{key.lstrip('/')}"
+
+
+def put_object(key: str, data: bytes, content_type: str = "application/octet-stream") -> str | None:
+ """Upload bytes to R2 at `key`. Returns the public URL, or None on failure/
+ missing creds (callers skip gracefully). Idempotent (overwrites same key)."""
+ if not enabled():
+ return None
+ key = key.lstrip("/")
+ url = f"https://api.cloudflare.com/client/v4/accounts/{ACCT}/r2/buckets/{BUCKET}/objects/{key}"
+ req = urllib.request.Request(
+ url, data=data, method="PUT",
+ headers={"X-Auth-Email": EMAIL, "X-Auth-Key": KEY, "Content-Type": content_type},
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=90) as r:
+ if r.status == 200:
+ return public_url(key)
+ except urllib.error.HTTPError:
+ return None
+ except Exception:
+ return None
+ return None
+
+
+def exists(key: str) -> bool:
+ """HEAD-style check so we don't re-upload what's already archived."""
+ if not enabled():
+ return False
+ key = key.lstrip("/")
+ url = f"https://api.cloudflare.com/client/v4/accounts/{ACCT}/r2/buckets/{BUCKET}/objects/{key}"
+ req = urllib.request.Request(url, method="GET", headers={"X-Auth-Email": EMAIL, "X-Auth-Key": KEY})
+ try:
+ with urllib.request.urlopen(req, timeout=30) as r:
+ return r.status == 200
+ except Exception:
+ return False
services/ingest/ingest/run.py +135 -2
@@ -5,7 +5,19 @@ Usage
-----
python -m ingest.run --once # run the whole pipeline once, in order
python -m ingest.run --source members # run one module's CLI
- python -m ingest.run --list # list pipeline steps
+ python -m ingest.run --list # list pipeline steps + refresh profiles
+ python -m ingest.run --refresh # FREQUENT light incremental federal tier
+ python -m ingest.run --daily # DAILY heavier tier (bounded enrichment)
+ python -m ingest.run --weekly # WEEKLY heaviest tier (full diff + state)
+
+Refresh profiles
+----------------
+The full --once pipeline includes heavy full-rescan steps (diff --all, the state
+pollers, historical voteview) that must NOT run on a frequent cadence. The three
+profiles below are tiered for an automatic, lock-safe refresh cadence. Each runs
+its steps SEQUENTIALLY via subprocess (a single sequential writer), so they are
+WAL-safe against the site's reader connections. Scheduled overlap between tiers
+is prevented at the systemd layer by a shared flock (see deploy/).
Each step is a module with its own tested `__main__` CLI (or a poll() function).
The orchestrator runs them in dependency order, best-effort: a failing step is
@@ -68,6 +80,95 @@ PIPELINE = [
# Continuous change detection is a wrapper around this runner: ingest.watcher.
_BY_LABEL = {label: (mod, args) for label, mod, args in PIPELINE}
+
+# ---------------------------------------------------------------------------
+# Tiered refresh profiles
+# ---------------------------------------------------------------------------
+# A profile is an ordered list of steps in canonical dependency order (a subset
+# of PIPELINE). An entry is either a bare label (use PIPELINE's default args) or
+# a (label, [override args]) tuple (replace the default args for this tier only).
+#
+# FREQUENT (--refresh): light incremental federal "what's new". Every step here
+# is watermark/`seen`/`poller_watermarks`-driven (incremental) or archives only
+# new rows (archive_votes). No full rescans, no enrichment backfills, no state.
+REFRESH_STEPS = [
+ "federal_register",
+ "openomb",
+ "congress",
+ "govinfo",
+ "bills_bulk",
+ "votes_senate",
+ "votes_house",
+ "votes_enrich",
+ "link_votes",
+ "archive_votes",
+ "crec",
+ "meetings",
+ "categorize",
+ "calendar",
+]
+
+# DAILY (--daily): everything in --refresh PLUS the roster, fiscal sources,
+# lobbying, bounded enrichment backfills, and state coverage. Still excludes the
+# heavy full-rescan diff chain and the full state pollers (those are --weekly).
+# bill_summaries/archive_bills are bounded with --limit so they never run away.
+DAILY_STEPS = [
+ "members",
+ "member_photos",
+ "usaspending",
+ "treasury",
+ "federal_register",
+ "openomb",
+ "congress",
+ "govinfo",
+ "bills_bulk",
+ "votes_senate",
+ "votes_house",
+ "votes_enrich",
+ "link_votes",
+ "archive_votes",
+ "crec",
+ "meetings",
+ "lobbying",
+ "state_coverage",
+ "categorize",
+ "calendar",
+ ("bill_summaries", ["--limit", "500"]),
+ ("archive_bills", ["--limit", "300"]),
+]
+
+# WEEKLY (--weekly): the heaviest steps. The bill-text diff engine (ingest.diff)
+# ONLY supports a full scan (`--all --congress 119`) or a single `--bill-id`; it
+# has no recent/changed-only mode, so the diff + rider_detect + amendments chain
+# lands here rather than in --daily. bill_versions runs first to refresh the
+# version chains the diff consumes. openstates/legiscan are the full state
+# pollers; state_coverage recomputes coverage after the state pull.
+WEEKLY_STEPS = [
+ "bill_versions",
+ ("diff", ["--all", "--congress", "119"]),
+ "rider_detect",
+ "amendments",
+ "openstates",
+ "legiscan",
+ "state_coverage",
+]
+
+_PROFILES = {
+ "refresh": REFRESH_STEPS,
+ "daily": DAILY_STEPS,
+ "weekly": WEEKLY_STEPS,
+}
+
+
+def _resolve(step):
+ """Resolve a profile entry to (label, module, extra_args)."""
+ if isinstance(step, tuple):
+ label, override = step
+ else:
+ label, override = step, None
+ mod, default_extra = _BY_LABEL[label]
+ extra = override if override is not None else default_extra
+ return label, mod, extra
# short aliases
_ALIASES = {"usa": "usaspending", "house": "votes_house", "senate": "votes_senate",
"fr": "federal_register", "omb": "openomb", "bills": "bills_bulk",
@@ -97,16 +198,48 @@ def _run_all() -> None:
logger.info("=== Ingest complete: %d/%d steps clean ===", ok, len(PIPELINE))
+def _run_profile(name: str, steps: list) -> None:
+ """Run a tiered profile's steps sequentially, best-effort (same as _run_all)."""
+ logger.info("=== Docket %s ingest starting (%d steps) ===", name, len(steps))
+ ok = 0
+ for step in steps:
+ label, mod, extra = _resolve(step)
+ if _run_step(label, mod, extra):
+ ok += 1
+ logger.info("=== %s ingest complete: %d/%d steps clean ===", name, ok, len(steps))
+
+
def main():
parser = argparse.ArgumentParser(description="Docket full-spectrum ingest runner")
parser.add_argument("--once", action="store_true", help="Run the whole pipeline once.")
parser.add_argument("--source", metavar="STEP", help="Run a single step by label.")
- parser.add_argument("--list", action="store_true", help="List pipeline steps.")
+ parser.add_argument("--list", action="store_true", help="List pipeline steps + profiles.")
+ parser.add_argument("--refresh", action="store_true",
+ help="FREQUENT tier: light incremental federal 'what's new'.")
+ parser.add_argument("--daily", action="store_true",
+ help="DAILY tier: refresh + roster/fiscal/lobbying + bounded enrichment + state coverage.")
+ parser.add_argument("--weekly", action="store_true",
+ help="WEEKLY tier: full diff/rider/amendments + state pollers + coverage.")
args = parser.parse_args()
if args.list:
+ print("Full pipeline (--once):")
for label, mod, extra in PIPELINE:
print(f" {label:16} {mod} {' '.join(extra)}")
+ for pname, steps in _PROFILES.items():
+ print(f"\nProfile --{pname} ({len(steps)} steps):")
+ for step in steps:
+ label, mod, extra = _resolve(step)
+ print(f" {label:16} {mod} {' '.join(extra)}")
+ return
+ if args.refresh:
+ _run_profile("refresh", REFRESH_STEPS)
+ return
+ if args.daily:
+ _run_profile("daily", DAILY_STEPS)
+ return
+ if args.weekly:
+ _run_profile("weekly", WEEKLY_STEPS)
return
if args.source:
label = _ALIASES.get(args.source.lower(), args.source.lower())