| 1 | # Docket - Automatic Data Refresh Cadence |
| 2 | |
| 3 | How Docket keeps `/opt/docket/data/docket.db` fresh day-to-day without overloading |
| 4 | the box or colliding with the live site's reads. Deployed on the deploy host. |
| 5 | |
| 6 | The previous approach (a single `docket-watcher` timer that ran the full pipeline, |
| 7 | including `diff --all`, every 30 minutes) was too heavy and is **removed**. It is |
| 8 | replaced by three tiered profiles in `ingest.run`, each on its own systemd timer, |
| 9 | all guarded by one shared `flock` so they can never overlap. |
| 10 | |
| 11 | ## The three tiers |
| 12 | |
| 13 | All three are profiles of `python -m ingest.run`. Each runs its steps |
| 14 | **sequentially** via subprocess - a single sequential writer, which coexists |
| 15 | safely with the site's WAL reader connections (`busy_timeout=60000`). |
| 16 | |
| 17 | ### `--refresh` (FREQUENT - twice daily, 06:00 & 18:00 America/Chicago) |
| 18 | Light incremental federal "what's new". Every step is watermark / `seen` / |
| 19 | `poller_watermarks`-driven, or only archives new rows. 14 steps, in order: |
| 20 | |
| 21 | ``` |
| 22 | federal_register, openomb, congress, govinfo, bills_bulk, |
| 23 | votes_senate, votes_house, votes_enrich, link_votes, archive_votes, |
| 24 | crec, meetings, categorize, calendar |
| 25 | ``` |
| 26 | |
| 27 | ### `--daily` (DAILY - 04:30 America/Chicago) |
| 28 | Everything in `--refresh` PLUS the roster, fiscal sources, lobbying, bounded |
| 29 | enrichment backfills, and state coverage. 22 steps, in order: |
| 30 | |
| 31 | ``` |
| 32 | members, member_photos, usaspending, treasury, |
| 33 | federal_register, openomb, congress, govinfo, bills_bulk, |
| 34 | votes_senate, votes_house, votes_enrich, link_votes, archive_votes, |
| 35 | crec, meetings, lobbying, state_coverage, categorize, calendar, |
| 36 | bill_summaries --limit 500, archive_bills --limit 300 |
| 37 | ``` |
| 38 | |
| 39 | Enrichment is **bounded** every run: `bill_summaries --limit 500` and |
| 40 | `archive_bills --limit 300` so they can never run away. |
| 41 | |
| 42 | ### `--weekly` (WEEKLY - Sundays 03:00 America/Chicago) |
| 43 | The heaviest steps. 7 steps, in order: |
| 44 | |
| 45 | ``` |
| 46 | bill_versions, diff --all --congress 119, rider_detect, amendments, |
| 47 | openstates, legiscan, state_coverage |
| 48 | ``` |
| 49 | |
| 50 | ## What is excluded from the frequent/daily tiers, and why |
| 51 | |
| 52 | - **`diff --all` + `rider_detect` + `amendments` → WEEKLY only.** The bill-text |
| 53 | diff engine (`ingest/diff.py`) has **no recent/changed-only mode** - its CLI |
| 54 | only supports a full scan (`--all --congress 119`) or one bill (`--bill-id`). |
| 55 | Because there is no incremental diff, the whole diff/rider/amendments chain |
| 56 | would be a full rescan on every run, so it lands in the weekly tier rather than |
| 57 | daily. `bill_versions` runs first in the weekly tier to refresh the version |
| 58 | chains the diff consumes. |
| 59 | - **`openstates` / `legiscan` (full state pollers) → WEEKLY only.** Heavy full |
| 60 | scans of state legislation; once a week is plenty. |
| 61 | - **`voteview` (historical roll-call backfill) → never scheduled.** One-time / |
| 62 | manual only: `python -m ingest.pollers.voteview --once --start 101 --end 119`. |
| 63 | |
| 64 | ## Lock-safety design |
| 65 | |
| 66 | 1. **Single sequential writer.** `run.py` runs every step in series via |
| 67 | `subprocess.run`. There is never more than one writer inside a single profile |
| 68 | run. The site reads via WAL with a 60 s `busy_timeout`, so a single sequential |
| 69 | writer never produces "database is locked". |
| 70 | 2. **Shared `flock` across all three tiers.** Each service's `ExecStart` is: |
| 71 | ``` |
| 72 | /bin/flock --nonblock --conflict-exit-code 0 /run/docket-ingest.lock <venv-python> -m ingest.run --<tier> |
| 73 | ``` |
| 74 | `--nonblock` + `--conflict-exit-code 0` means: if another tier already holds |
| 75 | `/run/docket-ingest.lock`, this tick **skips cleanly (exit 0)** instead of |
| 76 | starting a second concurrent writer. The next scheduled tick catches up |
| 77 | (watermarks mean nothing is lost). This guarantees the tiers can never run two |
| 78 | writers at once, regardless of how long a run takes. |
| 79 | 3. **Staggered schedules** (03:00 weekly Sun, 04:30 daily, 06:00/18:00 refresh) |
| 80 | so they don't even attempt to start together; the flock is the hard backstop. |
| 81 | 4. **`Persistent=true`** on every timer catches a missed fire across reboots. |
| 82 | |
| 83 | ## systemd units (on the deploy host, `/etc/systemd/system/`) |
| 84 | |
| 85 | | Service | Timer (America/Chicago) | Profile | TimeoutStartSec | |
| 86 | |---|---|---|---| |
| 87 | | `docket-refresh.service` | `*-*-* 06,18:00` | `--refresh` | 3600 | |
| 88 | | `docket-daily.service` | `*-*-* 04:30` | `--daily` | 10800 | |
| 89 | | `docket-weekly.service` | `Sun *-*-* 03:00`| `--weekly` | 21600 | |
| 90 | |
| 91 | All run as root, `WorkingDirectory=/opt/docket/services/ingest`, |
| 92 | `Environment=DOCKET_DB_PATH=/opt/docket/data/docket.db`, |
| 93 | `EnvironmentFile=-/opt/docket/.env`, venv python at |
| 94 | `/opt/docket/services/ingest/.venv/bin/python`. |
| 95 | |
| 96 | Source-of-truth copies live in `services/ingest/deploy/`. |
| 97 | |
| 98 | ## Running manually |
| 99 | |
| 100 | ```bash |
| 101 | cd /opt/docket/services/ingest |
| 102 | export DOCKET_DB_PATH=/opt/docket/data/docket.db |
| 103 | |
| 104 | ./.venv/bin/python -m ingest.run --list # show pipeline + all 3 profiles |
| 105 | ./.venv/bin/python -m ingest.run --refresh # run the frequent tier now |
| 106 | ./.venv/bin/python -m ingest.run --daily # run the daily tier now |
| 107 | ./.venv/bin/python -m ingest.run --weekly # run the weekly tier now |
| 108 | ./.venv/bin/python -m ingest.run --source congress # run one step |
| 109 | |
| 110 | # To run manually WITHOUT risking overlap with a scheduled tier, take the lock: |
| 111 | /bin/flock --nonblock /run/docket-ingest.lock \ |
| 112 | ./.venv/bin/python -m ingest.run --daily |
| 113 | |
| 114 | # Trigger a scheduled service immediately (also respects the flock): |
| 115 | systemctl start docket-refresh.service |
| 116 | ``` |
| 117 | |
| 118 | ## Pausing / resuming |
| 119 | |
| 120 | ```bash |
| 121 | # Pause a tier (stops future fires; a running job finishes) |
| 122 | systemctl disable --now docket-refresh.timer |
| 123 | |
| 124 | # Pause everything |
| 125 | systemctl disable --now docket-refresh.timer docket-daily.timer docket-weekly.timer |
| 126 | |
| 127 | # Resume |
| 128 | systemctl enable --now docket-refresh.timer docket-daily.timer docket-weekly.timer |
| 129 | |
| 130 | # See next fire times |
| 131 | systemctl list-timers 'docket-*' --all |
| 132 | |
| 133 | # Tail a run |
| 134 | journalctl -u docket-daily.service -f |
| 135 | ``` |
| 136 | |
| 137 | ## ISR / frontend caching (note) |
| 138 | |
| 139 | The Next.js pages currently use `export const revalidate = 300` (5 min) on data |
| 140 | pages and `3600` on the home page - already well under any freshness concern and |
| 141 | inside the "≤900 for list/register/votes/bills" guideline. No ISR change was made |
| 142 | with this work: changing `revalidate` requires a production rebuild to take |
| 143 | effect, and a rebuild was deliberately avoided while the federal/state backfills |
| 144 | were running (to protect the live site). If origin load ever needs trimming, the |
| 145 | safe slow-changing candidates to relax to 1800-3600 are `/states`, `/categories`, |
| 146 | `/who-pays`, `/how-we-pick`; keep `/votes`, `/bills`, `/register`, `/laws` at |
| 147 | ≤900 so daily data surfaces fast. Rebuild + QA off-peak if changed. |