| 1 | # Darwin - Ingestion Collectors (gap build) |
| 2 | |
| 3 | Reuse homelab-rag's change-only manifest engine verbatim. State manifest |
| 4 | `/var/tmp/homelab-rag/state.sqlite` (local ext4). Registry |
| 5 | `/shared/projects/homelab-rag/config/sources.yaml`. Driver `ops/ingest-all.sh`. Each collector below |
| 6 | is a NEW source entry + a small change-only shipper using the same delta logic |
| 7 | (mtime/size + tombstones, key = `source_id + normalized_path`). NO new index engine. |
| 8 | |
| 9 | Order: COVERED = reuse, BUILD = net-new (below). Confirmed gaps from reuse-map recon. |
| 10 | |
| 11 | ## BUILD-1 - White PC files (re-enable, narrow root) |
| 12 | - Source `desktop-qi823gr` (100.76.195.52) is disabled; only `C:\homelab` was scoped (absent). |
| 13 | - Re-enable with narrow roots: `C:\Users\zion\Documents`, `C:\Users\zion\Desktop`, |
| 14 | `C:\Users\zion\Downloads` (skip the slow full tree). Same `ssh-powershell -EncodedCommand` collector. |
| 15 | - Tailscale path; expect slower walks - keep roots tight, `timeout 150` per run. |
| 16 | |
| 17 | ## BUILD-2 - Browser history (White, 3060, Linux CTs) |
| 18 | The history DB **is** the index - ship it, don't re-crawl pages. |
| 19 | - Chrome/Edge: `%LOCALAPPDATA%\<Browser>\User Data\Default\History` (SQLite; tables `urls`, `visits`). |
| 20 | - Firefox: `%APPDATA%\Mozilla\Firefox\Profiles\*\places.sqlite`. |
| 21 | - Linux CTs: `~/.config/google-chrome/Default/History`, `~/.mozilla/firefox/*/places.sqlite`. |
| 22 | - The live DB is **locked while the browser runs** → collector copies it to a temp path first, then |
| 23 | reads rows with `last_visit_time` newer than the manifest's last watermark; ships the delta as |
| 24 | normalized records `{url, title, visit_time, source}`. |
| 25 | |
| 26 | ## BUILD-3 - PowerShell history (White, 3060) |
| 27 | - Single file `%APPDATA%\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt`. |
| 28 | - Append-only → ship by byte-offset / line-count delta stored in the manifest. Trivial. |
| 29 | |
| 30 | ## BUILD-4 - Linux shell history (per CT) |
| 31 | - `~/.bash_history` / `~/.zsh_history`. History flushes on shell exit, so set |
| 32 | `PROMPT_COMMAND='history -a'` (login shells) to flush live; then the existing Linux SSH collector |
| 33 | sees the changed mtime each run. Add an explicit per-CT history shipper (line-count delta) rather |
| 34 | than relying on incidental `/home/zion` capture. |
| 35 | |
| 36 | ## BUILD-5 - Clipboard history (the only resident-daemon gap) |
| 37 | - No native store. Needs a small always-on watcher: |
| 38 | - Windows: a clipboard watcher appending timestamped entries to a JSONL. |
| 39 | - Linux: `greenclip`/`clipnotify` → JSONL. |
| 40 | - Then ship that JSONL append-only via the same delta mechanism. This is the only collector that |
| 41 | needs a resident agent, not just a timer shipper. |
| 42 | |
| 43 | ## PRIVACY GATE (hard) |
| 44 | BUILD-1/2/3/5 on the **personal White desktop** capture browsing + clipboard + keystrokes-as-saved. |
| 45 | That is privacy-sensitive and outward-affecting. **Staged, not auto-deployed.** Deploy only on Zion's |
| 46 | explicit per-source approval. Homelab/Proxmox + the 3060 *service* box proceed freely; the personal |
| 47 | White box waits for the word. Provide a per-source on/off so Zion green-lights exactly what he wants. |
| 48 | |
| 49 | ## Acceptance |
| 50 | - Each new source appears in `sources.yaml`, ships only changed rows (proven by a run record showing |
| 51 | `changed << seen`), tombstones deletions, and never re-indexes `/shared/projects` (dedup contract). |
| 52 | - Darwin's Cortex `inbox`/`history` adapters can query the resulting chunks via homelab-rag :9876. |