● Zion Boggan
repos/Darwin/docs/DARWIN_BLUEPRINT.md
zionboggan.com ↗
281 lines · markdown
History for this file →
1
# DARWIN - Blueprint & North Star
2
 
3
**A living, evolving second brain for Zion's entire digital life.**
4
Status: Phase 0 (design + scaffold). Author: Claude (Opus), overseer build. Last updated: 2026-06-24.
5
 
6
> Darwin is not a fifth memory tool. It is the **face and the unifier** of the four that already
7
> exist (homelab-rag, cbm-homelab, Serena, memory-graph), plus a capture inbox, plus a recursive
8
> agent command deck - wrapped in an Awwwards-tier interface that is *alive*.
9
 
10
---
11
 
12
## 0. Why the past attempts failed (the thing we are correcting)
13
 
14
Every prior attempt built a **retrieval engine** and called it a brain. A retrieval engine is a
15
hippocampus. A *second brain* is a hippocampus **+ a front door + a capture habit + a freshness
16
discipline.** Zion has four excellent engines and never built the other three. Darwin builds the
17
connective tissue and the storefront. **This is an integration + front-door project, not a new
18
research project.** ~80% already exists; we wire it together and give it a face.
19
 
20
The named failure mode: **a static brain rots.** Stale facts look identical to true ones. (Proof:
21
this very session, the agent confidently reported Hermes drawdowns and DropHawk status as live -
22
both are dead/parked - because it read its own un-decayed memory.) Darwin's core thesis fixes this:
23
**knowledge under natural selection.**
24
 
25
---
26
 
27
## 1. The thesis - Darwin = evolution
28
 
29
The brain **evolves**, like its namesake:
30
 
31
- **Reinforcement** - facts/sources you reach for rank higher and link tighter.
32
- **Decay** - what you never touch sinks; not deleted, just demoted, so the surface stays signal.
33
- **Selection pressure** - every query, correction, and click is a fitness signal. Darwin gets
34
  fitter the more it's used, and it adapts ranking *to Zion specifically* over time.
35
- **Self-healing** - like a business, files/state change constantly; Darwin detects the delta and
36
  updates only what changed. It **never flushes the whole index.**
37
 
38
---
39
 
40
## 2. Architecture - three layers + one inbox
41
 
42
```
43
                    ┌─────────────────────────────────────────────┐
44
                    │  THE FACE  (Awwwards organism UI)            │
45
                    │  ask box · answer+receipts · agent nervous   │
46
                    │  system · inbox · health-as-anatomy          │
47
                    └───────────────┬─────────────────────────────┘
48
                                    │ one verb: ask()
49
                    ┌───────────────▼─────────────────────────────┐
50
                    │  THE CORTEX  (unified gateway + synthesizer) │
51
                    │  parallel fan-out · source-of-truth ranking  │
52
                    │  recency/confidence · clickable receipts     │
53
                    │  ALSO the single MCP every agent calls       │
54
                    └──┬──────┬───────┬────────┬──────────┬────────┘
55
                       │      │       │        │          │
56
              homelab-rag   cbm    Serena  memory-graph  INBOX
57
              (what/where) (code) (symbols) (decisions)  (captures)
58
                       │
59
        ┌──────────────┴───────────────────────────────────┐
60
        │  SELF-HEALING INGESTION (change-only rolling)     │
61
        │  reuse homelab-rag collectors; extend to White,   │
62
        │  3060, browser/shell/clipboard history            │
63
        └───────────────────────────────────────────────────┘
64
 
65
                    ┌─────────────────────────────────────────────┐
66
                    │  THE DECK  (recursive agent command center)  │
67
                    │  one event bus · live force-directed tree    │
68
                    │  you→agents→their agents→theirs · attach SSH │
69
                    └─────────────────────────────────────────────┘
70
```
71
 
72
### 2.1 The Cortex (the keystone - never built before)
73
A single service exposing one verb: **`ask`**. It:
74
1. Fans out **in parallel** to all engines, each answering what it's best at:
75
   - **homelab-rag** → content / what / where / why (765k chunks, FTS + Qdrant hybrid)
76
   - **cbm-homelab** → code architecture, call graph, blast radius
77
   - **Serena** → live symbol truth
78
   - **memory-graph** → decisions, ownership, handoffs
79
   - **Inbox** → Zion's raw captures
80
2. **Reconciles** conflicts via the source-of-truth hierarchy (§4).
81
3. **Synthesizes** one answer - Opus for hard questions, local qwen3 (3060) for cheap ones.
82
4. Attaches **clickable receipts** - every claim shows `{source, path, last_seen, confidence}`.
83
   No claim ships without provenance. This is how we kill "didn't trust the answers."
84
5. Is **also exposed as the unified MCP** (`darwin.ask`) so Zion and the agents share one brain.
85
   Four tools collapse into one verb.
86
 
87
### 2.2 The Inbox (capture / accounts-receivable)
88
Frictionless capture, zero filing decisions. Reuse what exists: **Memos / notes.lab** + the
89
**cand_memos voice pipeline** land into the Cortex within seconds. Capture-to-searchable is the
90
A/R department; today it exists but isn't wired to the brain.
91
 
92
### 2.3 The Deck (recursive agent command center)
93
One **event bus**. Every agent (Claude, GLM, Codex, schedulers, loops) emits structured events
94
(`spawned`, `tool_call`, `child_spawned`, `returned`, `error`) to a single stream. The Deck renders
95
a **live force-directed tree**: you → your agents → their agents → theirs, arbitrarily deep, each
96
node showing model, status, token spend, current tool. Click a node → attach to its terminal (the
97
Odysseus Cockpit already does in-SPA SSH panes; we extend that, not rebuild).
98
 
99
### 2.4 The Face - see §6.
100
 
101
---
102
 
103
## 3. Self-healing ingestion (the silo that heals itself)
104
 
105
**Reuse, don't rebuild.** homelab-rag already does change-only rolling ingest across many endpoints
106
(detects changed files, updates only those, never flushes; state DB on local ext4 at
107
`/var/tmp/homelab-rag/state.sqlite`; dedup contract: p-drive is the only source for
108
`/shared/projects`). Darwin **extends collectors**, it does not replace the engine.
109
 
110
**Contract (every source obeys):**
111
- content-hash per item → only changed items re-embed
112
- deletions **tombstoned**, never orphaned
113
- every item carries `{source, path, mtime, last_seen, confidence}`
114
- one source per endpoint; no overlapping roots (preserve the dedup contract)
115
 
116
**New territory to grab** (gaps confirmed by reuse-map recon - see §9):
117
- **White** desktop: all files + **browser history** (Chrome/Edge `History` SQLite),
118
  **PowerShell PSReadLine** history, app data, chat logs
119
- **3060** box: files + endpoint data
120
- **clipboard history**, **shell history** across CTs
121
- Going-forward continuous capture agent (clipboard + browser + shell + app activity)
122
 
123
**Honest boundary (precision matters):** we can index everything ever *saved*. Keystrokes typed and
124
never persisted anywhere are unrecoverable; nothing can retroactively recover those. A forward
125
capture agent tightens the net from here on. No keylogger time-machine promises.
126
 
127
**PRIVACY GATE:** deploying a continuous capture agent onto Zion's *personal* White desktop
128
(browser history, keystrokes) is privacy-sensitive and outward-affecting. It is **staged + flagged
129
for Zion's explicit approval**, never silently deployed - even under autonomy. Homelab/Proxmox
130
indexing proceeds freely.
131
 
132
---
133
 
134
## 4. Source of truth (Zion's ruling, recency-gated)
135
 
136
When sources disagree, authority order:
137
 
138
```
139
live infra state  >  recent files  >  agent memory  >  older notes
140
```
141
 
142
**Guardrail:** agent memory rides above older notes *only while fresh*. Every fact carries
143
`last_seen` + confidence; once stale it **decays below** the notes it would normally outrank. This
144
is the exact valve that turns "trust agent memory" from the liability that lied about Hermes back
145
into an asset. Staleness is structural, not vibes.
146
 
147
---
148
 
149
## 5. Proactive model (comes to you, doesn't nag)
150
 
151
- **Reactive by default** - answers when asked.
152
- **24-hour dead-man's check-in** - if Zion hasn't talked to Darwin in 24h, it reaches out with a
153
  tight brief (what changed, what's stale, what's stuck waiting on him). Surfaced as *"Darwin
154
  stirred -"*.
155
- **No push-push-push.** Beyond the daily heartbeat, it stays silent unless asked.
156
 
157
---
158
 
159
## 6. The Face - design (locked direction: surprise accepted)
160
 
161
**Darwin IS a living organism, and the organism is made of Zion's data.** Not a Greek bust (the refs
162
use dead statues because they have nothing alive to show; Zion does). Every cell = a real data
163
source (Proxmox, White, 3060, browser history, each agent).
164
 
165
- **On load:** it doesn't fade in, it **evolves in** - a primordial speck self-assembles
166
  (cell → cluster → organism) in ~1.2s on easeOutExpo.
167
- **On scroll:** it specializes; the agent section lights up the **nervous system = the recursive
168
  agent tree**, synapses firing as agents actually work.
169
- **On focus (ask-field):** the organism collapses inward and folds into the cursor - "the brain
170
  becomes a question." Ask → synapses fire down the exact paths queried (RAG→cbm→Serena→graph) →
171
  the answer assembles from glowing trails, each with a clickable receipt.
172
- **Health = anatomy (the instrument, not decoration):** fresh/healthy data → vibrant,
173
  bioluminescent, moving. A stale source → that limb **greys and withers** on screen. Hermes would
174
  have been a visibly dead limb you'd *see* before anyone spoke it. Self-healing = visible
175
  regeneration.
176
- **Darwinian touch:** cursor lingering → organism grows toward you (attention = selection pressure,
177
  visible reinforcement). Neglected past the 24h window → it hibernates and dims.
178
 
179
**Aesthetic:** near-black canvas; brutalist monolithic type holding the corners (Monolith's
180
discipline); one **bioluminescent accent** (deep-sea glow, Odysseus teal). Tension between cold
181
brutalist geometry and a warm living form *is* the signature. One surface, both modes (Zion's
182
choice): left = living agent tree, center = ask → answer + receipts, right = raw hits + inbox.
183
 
184
### 6.1 Motion system (from confirmed reference teardown)
185
Premium feel = **one signature move + a disciplined easing dialect**, not WebGL everywhere (only 1
186
of 5 refs used real 3D, and pre-baked at 30fps). Stack: **React + GSAP (ScrollTrigger/Flip/SplitText)
187
+ Lenis + React-Three-Fiber**.
188
 
189
Two global easing tokens drive everything:
190
```
191
--ease-out-expo:   cubic-bezier(0.16, 1, 0.3, 1)   // entrances, reveals  (GSAP "expo.out")
192
--ease-inout-expo: cubic-bezier(0.87, 0, 0.13, 1)  // symmetric flips     (GSAP "power4.inOut")
193
```
194
Buildable primitives (durations/triggers in `_ops/motion-system.md`): `lenis-scroll-base`,
195
`fade-rise-stagger` (opacity0→1,y24→0, 0.6s, stagger .09), `mask-reveal-up` (SplitText lines,
196
yPercent110→0, 0.9s), `clip-wipe-reveal`, the **signature** `evolving-organism` (R3F, scroll-scrubbed,
197
pre-baked path available so the 3060 doesn't sweat), `magnetic-hover`, `invert-cursor`,
198
`gradient-aurora` (bake static per thirdweb #703), `logo-marquee`. 80% of polish = Lenis base +
199
fade-rise + mask-reveal + the one signature + magnetic hover, all in the easeOutExpo dialect with
200
low-amplitude restraint.
201
 
202
---
203
 
204
## 7. Hosting & cutover (HARD GATES)
205
 
206
1. **Build the new Darwin CT fresh** - clean, isolated, fully working. Nothing touches Odysseus.
207
2. **Checkpoint + archive the ENTIRE Odysseus (CT245)** - full PBS backup, verified restorable,
208
   **non-destructive**. Safe anytime.
209
3. **STOP.** Do **NOT** tear down CT245 or reallocate its resources until Zion explicitly says go -
210
   that teardown kills the session he types in (cockpit window + phone Claude code-tab remote both
211
   ride the cockpit transport). Hard gate. His word only.
212
4. **Cutover choreography:** Darwin's CT must be a fully working front door - *including
213
   remote-control transport* - **before** Odysseus dies. New door opens → Zion walks through →
214
   *then* the go-word → reallocation. Zero blackout.
215
 
216
---
217
 
218
## 8. Phase plan + delegation matrix (overseer; right-size the model)
219
 
220
| Phase | Deliverable | Driver | Risk |
221
|---|---|---|---|
222
| 0 | Blueprint + scaffold + recon | Opus (me) | none |
223
| 1 | New Darwin CT provisioned | me + SSH/PVE | low (reversible) |
224
| 1b | Odysseus checkpointed (non-destructive) | me | none |
225
| 2 | Cortex gateway + unified `ask` MCP | Opus design, **Sonnet/Codex** impl | med |
226
| 3 | Inbox wire-in (Memos + voice) | **Sonnet** | low |
227
| 4 | Deck event bus + recursive tree | **Sonnet/GLM** | med |
228
| 5 | The Face (organism UI) | **GLM-5.2 + Codex**, Opus vision-gate | med |
229
| 6 | Ingestion extension (White/3060/history) | **Haiku** collectors, Opus privacy-gate | med (privacy) |
230
| 7 | Cutover (gated on Zion's go) | me | HIGH - gated |
231
 
232
Model routing: Haiku = grunt indexing/collectors; Sonnet = services; Opus = architecture + hard
233
synthesis + quality gate; GLM-5.2 + Codex = front-end masterpiece + parallel tracks. Every delegated
234
agent carries the RAG-first + targeted-discovery rule.
235
 
236
---
237
 
238
## 9. Recon findings (folded in from grounding agents)
239
 
240
_(populated from infra-grounding + reuse-map agents - see §9.1, §9.2)_
241
 
242
### 9.1 Infra / capacity
243
_pending agent a7f3bd8 - VMID, node, capacity, Odysseus footprint, PBS checkpoint command._
244
 
245
### 9.2 Reuse map (CONFIRMED)
246
 
247
**Four memory stores Darwin's gateway fans out to (all CT215, all SSE):**
248
- `homelab-rag` - content RAG, FTS + Qdrant (`homelab_chunks`, qdrant `127.0.0.1:6333`), ~765k chunks - **SSE :9876**
249
- `cbm-homelab` - code graph (~1.9M nodes, 112 projects), store `/var/tmp/cbm-homelab/graph` - **SSE :9877**
250
- `Serena` - live symbol nav (one project at a time) - **SSE :9878**
251
- `memory-graph` - decisions/handoffs, SQLite `/var/tmp/memory-graph/graph.sqlite` - **SSE :9879**
252
 
253
**Change-only engine to REUSE verbatim (Darwin's self-heal primitive):**
254
- State manifest `/var/tmp/homelab-rag/state.sqlite` (local ext4 - NEVER on CIFS, returns `database is locked`)
255
- Sources registry `/shared/projects/homelab-rag/config/sources.yaml`; driver `ops/ingest-all.sh`; hourly `homelab-rag-ingest.timer`
256
- Delta logic: walk roots → compare mtime+size (sha256+size for cross-dup) → emit `new/changed/unchanged/tombstoned`; deletions tombstoned not orphaned; key = `source_id + normalized_path`
257
- Live proof: p-drive saw 31,127 files, processed only 28 changed + 8 tombstoned. **No flush.**
258
- Collectors: local walker (p-drive); Linux SSH python manifest walker; **Windows PowerShell-over-SSH `-EncodedCommand`** (base64 UTF-16LE)
259
- Dedup contract: p-drive is the ONLY source for `/shared/projects`; one source per endpoint
260
 
261
**Coverage - REUSE vs BUILD:**
262
| Target | Status |
263
|---|---|
264
| Fleet config/code/docs across ~16 CTs | ✅ COVERED (reuse) |
265
| 3060 box (`desktop-f8o1v2s`, .19) `C:\homelab` + Documents, 38k files | ✅ COVERED |
266
| White PC (`desktop-qi823gr`, 100.76.195.52) files | ❌ GAP - source **disabled**, only scoped `C:\homelab` (absent). Re-enable with narrow root `C:\Users\zion\{Documents,Desktop}` |
267
| Browser history (anywhere) | ❌ GAP - ship the History SQLite DBs directly (Chrome/Edge `…\User Data\Default\History`; Firefox `places.sqlite`; copy locked DB to temp, ship rows by `last_visit_time` delta) |
268
| PowerShell history | ❌ GAP - `…\PSReadLine\ConsoleHost_history.txt`, append-only byte-offset delta |
269
| Linux shell history per-CT | ❌ GAP - `~/.bash_history`; set `PROMPT_COMMAND='history -a'` to flush live, ship line delta |
270
| Clipboard history | ❌ GAP - only gap needing a **resident daemon** (Windows watcher / greenclip→JSONL), then append-only ship |
271
 
272
All gaps reuse the same manifest + delta + transports. No new index engine.
273
 
274
**Housekeeping noticed:** `ct243-media` stale since 06-19; `desktop-qi823gr` (White) disabled; `ct236`/`ct120` sources disabled (CTs stopped/offline).
275
 
276
---
277
 
278
## 10. Naming & identity
279
- **Darwin** - the brain that evolves. Health = anatomy. Decay = honesty.
280
- Lives at (proposed) `REDACTED-HOST` behind Authentik SSO.
281
- Pantheon-adjacent to Hermes/Perseus/Odysseus but its own metaphor: not a god, a *living thing*.