| 1 | # Zion PM v2 - Ops Runbook |
| 2 | |
| 3 | Host: **CT120 (zion-pm, REDACTED-IP)** on the pve node. App at `/opt/zion-pm`, served on `:4100`, fronted at `https://REDACTED-HOST` (Traefik CT244 + Authentik CT246). |
| 4 | |
| 5 | ## Service control (CT120) |
| 6 | |
| 7 | ```bash |
| 8 | systemctl status zion-pm |
| 9 | systemctl restart zion-pm |
| 10 | journalctl -u zion-pm -f # live logs |
| 11 | ss -tlnp | grep 4100 # confirm listening |
| 12 | ``` |
| 13 | |
| 14 | ## Deploy a change |
| 15 | |
| 16 | Edit on CT215 (canonical repo `/shared/projects/zion-pm`), commit + push to GitHub. CT120 has **no GitHub egress**, so CT215 syncs the source over SSH (CT215-mediated deploy): |
| 17 | |
| 18 | ```bash |
| 19 | # on CT215 |
| 20 | cd /shared/projects/zion-pm && git add -A && git commit -m "..." && git push |
| 21 | tar czf - app lib public package.json pnpm-lock.yaml tsconfig.json next.config.ts postcss.config.mjs \ |
| 22 | | ssh root@REDACTED-IP 'tar xzf - -C /opt/zion-pm' |
| 23 | ssh root@REDACTED-IP 'cd /opt/zion-pm && pnpm install && pnpm build && systemctl restart zion-pm' |
| 24 | ``` |
| 25 | |
| 26 | Never edit/run from the share (`/shared/projects` is noexec CIFS). Canonical repo on share, app runs from `/opt`. (If CT120 GitHub egress is opened later, it can `git clone`/`pull` directly instead.) |
| 27 | |
| 28 | ## Health checks |
| 29 | |
| 30 | ```bash |
| 31 | curl -s localhost:4100/api/channels # 4 seeded channels |
| 32 | curl -s localhost:4100/api/fleet # worker availability |
| 33 | curl -s -X POST localhost:4100/api/fleet/ping -H 'content-type: application/json' -d '{"model":"glm"}' |
| 34 | curl -sk -D - -o /dev/null https://REDACTED-HOST/ # expect 302 -> auth.lab |
| 35 | ``` |
| 36 | |
| 37 | ## Fleet dependencies |
| 38 | |
| 39 | | Worker | Endpoint | Auth | Notes | |
| 40 | |---|---|---|---| |
| 41 | | glm (brain) | `https://api.z.ai/api/anthropic/v1` | `ZAI_API_KEY` | subscription, weekly cap (not PAYG) | |
| 42 | | claude | SSH `root@REDACTED-IP` | CT120 key | `IS_SANDBOX=1` + `~/.local/bin` PATH | |
| 43 | | codex | SSH `zion@REDACTED-IP` | CT120 key | `codex exec`, cwd `/tmp` (CIFS stalls sandbox) | |
| 44 | | ollama | `http://REDACTED-IP:11434` | none | 3060; CT205 was offline at build | |
| 45 | | queryMemory | MCP SSE `:9876` (CT215) | none | homelab-rag | |
| 46 | | queryGraph | MCP SSE `:9877` (CT215) | none | cbm-homelab | |
| 47 | |
| 48 | ## Common issues |
| 49 | |
| 50 | - **Worker `exit 127`**: binary not on non-login SSH PATH. Adapters prepend `~/.local/bin:/usr/local/bin`. |
| 51 | - **claude `401`**: zion's claude login is invalid; the worker uses `root@CT215` (valid). Re-auth with `claude` as the target user if it breaks. |
| 52 | - **claude refuses `--dangerously-skip-permissions` as root**: set `IS_SANDBOX=1` (the adapter does). |
| 53 | - **codex timeout**: codex exec is slow (~30s+ cold). Default timeout 240s; keep cwd small/local. |
| 54 | - **ollama unreachable**: if the 3060 sleeps, wake via CT244 control plane `curl http://REDACTED-IP:8765/wake/3060`. If CT205/pve-gpu returns, point `ZPM_OLLAMA_URL` back at `http://REDACTED-IP:11434`. |
| 55 | - **pm.lab 404 from Authentik**: the `zion-pm` application/provider is missing on the embedded outpost - recreate it (see SEED build log) and the outpost reloads. |
| 56 | - **Native module "bindings not found"**: `cd /opt/zion-pm/node_modules/.pnpm/better-sqlite3@*/node_modules/better-sqlite3 && npm run build-release`. |
| 57 | |
| 58 | ## Recovery |
| 59 | |
| 60 | - **CT120 rebuild**: the parked perseus CT120 was backed up to PBS as `ct/120/2026-06-21T00:30:53Z` before the rebuild. The old KeePassXC vault is rescued at `/root/ct120-vault-rescue/homelab.kdbx` on CT215. `pct restore 120 <backup>` restores the old perseus. |
| 61 | |
| 62 | ## Edge / SSO config |
| 63 | |
| 64 | - Traefik route: `CT244:/opt/edge/traefik/dynamic/pm.yml` → `http://REDACTED-IP:4100`, middleware `sso-auth@file`. |
| 65 | - Authentik: proxy provider "Zion PM forwardAuth" + app `zion-pm` on the embedded outpost (CT246). |
| 66 | - DNS: `REDACTED-HOST` resolves to CT244 via the existing `*.lab` wildcard. |