Zion Boggan
repos/Oversight/docs/ROADMAP.md
zionboggan.com ↗
358 lines · markdown
History for this file →
1
# Oversight Roadmap
2
 
3
Last revised 2026-06-16. The launch plan is gated on product usability and
4
threat-model honesty, not on a calendar date.
5
 
6
## Where we are
7
 
8
1. **L3 safety fixes and collusion documentation** shipped in v0.4.5. L3
9
   defaults off for wording-sensitive document classes, requires explicit
10
   disclosure when enabled, records `canonical_content_hash`, and supports a
11
   boilerplate-only mode for contracts and filings.
12
2. **SIEM integration ahead of SOC 2** shipped in v0.4.6. `oversight_core.siem`
13
   and the `oversight siem export` CLI emit beacon events as Splunk HEC,
14
   Elastic Common Schema 8.x, or Microsoft Sentinel Log Analytics records.
15
   Operator guide at `docs/SIEM.md`.
16
3. **Registry federation hardening** shipped in v0.4.7. The v1 interop spec
17
   at `docs/spec/registry-v1.md` is aligned against the reference server:
18
   canonical-JSON algorithm, uniform error envelope, normative endpoint and
19
   beacon paths, `/evidence` bundle shape, and `/tlog/head|proof|range` are
20
   pinned. `tests/test_registry_conformance.py` runs 38 checks in-process
21
   or against a live URL. An operator claims v1 compatibility with
22
   `OVERSIGHT_REGISTRY_URL=https://registry.example.org python3 tests/test_registry_conformance.py`.
23
4. **Browser inspector and classic-suite decrypt** shipped on
24
   `oversightprotocol.dev/viewer/`. Drag-drop `.sealed`
25
   parsing, WebCrypto Ed25519 signature verification, canonical JSON
26
   byte-identical to Python, optional registry lookups, and full
27
   decryption of classic-suite sealed files using WebCrypto X25519 + HKDF-SHA256
28
   with a vendored `@noble/ciphers` XChaCha20-Poly1305. Post-decrypt
29
   SHA-256 matches `content_hash` or the flow aborts. Hybrid
30
   (post-quantum) in-browser decrypt **shipped 2026-05-03** using a
31
   vendored ML-KEM-768 from `@noble/post-quantum` for the post-quantum
32
   half of the KEM, with X-wing-style HKDF binding over both shared
33
   secrets. The viewer now decrypts both `OSGT-CLASSIC-v1` and
34
   `OSGT-HYBRID-v1` sealed files.
35
5. **Outlook add-in first** for the first ecosystem integration. Drive,
36
   Box, SharePoint, and Teams plugins are deferred until a maintainer or
37
   design partner funds them.
38
6. **SOC 2 Type 1 scoping** becomes realistic after a design-partner
39
   engagement. ISO 27001 follows SOC 2. FedRAMP is dropped from near-term
40
   planning; it is a multi-year commercial program requiring sponsor-agency
41
   backing.
42
 
43
## Public launch sequence
44
 
45
1. L3 safety and collusion documentation. **Shipped in v0.4.5.**
46
2. Browser inspector and drag-drop share workflow. **Shipped** -
47
   inspector, classic-suite decrypt, and hybrid (post-quantum) decrypt
48
   are all live.
49
3. Outlook add-in. **Scaffold landed 2026-05-07** (`integrations/outlook/`,
50
   `docs/OUTLOOK.md`); hosted pilot page landed 2026-05-26, Outlook tenant
51
   load-test pending.
52
4. One regulated-industry design-partner deployment.
53
5. SOC 2 Type 1 scoping in parallel with the design partner.
54
6. Broad public launch (HN, Reddit, conferences). Not before the inspector,
55
   the Outlook add-in, and a real deployment are all in hand.
56
 
57
---
58
 
59
## Shipped
60
 
61
### RFC 3161 qualified timestamps - v0.3
62
 
63
`oversight_core/timestamp.py` and `registry/server.py` perform real RFC 3161
64
requests. The default TSA chain tries FreeTSA first, falls back to DigiCert,
65
and falls back to the registry's own clock if both are unreachable. Verified
66
live: 4667-byte signed TimeStampToken, valid P-384 signature, correct
67
gen_time, correct nonce. FreeTSA is free and research-grade; GlobalSign or
68
GLOBALTRUST drop in for deployments that require eIDAS-qualified status.
69
 
70
### Rust canonical port of the hot path - v0.4
71
 
72
The seal, open, manifest, policy, semantic, tlog, and rekor path is
73
implemented as a Rust workspace under `oversight-rust/`. `cargo build
74
--workspace --release` passes with zero warnings. Python is the reference
75
implementation; Rust is canonical for production deployments. A conformance
76
suite proves bit-identical output for every manifest and envelope. Format
77
adapter parity is being closed in bounded slices; current `main` has parsed
78
PDF page/content-stream text extraction for fingerprinting and DCT mid-band
79
image watermarking in the Rust adapter.
80
 
81
### Fail-closed security hardening - v0.4.4
82
 
83
Codex review pass across the policy engine, Rekor verification, registry
84
input validation, and the container format. Nine findings, nine fixes.
85
Failed decryption no longer consumes open counts. REGISTRY and HYBRID
86
policy modes fail closed instead of silent local fallback. DNS event
87
callbacks from non-loopback clients must carry `OVERSIGHT_DNS_EVENT_SECRET`.
88
Multi-recipient sealing disabled until the manifest format can honestly
89
bind every recipient.
90
 
91
### L3 safety and GUI starter - v0.4.5
92
 
93
`oversight_core/l3_policy.py` gates L3 on document class: legal,
94
regulatory, technical specifications, source code, SQL, logs, and
95
structured data default to off. Explicit `full`, `boilerplate`, and `off`
96
modes are supported. The CLI refuses to seal with L3 enabled unless the
97
caller acknowledges that the recipient copy is textually non-identical to
98
the canonical source. The manifest records `canonical_content_hash` and
99
`l3_policy` so disputes can reach ground truth. `oversight gui` launches
100
a Tkinter desktop starter covering keygen, seal, and open.
101
 
102
### Sigstore Rekor v2 transparency log - v0.5
103
 
104
Seal registrations are attested to the public Sigstore Rekor v2 log as
105
DSSE envelopes. The predicate type resolves to a git-tagged GitHub path
106
so third-party verifiers can resolve the schema without a live
107
`oversight.dev` endpoint. Recipient X25519 public keys are SHA-256 hashed
108
before going on-log. Opt-in by default via `OVERSIGHT_REKOR_ENABLED=1`;
109
failures are non-fatal and the local SQLite tlog stays authoritative.
110
Offline bundles carry the log public key, checkpoint, entry schema, and
111
optional RFC 3161 chain. Self-hosted Rekor v2 is supported via
112
`OVERSIGHT_REKOR_URL`.
113
 
114
### SIEM export - v0.4.6
115
 
116
`oversight_core/siem.py` ships a normalized `OversightEvent` model that
117
maps the registry `events` table onto three schema-stable formatters:
118
Splunk HEC envelopes, Elastic Common Schema 8.x documents, and Microsoft
119
Sentinel Log Analytics custom-log rows. `sentinel_authorization()`
120
implements the Data Collector API HMAC signing recipe. `FileSink`,
121
`StdoutSink`, and `HTTPJSONSink` cover the transport surface without
122
pulling SIEM credentials into the Oversight process by default.
123
 
124
The `oversight siem export` CLI streams events as JSON lines to stdout,
125
a file, or a generic HTTPS collector. Supports `--since`, `--limit`,
126
repeatable `--header`, and Splunk source/sourcetype/index overrides.
127
Opens the registry database read-only so it is safe to run against a
128
live service. Full operator guide at `docs/SIEM.md`. 11 unit tests cover
129
envelope shape, None-field suppression, SQLite row mapping, read-only
130
iteration, Sentinel HMAC determinism, and action-name coverage.
131
 
132
### Registry federation hardening - v0.4.7
133
 
134
`docs/spec/registry-v1.md` rewrites the interop contract against what
135
the reference server actually serves. The spec now pins:
136
 
137
- Canonicalization algorithm: sort keys, compact separators, UTF-8,
138
  matching `json.dumps(manifest, sort_keys=True, separators=(",", ":"))`
139
- Uniform error envelope with a defined `code` vocabulary
140
- Full endpoint table including normative beacon paths
141
  (`/p/{token_id}.png`, `/r/{token_id}`, `/v/{token_id}`)
142
- `/.well-known/oversight-registry` identity shape
143
- `/evidence/{file_id}` bundle fields
144
- `/tlog/head|proof|range` for federated verifiers
145
 
146
`tests/test_registry_conformance.py` is a 38-check harness with two
147
modes. In-process against a FastAPI TestClient for CI, or against a
148
live URL when `OVERSIGHT_REGISTRY_URL` is set. An independent operator
149
who passes the harness claims v1 compatibility.
150
 
151
CORS middleware on the live registry lets the browser inspector read
152
`/health`, `/.well-known/oversight-registry`, and `/evidence/{file_id}`
153
from the public site origin. Methods are restricted to GET and OPTIONS;
154
registration, DNS events, and attribution stay browser-unreachable.
155
 
156
### Browser inspector and classic-suite decrypt - post-v0.4.7
157
 
158
`site/viewer/` is a static page that parses the `.sealed` container
159
in the browser, verifies the issuer Ed25519 signature via WebCrypto,
160
and renders the manifest, watermarks, beacons, and policy. Canonical
161
JSON in JS is byte-identical to the Python reference, validated
162
against a Python-generated test vector.
163
 
164
Classic-suite decryption runs locally. WebCrypto performs X25519 ECDH
165
via JWK import and derives the key-encryption key with HKDF-SHA256;
166
a vendored pinned copy of `@noble/ciphers` (sha256 `b31ecc4f`) provides
167
the XChaCha20-Poly1305 primitive that WebCrypto does not expose. After
168
decrypt, the plaintext SHA-256 is compared against `manifest.content_hash`
169
and a mismatch aborts. Wrong private key, mismatched recipient, tampered
170
ciphertext, and hybrid-suite inputs all fail with explicit messages.
171
 
172
A tutorial `.sealed` and its recipient `identity.json` are published
173
under `viewer/samples/` with an in-file note that the keypair is a
174
public demo key.
175
 
176
### Operational hygiene - ongoing
177
 
178
History rewrite removed every occurrence of RFC 1918 addresses,
179
internal workspace paths, and container identifiers that had leaked
180
into early commits. Two internal-only files (`SESSION_RESUME.md` and
181
`docs/RUNBOOK.md`) were removed from every commit and every tag.
182
 
183
`scripts/opsec-scan.sh` scans either the whole tree or the staged
184
diff for the same patterns plus GitHub PATs, OpenAI and Slack tokens,
185
and raw private-key PEMs. `.github/workflows/opsec.yml` runs the
186
scanner on every pull request and push to main. `scripts/githooks/pre-commit`
187
is the optional local hook. `.gitignore` blocks session, handoff,
188
runbook, `private/`, `secrets/`, and `*.local.md` filename patterns
189
so notes of that flavor cannot be accidentally committed in the first
190
place.
191
 
192
---
193
 
194
## Next
195
 
196
### Outlook add-in
197
 
198
**Scaffold landed 2026-05-07.** `integrations/outlook/` ships the Office
199
add-in 1.1 manifest (`MailApp`, read-mode task pane, `ReadItem` only),
200
the task-pane HTML, and JS that imports the public viewer's parse /
201
verify / decrypt directly from `oversightprotocol.dev/viewer/`. No
202
second crypto stack. Both classic and hybrid suites decrypt. Decision
203
record at `docs/OUTLOOK.md`.
204
 
205
As of 2026-05-26, the hosted pilot page and manifest URL are live under
206
`oversightprotocol.dev/integrations/outlook/`. Remaining for a real pilot:
207
an Outlook tenant load-test against classic and hybrid sealed attachments,
208
plus a final icon design pass before AppSource review. Sealing-from-Outlook
209
(compose mode) is intentionally deferred to v2.
210
 
211
### Hardware `KeyProvider` in Rust
212
 
213
`docs/HARDWARE_KEYS.md` already documents the vendor-neutral setup
214
covering YubiKey 5C, OnlyKey, and Nitrokey 3. **Trait + `FileKeyProvider`
215
landed 2026-05-07** in `oversight-crypto`: `KeyProvider` abstracts the
216
recipient-side ECDH so a hardware backend can plug in without changing
217
call sites; `unwrap_dek_with_provider` is the new entry point and is
218
byte-identical to `unwrap_dek` for file-backed keys.
219
 
220
**`OSGT-HW-P256-v1` suite implementation landed 2026-05-07.** P-256 ECDH
221
wrap/unwrap, `WrappedDekP256` envelope, and `SoftwareP256KeyProvider`
222
(in-memory P-256 reference impl) are in `oversight-crypto`. Cross-suite
223
envelopes are rejected explicitly. 21/21 tests in the crate pass.
224
 
225
**v0.4.11 closed the software reference path across Rust, Python, and the
226
browser inspector.** `OSGT-HW-P256-v1` now has manifest/container plumbing,
227
Python wrap/unwrap parity, and a public viewer sample fixture. The remaining
228
hardware work is the `PivKeyProvider` (PKCS#11 against a YubiKey / Nitrokey /
229
OnlyKey PIV slot), a different `KeyProvider` implementation that calls into
230
`cryptoki` instead of holding the scalar in process. The registry records
231
whether each recipient pubkey is file-backed or hardware-backed so issuers can
232
require hardware backing for sensitive material.
233
 
234
### Registry in Rust
235
 
236
`oversight-rust/oversight-registry` is scaffolded with all endpoints
237
implemented under `#![forbid(unsafe_code)]`. As of 2026-05-14, the Axum
238
server passes the existing 38-check `tests/test_registry_conformance.py`
239
harness in live-URL mode against the registry v1 surface with
240
`OVERSIGHT_OPERATOR_TOKEN` enabled. The Rust registry now matches the Python
241
reference for write-side operator-token auth and DNS bridge bearer/header
242
auth. As of 2026-05-17, `oversight-registry --migrate-from` can copy the
243
Python registry's manifests, beacons, watermarks, events, and corpus rows
244
into the Rust SQLite schema, with `--migrate-dry-run` for count-only
245
preflight. As of 2026-05-20, `--validate-db` checks the copied Rust database
246
for orphan rows, identity mismatches, malformed manifest JSON, invalid
247
manifest signatures, and manifest/file ID divergence. As of 2026-05-21, that
248
validation also covers event/corpus JSON sidecars and tlog index uniqueness.
249
As of 2026-05-22, registry writes fail closed when tlog append fails and
250
`--validate-db` compares event tlog indexes against the on-disk tlog size.
251
As of 2026-05-24, validation also checks that each event's indexed tlog leaf
252
matches the event row rather than unrelated evidence.
253
As of 2026-05-25, local tlog recovery rejects malformed leaf records,
254
non-contiguous indexes, and leaf-hash mismatches instead of silently ignoring
255
corrupted lines.
256
As of 2026-05-28, `/tlog/range` reads through the validated tlog record API
257
instead of parsing `leaves.jsonl` directly, so monitor responses fail closed
258
when an on-disk leaf is malformed or hash-mismatched.
259
The Python reference registry now mirrors that fail-closed tlog recovery and
260
range behavior, with `leaf_data_hex` on newly appended local tlog records.
261
Both registry implementations now return the registry v1 `{error: {code,
262
message}}` envelope for representative client and server errors, and the
263
conformance harness checks those envelopes.
264
As of 2026-05-31, `docs/REGISTRY_V1_STABILITY.md` records the registry v1.0
265
candidate wire-format freeze for operator burn-in. Remaining work:
266
longer-running deployment tests and a final v1.0 release tag against that
267
candidate profile.
268
 
269
---
270
 
271
## Mid-term
272
 
273
### Spec publication
274
 
275
- **GitHub** - live at `github.com/oversight-protocol/oversight` under
276
  Apache 2.0 with test vectors.
277
- **arXiv preprint** (~15 pages, `cs.CR`): motivation, threat model,
278
  protocol, cryptographic construction, security arguments, implementation,
279
  evaluation, limitations, related work.
280
- **IETF Internet-Draft** as `draft-oversight-00`. Submit to
281
  datatracker, present at an informal BoF of SUIT, OHAI, LAKE, or CFRG
282
  depending on framing. Iterate 6 to 12 months before an RFC stage.
283
  Multiple independent implementations are required before RFC.
284
 
285
### Conference targets
286
 
287
- USENIX Security Cycle 2 (June 2026).
288
- Black Hat Europe 2026 (December 2026).
289
- ACSAC 2026.
290
- Black Hat USA 2027.
291
 
292
Demonstration material: live seal and open in Python and Rust, live leak
293
simulation with real-time attribution, hybrid PQ sizing, air-gap strip
294
via VM and retype, hardware-key pull mid-open.
295
 
296
---
297
 
298
## 2027
299
 
300
- Independent security audit. Trail of Bits, NCC Group, Cure53, and
301
  Zellic are the plausible candidates. Typical engagement: 4 to 8
302
  engineer-weeks, $75K to $200K, 60-day disclosure window. Prerequisites:
303
  spec freeze, threat model document, fuzz campaign, internal review
304
  pass.
305
- v1.0 release, spec freeze, RFC shepherding.
306
- Black Hat USA 2027 Briefings.
307
- ISO 27001 after SOC 2 Type 2.
308
 
309
---
310
 
311
## Explicitly dropped or deferred
312
 
313
- **FedRAMP.** Multi-year, seven-figure, requires commercial entity and
314
  sponsor agency. Revisit only if a commercial pivot occurs.
315
- **Cloud-TEE key custody.** Ties Oversight to a single cloud vendor
316
  and contradicts the open-source goal. Hardware keys replace it.
317
- **Drive, Box, SharePoint, Teams plugins.** Deferred until a maintainer
318
  or design partner funds them.
319
- **Broad HN and Reddit launch.** Gated on Outlook add-in plus one
320
  design-partner deployment.
321
 
322
---
323
 
324
## Phased status
325
 
326
| Phase | Items | Status |
327
|---|---|---|
328
| 0 | GitHub org, Apache 2.0, SECURITY.md | Shipped |
329
| 1 | FreeTSA + DigiCert RFC 3161 chain | Shipped (v0.3) |
330
| 2 | Rust canonical port, conformance suite | Shipped (v0.4) |
331
| 3 | Fail-closed security hardening | Shipped (v0.4.4) |
332
| 4 | L3 safety, GUI starter, canonical_content_hash | Shipped (v0.4.5) |
333
| 5 | Rekor v2 integration, cross-language parity | Shipped (v0.5) |
334
| 6 | SIEM export: Splunk, Sentinel, ECS | Shipped (v0.4.6) |
335
| 7 | Registry v1 spec + conformance harness + CORS | Shipped (v0.4.7) |
336
| 8 | Browser inspector, classic-suite decrypt, opsec scanner + CI | Shipped |
337
| 9 | Hybrid PQ decrypt in browser | Shipped (2026-05-03) |
338
| 10 | Outlook add-in | Hosted pilot page live; tenant load-test next |
339
| 11 | Hardware KeyProvider in Rust | Suite shipped (v0.4.11); PIV provider next |
340
| 12 | Rust Axum registry, migration tooling | Migration validation shipped; deployment burn-in next |
341
| 13 | arXiv preprint, threat-model repo document | Mid-term |
342
| 14 | IETF Internet-Draft, CFRG or equivalent BoF | Mid-term |
343
| 15 | USENIX Security Cycle 2, Black Hat EU 2026 | Mid-term |
344
| 16 | Independent security audit | 2027 |
345
| 17 | v1.0 release, RFC shepherding, Black Hat USA 2027 | 2027 |
346
 
347
## Cost outlook
348
 
349
| Item | Cost |
350
|---|---|
351
| FreeTSA, DigiCert, Sigstore Rekor, GitHub Actions | $0 |
352
| Hardware keys for development and testing | ~$100 |
353
| Domain, DNS, public beacon hosting (annual) | ~$60 |
354
| Conference registration and travel | ~$6K |
355
| Independent security audit (2027) | $75K to $200K |
356
 
357
All year-one work runs on free and open infrastructure. Paid dependencies
358
are optional.