Zion Boggan
repos/Oversight/CHANGELOG.md
zionboggan.com ↗
662 lines · markdown
History for this file →
1
# Oversight CHANGELOG
2
 
3
## v0.4.12 - 2026-06-17 Security hardening, cross-language parity, JCS canonicalization, and the post-quantum KEM mirror
4
 
5
- **Minimum supported Rust version raised to 1.85.** The post-quantum KEM
6
  mirror depends on RustCrypto's `ml-kem` crate, which requires Rust 1.85
7
  (edition 2024). The workspace `rust-version` is bumped from 1.75 to 1.85
8
  to declare this accurately; the workspace itself stays on edition 2021.
9
  Consumers building from source need a 1.85+ toolchain.
10
 
11
- **Post-quantum KEM mirror (OSGT-HYBRID-v1), Phase A.** The Rust crypto
12
  crate now implements the ML-KEM-768 half of the hybrid DEK wrap,
13
  byte-identical to the Python reference. New functions
14
  `oversight_crypto::hybrid_wrap_dek`, `hybrid_unwrap_dek`, and
15
  `mlkem768_generate_keypair` reproduce the Python construction exactly:
16
  IKM is `ss_x || ss_pq || x25519_ephemeral_pub || mlkem_ciphertext`,
17
  HKDF-SHA256 with a zero salt and `info=b"oversight-hybrid-v1-dek-wrap"`,
18
  then XChaCha20-Poly1305 with `aad=b"oversight-hybrid-dek"`. The envelope
19
  JSON shape (`suite`, `x25519_ephemeral_pub`, `mlkem_ciphertext`, `nonce`,
20
  `wrapped_dek`, all hex) matches `oversight_core.crypto.hybrid_wrap_dek`,
21
  so a Python-wrapped envelope opens in Rust and vice versa. The primitive
22
  is RustCrypto's pure-Rust `ml-kem` crate (FIPS 203, no C build, chosen
23
  over liboqs because the build environment lacks cmake and the pure-Rust
24
  crate satisfies the same NIST-primitives-via-RustCrypto/liboqs rule). Rust keeps its
25
  decapsulation keys in the recommended 64-byte seed form; only the
26
  1184-byte public key ever crosses the language boundary, so no deprecated
27
  expanded-key import is needed. Six Rust unit tests mirror
28
  `tests/test_pq.py` (round trip, JSON shape, tamper-classical,
29
  tamper-PQ, wrong recipient, overhead bound), and a new cross-language
30
  conformance vector (`tests/conformance_hybrid_kem.py`) proves both
31
  directions (PY wrap to RS unwrap, RS wrap to PY unwrap) against a live
32
  liboqs, wired into `conformance_cross_lang.sh` as step 5. Scope: KEM
33
  only. The container HYBRID seal/open dispatch and ML-DSA manifest
34
  signing are deferred (Phase B and Phase C respectively); per the owner's
35
  decision, Phase C will use dual Ed25519 AND ML-DSA-65 signatures.
36
 
37
- **Suppress liboqs-python import-time stdout.** `liboqs-python` attaches a
38
  `StreamHandler(sys.stdout)` at INFO and logs a line when imported, which
39
  contaminated stdout for any caller importing `oversight_core.crypto` and
40
  broke byte-identity conformance capture. `oversight_core/crypto.py` now
41
  redirects stdout during the `import oqs` so the message is dropped.
42
  Behavior of the PQ primitives is unchanged.
43
 
44
- **Rust `open_sealed` now enforces the `jurisdiction` policy.** The
45
  `oversight-policy::check_policy` function already implemented
46
  jurisdiction matching (and had passing unit tests), but the container
47
  open path never called it: `open_sealed` and `open_sealed_with_provider`
48
  inlined only the `not_after` / `not_before` time checks and skipped
49
  jurisdiction entirely, so the policy crate's enforcement was dead code
50
  on the real open path. Python enforces at `container.py:226`
51
  (`check_policy` before decryption); Rust now mirrors that single
52
  chokepoint. Both Rust open functions now call
53
  `oversight_policy::check_policy(&manifest, policy_ctx)` after issuer
54
  trust and before DEK unwrap, which also retires the duplicated inline
55
  time checks onto the tested canonical path. Observable change: time
56
  violations now surface as `ContainerError::Policy` (typed, matching
57
  Python's `PolicyViolation`) rather than `ContainerError::Precondition`.
58
  Four new container-level tests pin the behavior (mismatch rejected,
59
  match allowed, GLOBAL allowed, no-context skip), and the existing
60
  `expired_file_rejected` assertion is updated to the typed error.
61
  Backward compatibility: GLOBAL manifests (the default) and opens with
62
  no `PolicyContext` (the raw-open path used by the cross-language
63
  conformance harness) are unaffected. Scope honesty: this closes the
64
  open-time gap the review flagged. It does NOT implement the
65
  registry-time MUST in SPEC.md §8.4 (`policy.jurisdiction` mismatch
66
  causing `/register` rejection), which is unimplemented in BOTH
67
  languages and tracked separately. A `--jurisdiction` CLI flag for an
68
  opener asserting a non-default region is a deferred follow-up, not
69
  required for the gap closure (the CLI's default `GLOBAL` context
70
  already rejects a manifest requiring a specific jurisdiction).
71
 
72
- **Constant-time token comparison via `subtle::ConstantTimeEq`.** The
73
  Oversight Rust registry's operator-token and DNS-secret comparison was
74
  a hand-rolled loop in `oversight-registry/src/auth.rs` with an early
75
  return on length mismatch, which trivially leaked whether an attacker's
76
  guessed length was correct in O(1). Replaced with
77
  `subtle::ConstantTimeEq` (audited RustCrypto crate, already a transitive
78
  dependency via `ed25519-dalek` and `chacha20poly1305`, now declared
79
  explicitly). Four new registry unit tests pin the correctness matrix
80
  (equal inputs across lengths, same-length different content,
81
  mismatched-length no-early-return, single-bit difference). The
82
  hand-rolled early-return leak is closed; the residual
83
  max(supplied, expected) timing observation from `subtle`'s slice impl
84
  is documented in the function-level rationale and is acceptable for
85
  Oversight's high-entropy operator tokens.
86
 
87
- **Canonicalization unified on RFC 8785 JCS.** The Python reference now
88
  canonicalizes via a vendored `oversight_core.jcs.jcs_dumps` that is
89
  byte-exact with the Rust reference's `serde_jcs::to_vec`. Previously
90
  Python used `json.dumps(sort_keys=True, separators=(",",":"))` with the
91
  default `ensure_ascii=True`, which diverged from Rust for any non-ASCII
92
  string value (Python escaped non-ASCII as `\uXXXX`; Rust emitted raw
93
  UTF-8). The divergence was latent because every committed test fixture
94
  and the conformance harness used ASCII-only content, but any production
95
  manifest with a non-ASCII recipient_id, issuer_id, or filename would
96
  have signed to different bytes across the two implementations and
97
  failed cross-language verification. The port covers manifest signing,
98
  manifest wire form, transparency-log leaf payloads, transparency-log
99
  signed heads, sealed-container wrapped_dek, DSSE statement payloads,
100
  DSSE envelope serialization, registry manifest verification, registry
101
  sidecar comparison, and registry evidence-bundle signing. Standalone
102
  tooling (sample generators, live demo, canary keeper) uses the
103
  `sort_keys=True, ensure_ascii=False` form, which is byte-identical to
104
  JCS for the no-floats subset these tools emit. `conformance_cross_lang.sh`
105
  gains a non-ASCII `recipient_id` round trip that exercises the
106
  divergence end-to-end and would fail under any non-JCS serialization.
107
  Backward compatibility: for ASCII-only content (every committed fixture
108
  and every existing test vector), the new JCS bytes are identical to the
109
  old sort_keys bytes, so existing signatures continue to verify.
110
 
111
### Earlier batches in v0.4.12 (pre-JCS, kept for review provenance)
112
 
113
- **Live registry deployment config.** `docker-compose.yml` now has a `live`
114
  Caddy profile with public TLS routing for the registry, beacon, OCSP-style,
115
  and license-style hostnames. `Caddyfile` covers the full registry v1
116
  read/evidence/tlog surface plus beacon routes, with all hostnames coming
117
  from environment variables. `.env.example` documents public-safe defaults
118
  and leaves secrets blank.
119
- **Registry operator token.** The Python reference registry can now require
120
  `OVERSIGHT_OPERATOR_TOKEN` for `POST /register` and `POST /attribute`.
121
  The token is optional so local development and unauthenticated conformance
122
  runs keep working, but production operators can protect write-side APIs
123
  without changing route shapes. The conformance harness sends the token as
124
  a bearer header when `OVERSIGHT_OPERATOR_TOKEN` is set.
125
- **Rust registry operator-token parity.** The Axum + SQLx registry now reads
126
  `OVERSIGHT_OPERATOR_TOKEN` too and enforces it on `POST /register` and
127
  `POST /attribute` with the same bearer/header contract as the Python
128
  registry. Its DNS event route also accepts either `Authorization: Bearer`
129
  or `X-Oversight-DNS-Secret`, matching the live deployment guide.
130
- **Rust registry migration tooling.** `oversight-registry` now supports
131
  `--migrate-from <python-registry.sqlite>` plus `--migrate-dry-run`. It
132
  copies the Python reference registry's manifests, beacons, watermarks,
133
  events, and corpus rows into the Rust SQLite schema while preserving event
134
  IDs, corpus metadata, and registry evidence relationships.
135
- **Rust registry integrity validation.** `oversight-registry --validate-db`
136
  now checks migrated Rust registry databases for orphaned attribution rows,
137
  identity mismatches, malformed manifest JSON, invalid manifest signatures,
138
  and manifest/file ID divergence before operators declare migration burn-in
139
  complete. It also validates event/corpus JSON sidecars and tlog index
140
  uniqueness so corrupted migrated evidence cannot look clean. Rust registry
141
  writes now fail closed if the local transparency log cannot append, and
142
  validation checks missing or out-of-range event tlog indexes against the
143
  on-disk tlog size. Validation also compares event rows to the corresponding
144
  tlog leaf payload so an index cannot point at unrelated evidence and still
145
  pass burn-in checks. Local tlog recovery now rejects malformed records,
146
  non-contiguous indexes, and leaf-hash mismatches instead of silently
147
  ignoring corrupted lines during startup or validation. `/tlog/range` now
148
  reads through the same validated tlog API, so malformed or hash-mismatched
149
  records fail the range request instead of being silently omitted from
150
  monitor responses. The Python reference tlog now matches that behavior:
151
  startup and `/tlog/range` fail closed on corrupt leaf records, and new
152
  leaves carry `leaf_data_hex` so exact leaf bytes survive recovery.
153
- **Registry v1 error envelope parity.** Python and Rust registry errors now
154
  return the spec envelope `{error: {code, message}}` for registry failures
155
  instead of the framework-native string-only shapes. The conformance harness
156
  now checks `/tlog/range` response shape plus representative
157
  `signature_invalid`, `sidecar_mismatch`, `missing_field`, and `not_found`
158
  error envelopes, raising the live/in-process harness to 38 checks.
159
- **Registry v1 candidate stability note.** Added
160
  `docs/REGISTRY_V1_STABILITY.md`, which names the candidate-frozen route
161
  surface, JSON field families, error envelope, conformance gate, breaking
162
  change rules, and remaining v1.0 burn-in gates for independent registry
163
  operators.
164
- **GitHub Actions runtime hygiene.** Main CI workflows opt into the GitHub
165
  Actions Node 24 runtime before the hosted runner default changes.
166
- **Rust policy test parity.** Fixed the `oversight-policy` crate's manifest
167
  fixture after the v0.4.11 `Recipient.p256_pub` schema addition so the full
168
  Rust workspace test suite compiles again.
169
- **Deployment docs.** Added `docs/REGISTRY_DEPLOYMENT.md` covering the live
170
  Compose/Caddy flow, route map, token headers, DNS bridge secret, and local
171
  versus live conformance commands.
172
- **Public description refresh.** Updated README/roadmap/embedding copy to
173
  describe v0.4.11 as the current stable line and the post-tag Rust registry
174
  deployment and migration work on `main`.
175
- **Code comment style.** Added `CONTRIBUTING.md` guidance to prefer
176
  self-explanatory code and tests over prose-style inline comments, then
177
  removed noisy implementation comments from the Rust registry path.
178
- **Source comment guard.** Added `scripts/check_source_comments.py`, a pytest
179
  wrapper, and a `source-style` GitHub Actions workflow so strict comment-light
180
  paths fail CI if prose comments are reintroduced.
181
- **Rust PDF extraction parity.** `oversight-formats` now uses lopdf page text
182
  extraction plus parsed content-stream operations for PDF fingerprint text
183
  instead of raw literal scanning. The fallback handles `Tj`, `TJ`, quote
184
  operators, and array spacing, with new Rust tests covering page-level PDF
185
  text extraction.
186
- **Rust image DCT parity.** `oversight-formats` now ports the Python image
187
  adapter's DCT mid-band spread-spectrum watermarking path using `rustdct`.
188
  Image watermarking writes the DCT mark and then preserves blind LSB recovery,
189
  with tests for the Python-compatible mark sequence, DCT verification, and
190
  adapter round trip.
191
- **Outlook add-in hosted pilot page.** `integrations/outlook/index.html`
192
  documents the hosted manifest URL, task pane URL, requested `ReadItem`
193
  permission, same-origin viewer reuse, sideload steps, and remaining tenant
194
  load-test gates for the Outlook read-mode inspector.
195
 
196
## v0.4.11 - 2026-05-08 Hardware-keys completion: Python parity, browser support, end-to-end seal
197
 
198
The `OSGT-HW-P256-v1` suite is now implemented end-to-end across all
199
three reference implementations: Rust core, Python core, and the
200
public browser inspector. Every layer of the protocol ships every
201
suite. The only piece deferred to a follow-up is the `PivKeyProvider`
202
(PKCS#11 binding to actual hardware tokens) and the matching
203
`--recipient-hw` CLI flag.
204
 
205
- **`oversight_core.crypto`: Python parity for `OSGT-HW-P256-v1`
206
  (2026-05-08).** New `wrap_dek_for_recipient_p256` and `unwrap_dek_p256`
207
  mirror the Rust reference byte-for-byte: same HKDF info string
208
  (`"oversight-hw-p256-v1-dek-wrap"`), same AEAD AAD
209
  (`"oversight-hw-p256-dek"`), same SEC1 uncompressed (65 byte) wire
210
  format for the ephemeral public key, same wrapped envelope JSON shape
211
  including the explicit `"suite"` field. `unwrap_dek_p256` accepts
212
  either an `EllipticCurvePrivateKey`, a PKCS#8-encoded private key, or
213
  a raw integer scalar so a future PIV / PKCS#11 binding has a portable
214
  on-ramp. `oversight_core.container` now recognizes `suite_id = 3` and
215
  maps it to `OSGT-HW-P256-v1` in `SUITE_ID_TO_NAME`. New
216
  `tests/test_hw_p256.py` (10 tests) covers the round trip across all
217
  three private-key input forms, the on-wire envelope shape against
218
  `SPEC.md` &sect; 5.2, and the negative paths (wrong recipient, wrong
219
  ephemeral key length, missing fields, AAD binding so a classic
220
  envelope's bytes do not silently decrypt through the hardware path).
221
 
222
- **`oversight-container`: end-to-end seal/open for `OSGT-HW-P256-v1`
223
  (2026-05-07).** New `seal_hw_p256` mirrors `seal` but consumes a P-256
224
  SEC1 uncompressed recipient public key and writes a container with
225
  `suite_id = 3`. New `open_sealed_with_provider` is the polymorphic
226
  open path: dispatches on the container's `suite_id` and delegates the
227
  recipient-side ECDH to a `KeyProvider`. Today it supports classic (with
228
  `FileKeyProvider`) and HW P-256 (with `SoftwareP256KeyProvider` or any
229
  future `PivKeyProvider`); a hybrid-aware provider extension lands later.
230
  Cross-suite mismatches (e.g. an X25519 provider on a HW P-256 container)
231
  are refused explicitly. `oversight-manifest::Recipient` gains an
232
  optional `p256_pub: Option<String>` field, gated by `serde(default,
233
  skip_serializing_if = "Option::is_none")` so existing JSON manifests
234
  parse unchanged. Five new round-trip / negative tests; `oversight-
235
  container` now 17/17, workspace builds clean.
236
 
237
## v0.4.10 - 2026-05-07 Hardware-keys foundation: KeyProvider trait + OSGT-HW-P256-v1
238
 
239
This release lands the abstraction and pure-Rust reference path that the
240
upcoming `PivKeyProvider` (PKCS#11 against YubiKey / Nitrokey / OnlyKey)
241
plugs into. Public API is purely additive; all existing v0.4.9 callers
242
keep working unchanged.
243
 
244
- **`oversight-container`: `OSGT-HW-P256-v1` recognized by the binary
245
  container (2026-05-07).** Added `SUITE_HW_P256_V1_ID = 3` and extended
246
  `suite_id_for_manifest` to map the new manifest suite. This is the bridge
247
  that lets a future `seal_hw_p256` ride the existing container layout
248
  without reinventing it. New unit test covers the full mapping and asserts
249
  unknown suites still return `None`. 12/12 container tests; workspace
250
  build clean.
251
 
252
- **`oversight-crypto`: `OSGT-HW-P256-v1` suite implementation (2026-05-07).**
253
  P-256 ECDH wrap/unwrap landed alongside the X25519 path so hardware-backed
254
  recipients (YubiKey / Nitrokey / OnlyKey via PIV) have a complete pure-Rust
255
  reference to plug into. `wrap_dek_for_recipient_p256` accepts SEC1
256
  uncompressed (65 byte) recipient public keys and produces `WrappedDekP256`.
257
  `SoftwareP256KeyProvider` is the in-memory `KeyProvider` impl that
258
  `PivKeyProvider` will mirror against PKCS#11 next. Cross-suite envelopes
259
  are rejected explicitly: an X25519 provider passed to a P-256 envelope
260
  errors out instead of producing garbage. Eight new unit tests covering
261
  round trips, wrong-recipient rejection, cross-suite rejection, JSON
262
  envelope round-trip, and a regression check that the classic path still
263
  works. `SUITE_HW_P256_V1` constant exported. Adds `p256` (RustCrypto) to
264
  workspace deps with `ecdh` + `arithmetic` features. `oversight-crypto`
265
  passes 21/21; workspace build clean.
266
 
267
- **`oversight-crypto`: `KeyProvider` trait + `FileKeyProvider` (2026-05-07).**
268
  The recipient-side ECDH path is now abstracted behind `pub trait KeyProvider`,
269
  with `FileKeyProvider` shipping as the X25519 file-backed default. New
270
  `unwrap_dek_with_provider` is byte-identical to `unwrap_dek` for file-backed
271
  keys (asserted by tests) and is the entry point hardware-backed providers
272
  (PIV / PKCS#11) will plug into next, per `docs/HARDWARE_KEYS.md`. Public API
273
  is purely additive: existing `unwrap_dek(wrapped, priv_bytes)` callers are
274
  unchanged. `KeyAlgorithm::P256` reserved for the upcoming `OSGT-HW-P256-v1`
275
  suite. Six new unit tests; workspace build clean; `oversight-crypto` passes
276
  13/13.
277
 
278
## v0.4.9 - 2026-05-07 Hybrid browser decrypt, Rust registry v1, Outlook scaffold
279
 
280
The browser inspector now decrypts post-quantum sealed files end-to-end,
281
the Rust registry passes the v1 conformance harness 33/33, and a thin
282
Outlook task-pane scaffold lets us start a tenant pilot. Format watermark
283
regressions in `oversight-rust/oversight-formats` are also resolved.
284
 
285
- **Outlook add-in scaffold landed (2026-05-07).** New `integrations/outlook/`
286
  with the Office add-in 1.1 manifest (`MailApp`, read-mode task pane,
287
  `ReadItem` only), task-pane HTML, and JS that imports the public viewer's
288
  `parseSealed`, `verifyManifestSignature`, and `decryptSealed` directly
289
  from `oversightprotocol.dev/viewer/...` rather than reimplementing crypto.
290
  Decrypts both classic and hybrid suites. Architecture decision recorded in
291
  `docs/OUTLOOK.md`. Status: scaffold; not yet load-tested in an Outlook
292
  tenant. Icons (64/128 px) still pending.
293
- **Browser inspector: hybrid (post-quantum) decrypt shipped (2026-05-03).**
294
  The viewer at `oversight-protocol.github.io/oversight/viewer/` now decrypts
295
  `OSGT-HYBRID-v1` sealed files end-to-end, in addition to the
296
  `OSGT-CLASSIC-v1` path that shipped earlier. Implementation reuses
297
  WebCrypto X25519 + HKDF-SHA256 and the existing vendored `@noble/ciphers`
298
  XChaCha20-Poly1305, plus a newly vendored `@noble/post-quantum` ML-KEM-768
299
  for the post-quantum half of the KEM. KEK is bound X-wing-style over both
300
  shared secrets and both ephemeral inputs (`ss_x || ss_pq || eph_pub ||
301
  mlkem_ct`), matching `oversight_core.crypto.hybrid_wrap_dek`. New files
302
  on the site: `viewer/vendor/noble-post-quantum-ml-kem-0.6.1.js` (+ three
303
  vendored transitive deps from `@noble/hashes` and `@noble/curves`),
304
  `viewer/samples/tutorial-hybrid.sealed`, and
305
  `viewer/samples/tutorial-hybrid-identity.json`. New "Load hybrid tutorial
306
  identity" button surfaces the test fixture. New tooling:
307
  `tools/gen_hybrid_sample.py` (self-contained sample generator that mirrors
308
  the production hybrid wrap construction, runs anywhere `oqs` and
309
  `cryptography` are available), and `tools/test_hybrid_decrypt_node.mjs`
310
  (Node-based end-to-end smoke test against Node's WebCrypto).
311
- `oversight-rust/oversight-registry`: added the missing registry v1
312
  read-only and beacon surface (`/.well-known/oversight-registry`,
313
  `/evidence/{file_id}`, `/tlog/head|proof|range`, `/p/{token_id}.png`,
314
  `/r/{token_id}`, `/v/{token_id}`, `/candidates/semantic`) and tightened
315
  CORS to the public browser-inspector origins with GET/OPTIONS only. The
316
  Axum server now passes the existing 33-check
317
  `tests/test_registry_conformance.py` harness in live-URL mode.
318
- `oversight-rust/oversight-manifest`: added `canonical_content_hash` and
319
  `l3_policy` to the signed manifest model so Rust verifies Python-signed
320
  v0.4.5+ manifests without dropping signed fields before canonicalization,
321
  while retaining a fallback verification path for older manifests that lack
322
  those default fields.
323
- `oversight-rust/oversight-formats`: fixed Rust text/image watermark
324
  regressions that were failing the workspace test suite. Text embedding now
325
  keeps L2 trailing-whitespace marks at physical line endings after L1
326
  zero-width insertion, and image LSB embedding avoids duplicate pixel slots
327
  that could overwrite earlier payload bits.
328
 
329
## v0.4.8 - 2026-04-29 Mobile-build portability and rustls-webpki security bump
330
 
331
Patch release covering two upstream-driven fixes that landed on `main`
332
since v0.4.7. No new features and no breaking changes.
333
 
334
- `oversight-rust/oversight-container`: gate the 4 GiB
335
  `MAX_CIPHERTEXT_BYTES` literal to 64-bit targets and fall back to
336
  `usize::MAX` on 32-bit. Required to cross-compile the Rust core for
337
  Android `armv7-linux-androideabi` and `i686-linux-android`, which the
338
  mobile companion (`oversight-protocol/oversight-mobile`, Flutter +
339
  Rust via `flutter_rust_bridge`) embeds unchanged. Behavior is preserved
340
  for any realistic bundle on 32-bit; `usize::MAX` is just under 4 GiB
341
  on those targets. (PR #4, merged 2026-04-26.)
342
- `oversight-rust` Cargo.lock: bumped `rustls-webpki` from 0.103.12 to
343
  0.103.13. Patches a reachable panic in CRL parsing
344
  (GHSA-82j2-j2ch-gfr8) and an inverted-meaning URI excluded-subtree
345
  check (rustls/webpki#471). In scope because the Rust registry and
346
  Rekor clients use rustls for TLS. (Dependabot PR #3, merged 2026-04-29.)
347
 
348
## v0.4.7 - 2026-04-22 Registry federation hardening and conformance harness
349
 
350
Federation stops being aspirational when a second operator can prove
351
compatibility. v0.4.7 hardens the registry v1 interop spec against the
352
reference implementation and ships a conformance harness that any
353
operator can point at their deployment.
354
 
355
- `docs/spec/registry-v1.md`: expanded with the canonicalization algorithm
356
  (`json.dumps(sort_keys=True, separators=(",", ":"))` over UTF-8), the
357
  uniform error envelope and `code` vocabulary, a full endpoint table
358
  including the normative beacon paths (`/p/{token_id}.png`, `/r/{token_id}`,
359
  `/v/{token_id}`), the `/.well-known/oversight-registry` shape, the
360
  `/evidence/{file_id}` bundle fields, and the `/tlog/head|proof|range`
361
  endpoints federated verifiers rely on. Removed a phantom
362
  `/query/{file_id}` endpoint that was in the draft but never shipped.
363
- `tests/test_registry_conformance.py`: 32-check harness with two modes.
364
  In-process against a FastAPI `TestClient` for CI, or against a live URL
365
  when `OVERSIGHT_REGISTRY_URL` is set. Covers identity, liveness, a full
366
  signed-manifest registration round trip, attribution by token id,
367
  evidence bundle shape, transparency-log head, every beacon endpoint,
368
  and DNS event authentication.
369
- `docs/ROADMAP.md`: the registry federation item references the harness
370
  as the acceptance gate for federation.
371
- Version bumped to `0.4.7`. No breaking changes.
372
 
373
## v0.4.6 - 2026-04-22 SIEM export: Splunk, Sentinel, and Elastic
374
 
375
Registry beacon events can now be emitted in three SIEM-native formats so
376
security teams get Oversight data into the incident pipeline they already
377
run. Formatters are pure; transport is a thin sink layer.
378
 
379
- `oversight_core/siem.py`: new module. Normalized `OversightEvent` model
380
  built from the registry `events` table, pure formatters for Splunk HEC,
381
  Elastic Common Schema 8.x, and Microsoft Sentinel (Log Analytics custom
382
  logs), plus `sentinel_authorization()` helper that signs the Data
383
  Collector API `Authorization` header per Microsoft's recipe.
384
- `cli/oversight.py`: new `oversight siem export` subcommand. Streams
385
  events as JSON lines to stdout, a file, or an HTTPS collector. Supports
386
  `--since`, `--limit`, repeatable `--header`, and Splunk source/sourcetype/
387
  index overrides. Opens the registry database read-only so it is safe
388
  to run against a live service.
389
- `docs/SIEM.md`: operator integration guide covering each of the three
390
  SIEMs, the event field dictionary, the Sentinel HMAC signing window,
391
  and the honest beacon-absence caveat. Also surfaced from the website
392
  docs index.
393
- `tests/test_siem_unit.py`: 11 focused unit tests covering envelope
394
  shape per format, empty-field suppression, SQLite row mapping,
395
  read-only iteration, Sentinel HMAC stability, and action-name
396
  coverage for every beacon kind.
397
- `oversight_core/__init__.py` and `pyproject.toml`: version bumped to
398
  `0.4.6`. No breaking changes; SIEM is additive.
399
 
400
## v0.4.5 - 2026-04-20 L3 safety, GUI, and registry federation docs
401
 
402
Review-driven hardening from the local security review report.
403
 
404
- `oversight_core/l3_policy.py`: new L3 safety policy engine. L3 defaults off
405
  for legal, regulatory, technical/spec, source-code, SQL, log, and structured
406
  data classes; explicit `full`, `boilerplate`, and `off` modes are supported.
407
- `cli/oversight.py` and `cli/oversight_rich.py`: seal-time L3 disclosure now
408
  requires acknowledgement when L3 is enabled, and seal manifests record the
409
  applied L3 policy.
410
- `oversight_core/manifest.py`: manifests now carry `canonical_content_hash`
411
  so auditors can diff recipient copies against the original source bytes.
412
- `oversight_core/watermark.py` and `oversight_core/formats/text.py`: high-level
413
  L3 application is opt-in; L1/L2 remain available by default.
414
- `cli/gui.py`: added a Tkinter desktop GUI for key generation, sealing, and
415
  opening files (`oversight gui`) so non-technical users have a starter path.
416
- GUI and CLI output writes now fail closed against private-key overwrites,
417
  same-path writes, reserved Windows device names, malformed key files, and
418
  non-UTF-8 watermark attempts. Private-key writes use atomic replacement and
419
  restrictive permissions/ACL hardening where supported.
420
- `.sealed` parsing now rejects tampered suite IDs, malformed manifest/wrapped-DEK
421
  JSON, unknown manifest fields, and trailing bytes after ciphertext.
422
- `oversight-rust/oversight-container`: Rust now mirrors the Python parser's
423
  strictness by rejecting suite-byte tamper and trailing bytes after the
424
  authenticated ciphertext region.
425
- `docs/security.md`: documented L3 collusion/canonicalization limits, layer
426
  survival properties, passive beacon limits, jurisdiction-by-IP limits, and
427
  RFC 3161 timestamp semantics.
428
- `docs/spec/registry-v1.md`: added a registry federation/interoperability
429
  draft for independent compatible registry operators.
430
- `docs/ROADMAP.md`: corrected launch sequencing, dropped near-term FedRAMP,
431
  scoped ecosystem plugins to Outlook-first, and prioritized SIEM integration
432
  before SOC 2 / ISO 27001 work.
433
- Raised vulnerable dependency floors flagged by Dependabot/PyPI advisory
434
  checks: setuptools, cryptography, PyNaCl, pydantic, python-multipart,
435
  Pillow, and pypdf now require patched minimums; Rust manifest floors
436
  now pin patched minima for sqlx, tokio, rand_core, zip, chrono, regex,
437
  once_cell, and tracing-subscriber.
438
- Added focused regression coverage in `tests/test_l3_policy_unit.py`.
439
 
440
## v0.4.4 - 2026-04-20 security hardening
441
 
442
Security patch line started from the `v0.4.3` Python package baseline
443
(`0b1a4ab`) and incorporates the Codex review fixes made on 2026-04-20.
444
This is the current `main` download line. Historical `v0.5.0` Rekor/Rust
445
work remains in git history and the Rust workspace, but the Python package
446
metadata now intentionally advances from `0.4.3` to `0.4.4` so users do not
447
confuse the hardened tree with the vulnerable `v0.4.3` baseline.
448
 
449
- `oversight_core/container.py`: `max_opens` now increments only after a
450
  successful decrypt, and unsafe `seal_multi()` is disabled until the
451
  manifest format can honestly represent multiple recipients.
452
- `oversight_core/policy.py`: `LOCAL_ONLY` counter locking now works on
453
  Windows, and `REGISTRY` / `HYBRID` fail closed instead of silently using
454
  local state.
455
- `oversight_core/rekor.py`: offline verification now rejects DSSE envelopes
456
  whose subject digest does not match the expected content hash.
457
- `registry/server.py`: Rekor attestations now use real watermark mark IDs
458
  and the manifest's actual `content_hash`, and `/register` now rejects
459
  unsigned beacon / watermark sidecars that do not match the signed manifest.
460
- `oversight_core/formats/text.py`: text adapter now applies L3 before L2/L1,
461
  matching the core watermark pipeline.
462
- `oversight_core/tlog.py`: empty-tree roots now use the RFC 6962 Merkle
463
  hash (`SHA-256("")`) instead of an all-zero placeholder.
464
- `oversight_core/__init__.py`, `pyproject.toml`, and the Rich CLI banner:
465
  version metadata is now `0.4.4`, marking this post-`0.4.3` hardening train.
466
- `oversight_dns/server.py` and `registry/server.py`: DNS beacon callbacks now
467
  support a shared `OVERSIGHT_DNS_EVENT_SECRET`, and non-loopback callbacks
468
  fail closed when no secret is configured.
469
- `registry/server.py`: evidence bundles now include local transparency-log
470
  inclusion proofs for recorded events, not just the signed tree head.
471
- `oversight-rust`: removed the direct `rand` dependency in favor of
472
  `rand_core::OsRng`, clearing the low-severity `rand` advisory path.
473
- `oversight-rust/oversight-registry`: `/dns_event` now requires
474
  `OVERSIGHT_DNS_EVENT_SECRET` for non-loopback callbacks, signed
475
  beacon/watermark artifacts fail registration when malformed instead of being
476
  silently dropped, and Rekor attestation skips watermarkless registrations
477
  rather than logging `mark:<file_id>`.
478
- `oversight-rust/oversight-container` and `oversight-rust/oversight-policy`:
479
  Rust opens can now enforce `max_opens` after successful recipient decrypt,
480
  `REGISTRY` / `HYBRID` modes fail closed instead of falling back to local
481
  counters, and Rust `seal_multi()` fails closed until recipient-honest
482
  manifests exist.
483
- `oversight-rust/oversight-rekor`: offline verification now mirrors Python by
484
  rejecting DSSE envelopes whose subject digest does not match the expected
485
  content hash.
486
- `oversight-rust/oversight-formats`: DOCX metadata insertion no longer reports
487
  success when `<cp:keywords>` is missing, and PDF processing rejects indirect
488
  Launch / JavaScript / unsafe URI actions before rewriting files.
489
- Added focused regression coverage in `tests/test_policy_unit.py`,
490
  `tests/test_registry_unit.py`, `tests/test_rekor_unit.py`,
491
  `tests/test_text_format_unit.py`, and `tests/test_tlog_unit.py`.
492
 
493
Patch sequence on top of `v0.4.3`:
494
 
495
1. `0.4.3` / `0b1a4ab`: Rich CLI, anti-stripping defenses, and L3
496
   integration baseline.
497
2. `0.4.4` / `dab6157`: policy and Rekor verification hardening.
498
3. `0.4.4` / `4d60e3b`: registry Rekor mark indexing fix.
499
4. `0.4.4` / `20a566b`: multi-recipient sealing fails closed until the
500
   manifest can represent multiple recipients honestly.
501
5. `0.4.4` / `482f294`: default beacon/registry domain updated from
502
   `oversight.example` to `oversightprotocol.dev`.
503
6. `0.4.4` / `7712f98`: signed registry sidecars enforced and RFC 6962
504
   empty tlog roots fixed.
505
7. `0.4.4` / `0a7a2da`: package, core, and CLI version metadata
506
   aligned to the hardened `0.4.4` line.
507
8. `0.4.4` / `69e50aa`: public changelog patch chronology documented.
508
9. `0.4.4` / `26db8d3`: DNS evidence hardening, Rust RNG dependency
509
   cleanup, and evidence-bundle inclusion proofs.
510
10. `0.5.0+` / `b9bee41`: Claude-added Rust format adapters, Axum registry,
511
    and USENIX benchmark scaffolding.
512
11. `0.5.0+` / current hardening commit: Codex audit fixes for the new Rust
513
    registry/container/policy/Rekor/format-adapter security regressions.
514
 
515
## v0.5.0 - 2026-04-19
516
 
517
First release with public-Rekor attestations. Now hosted at
518
https://github.com/oversight-protocol/oversight (so the v0.5 predicate URI
519
resolves for any third-party verifier).
520
 
521
### Session B (registry wiring + e2e + backcompat)
522
- `registry/server.py`: `/register` now opt-in attests each registration into
523
  a public Rekor v2 log. Off by default; opt in with
524
  `OVERSIGHT_REKOR_ENABLED=1`. Failures non-fatal - local SQLite tlog stays
525
  authoritative for "list marks for issuer X" queries.
526
- `oversight_core/rekor.py upload_dsse`: fixed three wire-shape bugs against
527
  current rekor-tiles proto (`verifier``verifiers` array, `keyDetails` as
528
  sibling of `publicKey`, `raw_bytes` carries DER not PEM). Verified live
529
  against `log2025-1.rekor.sigstore.dev` - got real `log_index` returned.
530
- `tests/test_rekor_e2e.py`: 2 live tests, gated behind
531
  `OVERSIGHT_REKOR_E2E=1` so default runs do not append entries to the
532
  public log.
533
- `tests/test_rekor_backcompat.py`: 5 offline checks of v0.4 contract
534
  preservation.
535
 
536
### Session C (Rust crate + cross-language conformance + version bump)
537
- New crate `oversight-rust/oversight-rekor`: bit-identical port of
538
  `oversight_core.rekor`. 9 inline tests cover PAE byte-exactness,
539
  sign/verify round trip, tamper + wrong-key rejection, statement shape,
540
  canonical envelope JSON, and offline TLE inclusion check.
541
- New conformance suite `oversight-rust/tests/conformance_rekor.sh`: proves
542
  Python ↔ Rust bit-identity in 4 ways - PAE bytes, Python-signs/Rust-verifies,
543
  Rust-signs/Python-verifies, canonical payload bytes for the same statement.
544
- Version bumped to 0.5.0 across `oversight-rust/Cargo.toml`, `README.md`,
545
  `docs/SPEC.md`.
546
 
547
Hard constraints respected: no new crypto primitives (RustCrypto +
548
`cryptography`'s Ed25519 only), test count additions-only, Python ↔ Rust
549
bit-identity proven by conformance script.
550
 
551
## Unreleased - v0.5 Session A (2026-04-19)
552
- Added `docs/V05_REKOR_PLAN.md`: full Rekor v2 migration plan, verified
553
  against current upstream API (Rekor v2 GA 2025-10-10, DSSE + hashedrekord
554
  only, tile-backed reads, no online proof API, public log shard rotates
555
  ~6 months).
556
- Added `oversight_core/rekor.py` (~280 LOC): DSSE statement construction,
557
  PAE-exact signing/verification against the spec, Rekor v2 `/api/v2/log/entries`
558
  upload helper, offline inclusion-check helper, and `build_bundle()` shaper.
559
- Added `docs/predicates/registration-v1.md`: the URI the predicate type
560
  resolves to, with privacy contract and field schema.
561
- Added `tests/test_rekor_unit.py` with 10 offline unit tests covering DSSE
562
  PAE, sign/verify, tamper rejection, wrong-key rejection, statement shape,
563
  canonical envelope JSON, offline bundle verification, the recipient-pubkey
564
  privacy guarantee, predicate-version int, and 5-year-replay bundle fields.
565
- Six desktop-review fixes baked into Session A before commit:
566
  - Recipient X25519 pubkey now SHA-256 hashed before going on-log
567
    (deanonymization fix).
568
  - Predicate URI pinned to git-tagged GitHub path, not `oversight.dev`.
569
  - Bundle gained `bundle_schema: 2` integer + `log_pubkey_pem` +
570
    `checkpoint` + `log_entry_schema` + optional `rfc3161_chain`.
571
- Conformance script `oversight-rust/tests/conformance_cross_lang.sh` now
572
  derives REPO_ROOT from its own location instead of `/home/claude` hardcode.
573
- `HANDOFF.md` gained explicit "what NOT to accept from a future Claude
574
  session" section per the v0.4.1 retro.
575
 
576
Test count: 76 → 86 (additions only, baseline conformance still green).
577
 
578
## v0.4.1 - 2026-04-18
579
 
580
Cosmetic polish only, no functionality changes.
581
 
582
### Fixed
583
- Removed unused `std::path::Path` import from `oversight-policy` - clean
584
  `cargo build --workspace --release` with zero warnings.
585
- Rust workspace version bumped to 0.4.1 across all crates via
586
  `version.workspace = true`.
587
 
588
### No behavioral changes
589
All 76 tests (31 Python + 42 Rust + 3 conformance) still green.
590
 
591
---
592
 
593
## v0.4.0 - 2026-04-17
594
 
595
**Rust port expands from core to core+enforcement+semantics.** Three new Rust
596
crates; Python reference unchanged in functionality but with RFC 6962 fix.
597
 
598
### Added
599
 
600
- **`oversight-tlog`** Rust crate (367 LOC). RFC 6962-compliant Merkle tree
601
  from day one - left-heavy largest-power-of-2 split, not the promote-odd
602
  shortcut from the Python v0.2 tlog. Signed tree heads, inclusion proofs,
603
  durable append (fsync), automatic recovery on reopen. 7 tests.
604
- **`oversight-policy`** Rust crate (284 LOC). TOCTOU-safe `max_opens`
605
  enforcement via `fs2::FileExt::lock_exclusive` + atomic temp-file rename.
606
  File-ID sanitization against path traversal. Jurisdiction / not_after /
607
  not_before checks. 6 tests.
608
- **`oversight-semantic`** Rust crate (345 LOC + 156-line dictionary file).
609
  Full port of the 151-class synonym dictionary and L3 watermarking.
610
  Airgap-strip-survivor verified (tests embed, then strip zero-width and
611
  trailing whitespace, then verify - still attributes). URL / email / code
612
  / path / hex / base64 skip regions. 8 tests.
613
- **Fuzz harness** (`oversight-rust/fuzz/`) - two `cargo-fuzz` targets
614
  hammering the container parser and manifest parser. Excluded from main
615
  workspace so normal builds don't need nightly. README with 24-hour
616
  pre-audit run recommendation.
617
- **`docs/HARDWARE_KEYS.md`** - vendor-neutral setup guide for YubiKey /
618
  Nitrokey / OnlyKey. Covers PIN/PUK setup, PIV slot 9d provisioning,
619
  Oversight identity-file format for hardware-backed recipients, curve
620
  choice rationale (P-256 for PIV compat vs X25519 file-backed), revocation
621
  procedure, threat model, deployment checklist.
622
 
623
### Fixed
624
 
625
- **`oversight_core/tlog.py`** now RFC 6962 compliant. Replaced the
626
  promote-odd-trailing shortcut with the canonical largest-power-of-2
627
  left-heavy split. Added `_rfc6962_mth`, `_rfc6962_path`,
628
  `verify_inclusion_proof` helpers. Tested with asymmetric sizes
629
  (n ∈ {1,2,3,4,5,7,8,16,17,100}) - every leaf's proof verifies;
630
  tampered proofs rejected. Old custom Merkle logic removed.
631
- **Mutex self-deadlock** in `oversight-tlog::inclusion_proof` - was
632
  holding the leaves lock while calling `root()` which also locks.
633
  Fixed by dropping the lock before invoking `root()`.
634
- **`oversight-semantic` round-trip bug** - `embed_synonyms` could pick
635
  hyphenated variants like `"write-up"` that `WORD_RE` tokenizes as two
636
  separate words, desyncing the verify sequence. Both embed and verify
637
  now explicitly skip non-round-trippable variants (whitespace or hyphen).
638
 
639
### Changed
640
 
641
- **Workspace version** bumped to `0.4.0`. Python reference remains `v0.3`
642
  (unchanged feature set, one correctness fix).
643
- **SealedFile** gained `#[derive(Debug)]` to support test assertions with
644
  `{:?}` formatting.
645
 
646
### Known limitations (unchanged from v0.3)
647
 
648
- Paraphrasing attack defeats all three watermark levels.
649
- Airgapped readers leave no network beacon.
650
- Hardware-backed recipients require v0.5+ `KeyProvider` abstraction (not
651
  yet implemented - currently file-backed only).
652
- Format adapters (image DCT, PDF, DOCX) remain Python-only until v0.6.
653
- Registry server (FastAPI) remains Python-only until v1.0.
654
 
655
## v0.3.0 - 2026-04-17
656
 
657
See earlier commits. Initial Rust core + FreeTSA RFC 3161 + cross-language
658
conformance + SENTINEL→Oversight rename + Nitro→YubiKey pivot.
659
 
660
## v0.2.1 and earlier
661
 
662
Python-only; see git history.