Detects chain members that have drifted semantically away from their
chain mates after an edit. Re-embeds changed YAMLs with the same model
the corpus uses (BAAI/bge-small-en-v1.5) and reports the min cosine to
each chain mate.
Default invocation (advisory):
python3 scripts/check_chain_decay.py
# diffs against origin/dev, flags chains with min mate-cosine < 0.40
Other modes:
--files <a.yaml> <b.yaml> explicit files instead of git diff
--base HEAD~5 different base ref
--threshold 0.50 tighter cutoff (slow drift detection)
--strict exit non-zero on flag (use as CI gate)
Default is advisory not blocking — first ship intentionally doesn't
fail commits or CI. The threshold 0.40 is calibrated against the
post-Phase-1 corpus; tune as needed once you've seen what real-edit
deltas look like in practice.
Implementation notes:
- Reuses embeddings.npz for chain-mate vectors (no re-embedding the
whole corpus per run).
- Only the changed question gets re-embedded — fast for typical
PR-sized changes.
- Skips changed questions that aren't in chains; skips chain
memberships where the mate isn't in embeddings.npz (e.g., the
Phase 3 promoted drafts before they hit the next embedding rebuild).
Smoke checks:
- --base origin/dev finds 4 changed YAMLs (the Phase 3 promoted
drafts), correctly reports no chain memberships (those questions
aren't in chains.json yet — by design, gated on human review).
- --files <cloud-2520.yaml> on a real chain member: cos=0.79 vs
its L5 mate cloud-2521 (well above 0.40 threshold ✓).
Closes the loop on the pilot pattern from a750ab7bc (manual promotion
inline script). Reads draft-validation-scorecard.json and either
promotes every passing draft (--all-passing) or an explicit list
(--qids edge-2536,edge-2537).
Per draft:
- strips _authoring private metadata; replaces with proper schema
fields (provenance, status, authors, human_reviewed, created_at)
- adds gap-bridge:<lower>-<higher> tag for traceability
- renames .yaml.draft → .yaml
- appends id to id-registry.yaml (append-only — preserves the
CI-enforced ledger contract)
Optional flags:
--publish flip status to published (default: keep as draft so
the human reviewer's workflow stays explicit)
--reviewed-by X set human_reviewed.status=verified, by=X, date=now
(implies the reviewer has actually read the drafts)
--dry-run preview without writing
Refuses to overwrite a <id>.yaml that already exists. Skips
already-promoted drafts (with a warning) when called with
--all-passing on a scorecard whose drafts have been promoted earlier.
Smoke checks:
- --all-passing on the existing scorecard correctly identifies all 4
pilot drafts as already-promoted (they shipped in a750ab7bc).
- --qids edge-2535 --dry-run on the leftover failed-validation draft
previews the promotion as expected.
Pull in the dev work that landed since yaml-audit was last synced:
- --legacy-json renamed to --local-json (2b381bb949) — script/doc
updates needed below in this branch
- CI workflow refactor (validate-dev / validate-vault now reusable)
- all-contributors automation, gitignore tightening, codespell list
- PR #1622 navbar URL rewrite for dev preview
- PR #1619 clone-size refactor, #1618 milestone3 xor fix, #1617
perceptron seed, #1616 tito status M3
- Chapter 9 PDF layout refinement
- assorted staffml/practice fixes (pickRandom deps, GitHub star gate)
This merges the canonical dev state into yaml-audit so subsequent
work continues on top of the freshest base. Conflicts in
practice/page.tsx + corpus.ts + ARCHITECTURE.md resolved to keep both
sides' additive changes (Phase 2 tier work + dev's later refactors).
Two new scripts that together close the loop from a gap entry to a
reviewable candidate question with a multi-gate scorecard.
generate_question_for_gap.py (3.a):
- Reads a gap entry, loads between-questions + same-bucket exemplars,
prompts gemini-3.1-pro-preview, runs Pydantic Question validation,
and writes <track>/<area>/<id>.yaml.draft. The .draft suffix keeps
drafts out of vault check / vault build until promotion.
- ID allocator scans corpus + existing drafts so a batch run gets
distinct fresh IDs without touching id-registry.yaml.
- Modes: --gap-index, --gaps-from + --limit, --dry-run.
validate_drafts.py (3.b):
- Five gates per draft: schema (Pydantic), originality (cosine vs
in-bucket neighbours via BAAI/bge-small-en-v1.5; matches the corpus
embeddings.npz so values are comparable; cutoff 0.92), level_fit
(Gemini-judge against same-level exemplars), coherence
(Gemini-judge: scenario/question/solution consistency), and bridge
(Gemini-judge: chain-fit between the gap's two anchors).
- Final verdict pass iff every non-skipped gate passes.
- Skips: --no-originality, --no-llm-judge.
- Output: interviews/vault/draft-validation-scorecard.json.
Smoke checks:
- 3.a --dry-run --gap-index 0: resolves gap, builds prompt, allocates
cloud-4579. Synthetic Gemini response Pydantic-validates clean.
- 3.b on a synthetic /tmp draft: schema + originality pass (top
neighbour cosine 0.73 vs 0.92 threshold).
Phase 3.c (pilot run on 30 gaps) deferred: it generates new YAML
question content that needs human review before promotion. The
tooling ships ready; running it is a user-supervised step.
CHAIN_ROADMAP.md Progress Log + Phase 3 status updated.
Phase 1.2 + 1.3 of CHAIN_ROADMAP.md. The two land together because the
prompt template, validator Δ-rule, and tier-tagging must stay in lockstep
or chains.proposed.lenient.json would mis-validate.
build_chains_with_gemini.py:
- new LENIENT_PROMPT_TEMPLATE alongside renamed STRICT_PROMPT_TEMPLATE;
lenient template tells Gemini to accept Δ ∈ {0,1,2,3}, with Δ=0 only
for shared-scenario same-level pairs and Δ=3 last-resort
- MODE_CONFIG single-source-of-truth maps mode → (template, allowed Δ set)
- validate_chain now takes mode= and gates on the per-mode Δ set
- process_batch tags lenient-mode chains with tier="secondary" and
a chain_id suffix (-secondary) so primary/secondary IDs never collide
- new --mode {strict,lenient} flag (default strict — primary chains
keep producing under the same rules as before)
- new --buckets-from <chain-coverage.json> flag that restricts the run
to the uncovered_buckets list from diagnose_chain_coverage.py
(the Phase 1.4 second-pass entry point)
apply_proposed_chains.py:
- docstring note: tier field is intentionally not validated here
(it's a UI hint, not a structural invariant)
- already accepts Δ=0 chains via its non-strict monotonicity check, so
no logic change needed
tests/test_chain_validation.py:
- 19 cases covering both modes: strict accepts +1/+2 and rejects Δ=0,
Δ≥3, and backward; lenient accepts Δ=0/Δ=3 but still rejects Δ≥4 and
backward; both modes reject size-out-of-range, multi-topic, and
unknown qids. Loads the script via importlib (it's not part of the
importable vault_cli package).
Smoke check (--dry-run --buckets-from chain-coverage.json --mode lenient):
17 calls planned for the 211 uncovered buckets, well under the 200 cap.
Brings the vault chain rebuild + sidecar architecture work into dev:
- Hierarchical question layout (interviews/vault/questions/<track>/<area>/<id>.yaml)
completed in earlier dev merge; this branch adds the sidecar split
- chains.json is now the authoritative chain registry; YAML chains: field
stripped from all 10,701 question files
- 373 chains rebuilt via Gemini 3.1 Pro Preview with strict progression
rules (Δ ∈ {1,2}, single-track, single-topic, multi-membership cap=2)
- 138 gaps surfaced into gaps.proposed.json for Phase 3 authoring
- Tooling: build_chains_with_gemini.py, apply_proposed_chains.py,
summarize_proposed_chains.py, diagnose_chain_coverage.py
- CHAIN_ROADMAP.md captures the resumable Phase 1-4 plan
State at merge:
- vault check --strict: 10,701 loaded, 0 invariant failures
- vault build --legacy-json: clean, releaseId=dev, 9438 published, 373 chains
- playwright UI suite (last run on yaml-audit): 13/13 pass
Phase 1.1 (diagnose_chain_coverage.py) shipped on yaml-audit; Phase
1.2-1.6 (lenient sweep, tier merge) still pending. See CHAIN_ROADMAP.md
Progress Log for the resumable cursor.
Loads the published corpus (via vault_cli.policy — single source of truth)
and chains.json, buckets by (track, topic), and emits chain-coverage.json
with two cuts:
- uncovered_buckets: ≥3 questions, 0 chains
- under_covered_buckets: ≥6 questions, ≤1 chain
Plus per-track summary + top-10 uncovered for quick read.
Output is gitignored — regeneratable, fed to Phase 1.4's --buckets-from.
Phase 1.1 of CHAIN_ROADMAP.md. See progress log for the run results
(211 uncovered buckets, edge/mobile/tinyml chain density 0.6-0.8 vs
cloud's 2.95, biggest miss is cloud:roofline-analysis at 144q/0 chains).
The flag is the StaffML frontend's local-dev fallback (read corpus.json
from disk via NEXT_PUBLIC_VAULT_FALLBACK=static), not a deprecated path.
"Legacy" implied "soon to be removed"; "local-json" describes its actual
role and reads correctly in scripts and docs.
- vault-cli: rename CLI flag, parameter, result key, and help text.
- CI workflows + pre-commit config: invoke the new flag name.
- All scripts that print the command (suggest_exemplars,
pre_commit_corpus_guard, promote_validated, rename_legacy_ids,
export_to_staffml, the paper analyze_corpus/generate_*) updated.
- Comments and docs (ARCHITECTURE, CHANGELOG, REVIEWS, TESTING,
MASSIVE_BUILD_RUNBOOK, DEPRECATED, AUTHORING, plus frontend
comments and .env.example / .gitignore) updated.
The "legacy_json" sentinel string in corpus_stats.json._meta.source
is intentionally NOT renamed — it is a stable artifact format read
by downstream paper-generation tooling.
build_chains_with_gemini.py: prompt now asks Gemini to also surface
missing-rung gaps — e.g., 'this bucket has L1 + L3 questions on the same
scenario thread but no L2 to bridge them.' Gaps are captured to
interviews/vault/gaps.proposed.json as a separate authoring backlog.
This is a free signal: it costs no extra calls, identifies pedagogical
holes the corpus doesn't yet fill, and feeds a future generation pass
(with independent validation before any new question is committed).
corpus.ts: getChainForQuestion now accepts an optional preferredChainId
so multi-chain questions can disambiguate via URL (?chain=...). Adds
getAllChainsForQuestion() returning every chain a qid belongs to.
Default behavior unchanged when only one chain exists.
Gemini prompt + structural validator now enforce:
- Consecutive Bloom delta MUST be 1 or 2 (rejects Δ=0 same-level pairs
and Δ≥3 huge jumps; backward steps already impossible)
- Strict +1 preferred; +2 accepted only when no +1 candidate exists
- A question can appear in up to 2 chains, but only if it's L1 or L2
(foundational anchor pattern); 3+ chain memberships are rejected as
over-stuffing
Empirical alignment: 70% of legacy chains were strict +1, 19% had +2
jumps, 8% had +3 jumps that we now reject as too-large pedagogical
moves. The new rules tighten quality while keeping the bulk of
defensible existing structure expressible.
build_chains_with_gemini.py — adaptive batched chain proposal:
- Buckets corpus by (track, topic), packs into ~80K-token batches
- Calls gemini-3.1-pro-preview with structured-output prompt
- Validates each proposed chain (size 2-6, monotonic, single-topic,
members exist, no cross-chain duplicates)
- Writes staging chains.proposed.json (never touches live registry)
Full-corpus plan: 313 buckets pack into 44 calls (well under 250/day Pro
cap, uses ~70K input tokens per call out of 1M context).
Test on tinyml:network-bandwidth-bottlenecks (6 questions) -> 2 well-formed
chains, Bloom-monotonic with coherent rationale (Hailo-8 PCIe arc + BLE
network arc).
apply_proposed_chains.py — gated migration:
- Re-validates staging file against live YAML corpus
- Backs up chains.json -> chains.json.bak
- Refuses to apply if any structural invariant fails
Adds two subcommands and supporting modules:
vault chains audit
Reports chain health: orphans, position-drift (gaps from filtered
members), stale-registry, intra-chain cosine distribution, weakest
chains list. Embedding-aware via --no-embeddings escape hatch.
vault chains suggest
For each orphan singleton, ranks rescue candidates within the same
(track, topic) bucket. Hybrid scoring:
HARD filter: level_delta in {0, 1, 2} (matches 92% of observed
chain edges across the corpus)
SOFT rank: embedding cosine + delta=1 priority
Bands: strong-merge / review-merge / below-threshold
Embeddings: bge-small-en-v1.5 (BAAI). Calibrated via
scripts/calibrate_chain_embeddings.py against the 726 healthy chains.
Empirical findings (in script header docstring):
- bge-small precision@1 = 0.283, recall@3 = 0.447
- bge-large gains only +0.013 P@1 at 7x embedding time — not worth it
- Same-bucket questions are inherently close (μ_pos=0.785, μ_neg=0.757);
so this is suggestion-only, never auto-apply.
Cross-encoder rerank experiment script included for future research
(BAAI/bge-reranker-base) — current run OOM'd on 16GB; deferred.
Embedding cache (.npz) is gitignored — reproducible from source.
Update ARCHITECTURE.md to reflect 87 curated topics and 131 edges. Refactor exemplar_coverage_audit.py to use vault.db instead of retired corpus.json. Update exemplar-gaps.yaml inventory.
The reviewer-identity spoof check tried base refs in the order
(origin/main, origin/dev, HEAD~1) and returned the first that
resolved. On dev, where origin/main is 132 commits behind origin/dev,
this picked main and diffed every vault YAML changed since that point —
sweeping up 100+ files unrelated to the current push and reporting
each as a spoof-check failure.
Fix: respect GITHUB_BASE_REF when set (PR mode), otherwise diff
against HEAD~1 (push mode). This produces exactly the file set the
check is meant to validate — what this PR or push is proposing —
not the entire branch divergence from main.
Verified locally on the codespell+codegen-hashes commit: now reports
"no vault/questions/ changes in this PR" instead of 100+ spurious
failures.
Brings the last outlier workflow file into the repo-wide
<cluster>-<verb>-<scope>.yml naming convention. Every other cluster
(book, tinytorch, kits, labs, instructors, mlsysim, slides, site,
staffml) uses this pattern; vault-ci.yml was the only one that didn't.
vault-ci.yml → staffml-validate-vault.yml
name: '🎯 StaffML · 🔎 Vault CI' → '🎯 StaffML · ✅ Validate (Vault)'
Now staffml-validate-vault.yml is a direct sibling of
staffml-validate-dev.yml — the former validates the vault data + CLI
+ worker, the latter validates the site build. Same verb, different
scope, easy to reason about.
Updated references:
.github/workflows/staffml-validate-vault.yml — self-reference in
the paths trigger (so the workflow still fires when it's edited)
interviews/vault/ARCHITECTURE.md §19.3 and §51 — both path refs
interviews/vault/TESTING.md §4.1 — workflow name + display name
interviews/vault-cli/scripts/check_registry_append_only.py — docstring
No branch-protection settings change needed — GitHub matches required
checks on the workflow's 'name:' field, not the filename. Anyone with
a bookmark to the old Actions-tab URL will get a 404 (harmless).
Other workflow naming I surveyed but deliberately LEFT alone
(all consistent with existing conventions):
staffml-update-paper.yml matches tinytorch-update-pdfs pattern
staffml-auto-pr.yml matches bot-workflow convention
staffml-welcome.yml single-word verb, standard
auto-label / update-contributors / infra-* / publish-all-live
are cross-cutting (no cluster prefix) by design
Four deployment-level fixes landed on the live Cloudflare worker + D1
instance:
1. compiler.py — populate chains table from chains.json. Pre-v1.0 the
table was never filled, which only mattered once D1 (which enforces
FKs by default, unlike SQLite) tried to insert chain_questions. The
cutover failed with FOREIGN KEY constraint failed until chains(id)
was populated.
2. types.ts (worker) — add competency_area, bloom_level, phase, and
human_review_* fields. Worker SQL was already SELECT *, so the new
columns flow through without code changes, but the TypeScript row
interface needed updating for downstream consumers.
3. rate_limit.ts — Math.max(60, …) floor on expirationTtl. Old calc
could emit values as low as 11s, which D1's KV backend rejects
(minimum 60s). Was throwing 1101 on every request after the
deployment. Tail logs showed 'Invalid expiration_ttl of 14'.
4. wrangler.toml — bump SCHEMA_FINGERPRINT to match the v1.0 vault.db
(b97218dae6354b1b…). Without this, /manifest reports
schema_fingerprint_ok: false and clients degrade.
New script:
scripts/ship_d1.py — end-to-end reload of D1 from the current YAMLs.
'vault build' → SQL dump → 'wrangler d1 execute --file'. Handles FK
ordering (chains first, then questions, then chain_questions). Used
for this cutover; repeatable for future schema bumps.
Deployment state (2026-04-22):
Worker URL: https://staffml-vault.mlsysbook-ai-account.workers.dev
D1 database: staffml-vault (254f630f-…) — 9,199 questions loaded
Release hash: 997747a8f43bbd89e03c6bb0e67865f8de35ac8316fbb0457ee0b8f955afb32f
Manifest: curl …/manifest returns 9,199 / schema_fingerprint_ok=true
GET question: /questions/cloud-0185 returns the post-Phase-2 v1.0 record
(zone=mastery, level=L6+, competency_area=latency, …)
Filtered list: /questions?track=cloud&level=L6%2B works with pagination
Site cutover is NOT in this commit. The existing hybrid path
(bundled corpus.json primary + worker /search secondary) keeps
working unchanged. To flip the site entirely to the worker:
export NEXT_PUBLIC_VAULT_API=https://staffml-vault.mlsysbook-ai-account.workers.dev
unset NEXT_PUBLIC_VAULT_FALLBACK
# then: next build && next deploy
That flip converts every caller from sync 'getQuestions()' to async
via corpus-source.ts — deferred because callers need an audit pass
to handle async correctly.
- d1-schema.sql: regenerated to match compiler.py changes. Adds
competency_area, bloom_level, phase, human_review_* columns to
questions table. Adds idx_questions_human_review index.
chain_questions PK changes from (chain_id, position) to
(chain_id, question_id) for multi-chain + non-contiguous support.
Drops deep_dive_title/deep_dive_url.
- codegen-hashes.txt: new baseline covering the v1.0 models.py,
d1-schema.sql, and @staffml/vault-types/index.ts.
Fixes the vault codegen --check drift test that was failing CI.
Two new tools.
vault lint <path>
Author-facing linter. Accepts a single YAML file or a directory.
Severity levels:
ERROR schema violation; question cannot be loaded
WARN likely misclassification (zone-level affinity mismatch,
chain position duplication, etc.)
INFO hygiene suggestions (human-review-pending on published Qs)
Zone-level affinity warning implements paper §3.3 Table 2 (line 397):
'An L1 question tagged as evaluation is flagged for review, since
evaluation is cognitively inconsistent with Bloom's Remember level.'
The warning is soft — marking an outlier does not reject it; it
surfaces for reviewer judgement. Quickly identifies the ~943 L6+
questions currently carrying zone=design that should probably be
zone=mastery.
scripts/check_schema_sync.py
CI drift check. Compares enum values in schema/enums.py against
schema/question_schema.yaml (the authoritative LinkML schema) and
exits non-zero if they disagree. Prevents the three-schema drift
that caused the v0.1 migration defects from recurring.
Enums cross-checked: Track, Level, Zone, BloomLevel, Phase, Status,
Provenance, HumanReviewStatus. Output on success: 'OK: 8 enums in
sync.' Wire into CI in a follow-up PR.
Archives pre-v1.0 scripts under scripts/archive/ in both
interviews/vault/ and interviews/vault-cli/. ARCHITECTURE.md §3.3
rewritten with a post-mortem on why path-as-classification could not
represent the paper's full 11-zone × 6-level taxonomy. CHANGELOG.md
added documenting the full v1.0 migration.
Adds suggest_exemplars.py script for identifying high-quality candidates.
Moves 86 top-scoring questions (1 per topic) from vault/questions/ to
vault/exemplars/ with provenance upgraded to human. Scored by presence
of napkin_math, common_mistake, solution length, and scenario length.
vault generate now finds exemplars for topic-specific generation.
Published count: 9,113 (86 moved to exemplar pool).
The hook read '.git/COMMIT_EDITMSG' via a literal relative path. That
works in a regular clone where .git is a directory, but fails silently
in a git worktree where .git is a file pointing at a per-worktree
gitdir under the main repo. In a worktree, Path('.git/COMMIT_EDITMSG')
never exists, so commit_message_has_override() always returned False
and legitimate Vault-Override trailers were rejected.
Resolve via 'git rev-parse --git-path COMMIT_EDITMSG' which returns
the correct path in both regular clones and worktrees. This matches
the pattern already used by the Makefile's HOOKS_DIR resolution.
No behaviour change in a regular clone; worktrees can now commit with
the Vault-Override trailer as documented.
Chip R7 findings:
R7-H-1 (HIGH): sw.js ReferenceError on offline fetch failure
`cached` was const-scoped inside `if (!manifestStale)` block but
referenced in the outer catch's "if (cached) return cached" offline
fallback. Offline users hit ReferenceError instead of cache.
Fix: hoist to `let cached = null` above the gate.
R7-H-2 (MEDIUM-HIGH): schema_fingerprint portability across SQLite versions
Previous compiler hashed all sqlite_master including FTS5 shadow tables
(questions_fts_data/idx/docsize/content/config) whose DDL varies across
SQLite versions. Host Python SQLite \u2260 Cloudflare D1 SQLite \u2192
fingerprint permanent mismatch \u2192 worker pinned to degraded mode forever.
Fix: filter shadow tables out on both sides (compiler.py + worker/index.ts);
fingerprint covers only user-authored DDL.
R7-M-3 (MEDIUM): schemaOk sticky on transient D1 failure
Previously any probe exception pinned schemaOk=false until release
rollover. Now: 5-minute retry window via schemaCheckedAt tracking.
R7-M-4 (MEDIUM): vault dup --vault-dir + pass-through
ACKS_PATH was CWD-relative; invoking CLI from non-default cwd silently
missed the ack file, legitimate templates reddied nightly CI forever.
Fix: vault dup --vault-dir flag + pass through to ack_pairs(vault_dir);
validator._scenario_dedup_lsh takes vault_dir; slow_tier threads it.
R7-M-5 (MEDIUM): FTS5 probe memoization
Previously probed sqlite_master on every /search request \u2014 directly
undid part of R5-H-1's manifest memo cost fix. Now: module-level
ftsProbed memo, reset on release_id change (FTS5 presence can only
change across releases).
R7-L-6 (LOW): reviewer-identity name clarity
Var was `committer_emails` but git log %ae is AUTHOR email. Behavior
was correct (intentional, so rebase-by-maintainer preserves chain);
renamed to commit_author_emails and updated comments.
R7-L-7 (LOW): manifest memo race on release rollover
maybeInvalidateSchemaCache nulled manifestMemo mid-write causing
microsecond stampede. Now: don't null memo \u2014 60s TTL is forgiving
enough staleness bound for release rollover.
Dean R8 findings:
R8-H-1 (HIGH): SLI cron was structurally broken
Previous SLI reconstructed canonical content_hash from worker JSON
response \u2014 but reconstruction omitted tags, chain, generation_meta
(WHITELIST_TOP includes tags + chain). Every hourly run false-positive'd
on any question with a non-empty tag list, effectively a pager-DoS.
Fix: compare worker's stored content_hash directly against release
vault.db's stored content_hash. Same compilation source \u2192 mismatch
means real corruption.
R8-M-2 (MEDIUM): SLI 404 handling
urlopen crashed on deprecated IDs during release rollover. Fix:
classify by response code. 404 \u2192 id_missing_in_worker (expected);
5xx \u2192 transport_errors (separate tally); only real hash mismatch
pages the operator.
R8-M-3 (MEDIUM): vault deploy + rollback primitives spec-only
ARCHITECTURE \u00a76.2 said 'default rollback = snapshot restore \u2014 always
works' but no vault deploy or vault rollback command existed, so the
R2 snapshot substrate that makes §6.2 true was never built.
Fix: implemented `vault deploy` with synchronous R2 snapshot
(wrangler d1 export + wrangler r2 object put before migration) and
`vault rollback --method snapshot|sql`. CLI now has 26 subcommands.
Deploy requires authenticated wrangler; code path exists.
R8-LM-4 (LOW-MED): D1 bootstrap migration
wrangler.toml referenced migrations_dir='migrations' but that dir
didn't exist. First-deploy-from-scratch relied on manual operator steps.
Fix: generated interviews/staffml-vault-worker/migrations/0001_bootstrap.sql
from compiler.DDL so `wrangler d1 migrations apply` works on a fresh D1.
Test matrix (post-R7+R8 integration):
pytest: 38 green in 0.15s
vitest: 7 green in 131ms
ruff: All checks passed
vault build: release_hash fe69d4c4... stable (unchanged \u2014 fingerprint
filter change affects release_metadata content, not
the release Merkle per \u00a73.5)
vault --help: 26 subcommands (added deploy + rollback)
Convergence tracking:
R1-R5 closed 90+ findings
R6 (Gemini queued, not yet invoked) \u2014 will launch with R9/R10
R7-R8 produced 12 new findings (2H + 1MH + 5M + 4L), all closed here
Pattern: each round still finds 8-12 issues. Not yet stable.
Expect 2-3 more rounds to hit 'no new findings' signal.
Gemini 3.1 Pro reviewed the full branch (371KB / 43K words) with 1M
context. Caught 9 cross-file issues none of the 4 prior per-file
rounds saw because they required seeing multiple systems at once.
CRITICAL fixes:
R5-C-1: _MIGRATION_TABLES omitted release_metadata (release.py:118).
Result: after `vault ship`, release_metadata never propagated to D1
\u2192 worker kept serving old release_id forever \u2192 cache never
invalidated \u2192 new release functionally invisible.
Fix: added 'release_metadata' to migration-participating tables.
R5-C-2: SW offline wake-up deleted the real cache (sw.js).
Result: when SW woke offline, currentRelease=null, cacheName
defaulted to '...-unknown'. Activate pruned all caches not matching,
i.e. it deleted the real cache. Offline users: no cache.
Fix: persist currentRelease to IDB on fetch success; restore on
activate; move cache pruning from activate to updateReleaseFromManifest
so it only runs AFTER a successful online manifest fetch.
R5-C-3: schema_fingerprint hand-edited in wrangler.toml (compiler.py +
worker/src/index.ts). Every DDL change required manually recomputing
+ pasting a hash + Worker redeploy; forgetting any step put the site
in degraded mode.
Fix: compiler.py now computes fingerprint from sqlite_master at build
time and stores in release_metadata. Worker reads it from the DB via
getManifest; env.SCHEMA_FINGERPRINT path removed.
HIGH fixes:
R5-H-1: getManifest hit D1 on every request before Cache API check
(worker/src/index.ts:130). Destroyed the \u00a710.4 cost target.
Fix: module-level manifest memo with 60s TTL. Invalidated on
release_id change (natural cadence from Cloudflare's eventual
propagation).
R5-H-2: _insert_stmt emitted NULL for columns absent in row (release.py).
Result: rolling back past a new NOT NULL column would crash on SQLite
constraint violation.
Fix: emit only columns actually in row dict; let SQLite apply defaults.
R5-H-3: ARCHITECTURE.md \u00a713 promised CI rejects --reviewed-by spoofing,
but no check existed.
Fix: new scripts/check_reviewer_identity.py + CI step. Verifies for
every changed question with provenance=llm-then-human-edited that at
least one `authors` entry matches a commit email from the PR.
R5-H-4: LSH dedup told operator to run `vault dup --ack` but that
command didn't exist \u2014 legitimate templates would red nightly CI
forever.
Fix: implemented `vault dup --ack`/--unack/--show. Writes to
vault/dedup-acks.yaml. Validator reads the ack list and skips flagged
pairs.
MEDIUM / LOW fixes:
R5-M-1: `vault tag` swallowed git failures with check=False.
Result: 'tag already exists', 'nothing to commit', merge conflicts
all printed '[green]tagged[/green]' and exited 0.
Fix: explicit error check on every subprocess call; pre-existence
check on tag like ship.paper_forward does.
R5-L-1: applicability-matrix invariant case-sensitive; 'Cloud' vs
'cloud' silently failed enforcement.
Fix: lowercase-normalize both sides of the comparison.
State:
pytest: 38/38 green in 0.15s
vitest: 7/7 green (fingerprint test updated to mock via release_metadata)
ruff: All checks passed
CLI: 23 subcommands (added vault dup)
release_hash: fe69d4c4... (unchanged \u2014 schema_fingerprint addition
affects release_metadata table, not content Merkle per \u00a73.5)
v2.3 \u2192 v2.4. ARCHITECTURE.md header + Appendix reflect the completed
migration.
WHAT CLOSED (\u00a711.1 contract):
1. `vault build --legacy-json` regenerates the site's
interviews/staffml/src/data/corpus.json from YAML. 9,199 published
questions, site-compatible shape (chain_positions back to 0-indexed
dict form, bloom_level derived from zone, competency_area aliased
from topic, scope aliased from track). Deterministic via sort_keys +
id-sort.
2. Pre-commit hook INSTALLED via worktree-aware Makefile target
(`make -C interviews/vault-cli hooks`). Symlink points at
pre_commit_corpus_guard.py. Tested end-to-end: direct edit to
vault/corpus.json triggers exit-1 with §11.1 reference.
3. CI equivalence check added to .github/workflows/vault-ci.yml:
regenerates corpus.json from YAML, diffs against committed. Fails
PR on drift with actionable error message.
4. Legacy generators demoted with DEPRECATED headers:
- interviews/paper/scripts/analyze_corpus.py \u2192 vault export-paper
- interviews/staffml/scripts/sync-vault.py \u2192 vault build --legacy-json
- interviews/staffml/scripts/generate-manifest.py \u2192 vault publish
- interviews/vault/scripts/export_to_staffml.py \u2192 vault build --legacy-json
5. New DEPRECATED.md files at interviews/vault/scripts/ and
interviews/staffml/scripts/ map every legacy script to its
replacement. Both directories keep the old scripts for git-history
legibility and archaeology; new contributors see the vault CLI first.
6. ARCHITECTURE.md \u00a7Appendix rewritten as current-state table instead
of aspirational "gone. replaced by..." entries.
NEW TESTS (interviews/vault-cli/tests/test_legacy_export.py \u2014 +4):
- test_legacy_shape_matches_site_interface: every field corpus.ts
declares is present in regenerated JSON.
- test_chain_positions_legacy_shape: 1-indexed new schema \u2192
0-indexed legacy dict form.
- test_emitter_deterministic: byte-stable across reversed input order
(required for CI diff-check).
- test_competency_area_aliases_topic: legacy alias fields populated
correctly.
FULL MATRIX GREEN:
pytest: 38/38 passed in 0.19s (34 + 4 legacy-export)
ruff: All checks passed
hook: exit 0 on clean diff / exit 1 on corpus.json direct edit
e2e: vault build --legacy-json regenerates a bit-identical corpus.json
vs the committed one; CI check wired to catch drift
WHAT'S LEFT (deploy-gated, \u00a720.5 #1, #5, #6 partial, #8, #9):
- Production serves from D1: requires Phase-3 wrangler d1 create + deploy
- Manual QA per CUTOVER_QA.md: requires live staging
- Zero data loss D1-side verification: requires live D1
- 48h monitoring: requires production traffic
These are intrinsically user-action; the YAML-side migration is done.
vault-cli/scripts/normalize_chain_positions.py (NEW)
Phase-1 split kept only chain_ids[0] per question when legacy corpus
had multi-chain membership (up to 4 chains/question). Chains whose
members chose a different chain_ids[0] were left with position gaps.
Script walks vault/questions/, groups by chain_id, renumbers each
chain's members to contiguous [1..N] sorted by current position.
Idempotent. Rewrote 87 questions across 977 chains.
validator.py #18 (provenance-meta)
Tightened from 'any non-human provenance requires generation_meta'
to 'only llm-draft / llm-then-human-edited require it'. Imported
content legitimately has no LLM attribution and shouldn't carry
stub meta. Was incorrectly flagging 9,199 imported questions.
Re-ran vault build → new release_hash (input changed, which is
correct): fe69d4c4d3c2884efeab6189a67e929e4e970dc0f4de42ab9493531a4cabeda1.
Republished 0.9.0 release artifact. corpus-equivalence-hash.txt updated.
paper/macros.tex + corpus_stats.json regenerated (same counts:
9199/87/964 chains/31.9% coverage).
State: vault check --strict 100% clean on full 9,657-question corpus;
zero load errors; zero invariant failures. 28/28 pytest green.
vault verify 0.9.0 round-trips from YAML source. Citation property
holds on the new hash.
vault-cli/src/vault_cli/commands/stats.py (NEW, B.8)
vault stats — live scorecard over vault.db with --format-prometheus
scrape mode + --exemplar-coverage audit shim. Reports total / topics
/ chains / by_status / by_track / by_provenance. Resolves R3 gap
about missing stats subcommand.
vault-cli/src/vault_cli/commands/codegen.py (NEW, B.7)
vault codegen --check — Phase-1 presence-and-non-empty verification
of the 3 shared-artifact files (models.py, d1-schema.sql,
@staffml/vault-types/index.ts). Full LinkML-driven generation is
Phase-2 follow-up.
vault-cli/Makefile (NEW, B.2)
make install / test / lint / hooks / hooks-uninstall. Hooks target
symlinks pre_commit_corpus_guard.py into .git/hooks/pre-commit.
vault-cli/scripts/check_registry_append_only.py (NEW, B.3)
CI script verifying id-registry.yaml is append-only vs base branch.
Rejects removed or reordered lines — C-5 enforcement at merge time.
vault/questions/LICENSE (NEW)
CC-BY-4.0 for corpus content. BibTeX template with release_hash
placeholder. Scope note clarifies vault-cli is MIT separately.
vault-cli/LICENSE (NEW)
MIT for vault-cli Python package + scripts + docs. Scope note
clarifies corpus is CC-BY-4.0 separately.
staffml/src/lib/corpus-vault.ts (NEW, B.11)
Vault-API-backed data source mirroring corpus.ts public surface.
Adapts @staffml/vault-types Question → legacy Question shape so
callers don't need to change. Not wired into any component yet —
the swap happens via corpus-source.ts.
staffml/src/lib/corpus-source.ts (NEW, B.11)
Cutover router: getCorpusSource() returns 'static' or 'vault-api'
based on NEXT_PUBLIC_VAULT_FALLBACK. Components that opt into the
cutover import from here; others continue using corpus.ts directly
(unchanged behavior). Phase-4 cutover flips components one-by-one
rather than big-bang-replacing corpus.ts.
Phase-1/2 now has the full CLI surface (19 subcommands), LICENSEs
for legal Phase-3 deploy, and the site-side cutover pathway ready
for Phase-4 canary.
.github/workflows/vault-ci.yml
Matches repo workflow style (emoji-prefixed name, concurrency group,
path-scoped triggers). Phase-0 scope: pip install, vault --version,
ruff, pytest, exemplar-audit staleness check. Python 3.12 pinned for
hash stability per ARCHITECTURE.md §3.5. mypy --strict included but
non-blocking at Phase 0; enforces in Phase 1. Placeholder for
vault check --strict, vault build, vault codegen --check as those
commands land.
interviews/vault-cli/scripts/exemplar_coverage_audit.py
Reads corpus.json, groups by (track, level, zone), counts total
questions vs exemplar-eligible per cell (requires provenance ∈
{human, llm-then-human-edited}). Phase-0 honest output: provenance
field doesn't exist in corpus.json yet, so eligible=0 for every
cell until Phase-1 YAML split + provenance backfill. Audit shape is
stable so Phase-1 re-runs slot in without refactoring.
interviews/vault/exemplar-gaps.yaml
First audit snapshot: 190 cells catalogued, all gap=3 pending
Phase-1. Filling gaps unblocks vault generate in Phase 7, not a
Phase 0 blocker (Chip N-H3 resolution).
Phase 0 milestone: complete.
