cs249r_book/taxonomy/method-study/sampling-frame.md

Enumerated Sampling Frame and Split Hygiene for the ML Systems Taxonomy

This document specifies the five-stratum sampling frame, provenance-family clustering rules, and split allocation protocol (dev, held_out_test, and reserve) synthesized from Method-A (§3), Method-B (§3), and audited Method-C (§3.2), together with the executed Phase 3 allocation (taxonomy/method-study/preregistration.yaml) and post-Phase-6 global cluster-burn ledger (taxonomy/method-study/burned-clusters.json) for P4 and Q4. Public verification uses the committed files in taxonomy/method-study/ (preregistration.yaml, burned-clusters.json, genesis-0.4-baseline.json.gz, study-receipts-summary.json, rubric.yaml) and taxonomy/term-provenance.yml; raw maintainer review workspaces under .claude/_reviews/ remain local and unversioned under the repository's private AI-configuration policy.

Note: Sampling-frame strata S1–S5 denote source populations and are distinct from Stack area codes S1–S4.

---

1. Five Stratified Source Populations (S1–S5)

Every candidate concept and every coverage or indexing evaluation unit is drawn from an enumerated frame partitioned into five source populations (Method-A §3.1, Method-B §3.1). Source stratum records where an excerpt comes from; warrant type under ANSI/NISO Z39.19 §5.3.5.1–§5.3.5.3 is recorded separately so operational proxies are not conflated with direct user search logs or organizational term-preference mandates:

Stratum ID	Stratum Name	Unit of Sampling	Inclusion Criteria & Warrant Role
S1	Textbooks & Reference Monographs	Chapter / major section (1 work per unit prospectively)	Published or open-access textbooks covering ML systems architecture, distributed training/serving, hardware acceleration, or safety/reliability engineering (both external monographs and project-internal MLSysBook chapters). Provides literary & curriculum warrant (Z39.19 §5.3.5.1; internal MLSysBook chapters carry origin: internal).
S2	University Courses	Course syllabus (lecture module + assignment/lab scope; 1 course per unit prospectively)	Graduate or upper-division undergraduate ML systems courses at accredited universities with public lecture schedules and reading lists, stratified across hardware/compilers, systems/networking, and safe/reliable deployment (Method-A §3.2). Provides external literary & pedagogical warrant (except Harvard CS249r, which shares MLSysBook authorship and carries origin: internal).
S3	Peer-Reviewed Systems & ML Venues	Paper (1 paper per unit prospectively: title, abstract, architecture/evaluation passage, or blind venue topic entry)	Peer-reviewed papers from systems conferences (MLSys, OSDI, SOSP, NSDI, ASPLOS, ISCA, EuroSys, CAV, AISec) and systems/safety tracks of ML/NLP venues (NeurIPS, ICML, ICLR, EMNLP, FAccT, SATML) within the six-calendar-year study window (2021–2026 inclusive at study freeze) plus pre-registered foundational papers (>500 citations in Google Scholar / OpenAlex or standard S2 syllabus adoption). Provides external literary warrant.
S4	Practitioner Artifacts, Postmortems & Diagnostic Scenarios	Engineering design doc, incident postmortem, runtime spec, or diagnostic scenario	Public production engineering design guides, open-source serving/runtime documentation (Triton, Ray Serve, ONNX Runtime — used solely to extract underlying mechanisms, never product concepts), frontier deployment governance policies, and StaffML diagnostic scenarios (origin: internal). Provides operational literary warrant and (for StaffML tag counts across the 10,716-question bank) an assignment-frequency proxy, not direct user search/request warrant (Z39.19 §5.3.5.3).
S5	Standards, Assurance Guidance & Benchmark Specifications	Standard clause / assurance guidance module / benchmark scenario specification	Published technical standards, public assurance guidance reports, and benchmark specifications (NIST AI 100-1, NIST AI 600-1, ISO 26262, ISO 21448, ISO/IEC 42001, ISO/PAS 8800, UL 4600, OWASP Top 10 for LLM, MITRE ATLAS, MLPerf Inference/Training/Tiny). Provides normative literary warrant (Z39.19 §5.3.5.1), not internal organizational preferred-term mandates (Z39.19 §5.3.5.2) unless term-form adoption is separately logged.

Cross-Stratum Scale Representation (Method-C §3.2, Method-B §3.2)

To prevent datacenter LLM serving from crowding out resource-constrained or classical ML systems, the frame requires representation across execution scales where applicable to an area:

• Datacenter / Cloud Scale (multi-node accelerator cluster serving, training, and fleet scheduling)
• Enterprise / Single-Node / Workstation (gateway/microservice orchestration, CPU/single-accelerator inference, retrieval and compound pipelines)
• Edge / On-Device / Embedded (TinyML) (MCU/NPU/mobile quantization, static arena memory planning, battery/thermal envelopes)

---

2. Global Provenance-Family Clustering (provenance_family_id)

Before stratification, split assignment, or warrant counting, every unit in <campaign>/sampling-manifest.json receives an explicit unit_id, provenance_family_id, stratum (S1–S5), split (dev, held_out_test, reserve), origin (external | internal), editor_affiliated (true | false), and source_type (prose | heading, with passage_kind: primary_excerpt | curated_summary on prose units and authors or institutional_author: true on external units) (Method-A §2.1 & §3.3, Method-B §2.2 & §3.4):

1. Shared Authorship, Institutional Organization, & Global work_id Lineage (compute_lineage_components in binder/cli/core/taxonomy_schema.py): Let A_i denote the normalized set of all listed personal authors of external source i (authors array in sampling-manifest.json and per-work work_rosters in exposed_cluster_metadata, mandatory on every non-institutional origin: external unit; institutional standards without personal author rosters set institutional_author: true and institutional_organization / institutional_organizations on prospective units, seeded across all 49 external canonical clusters in exposed_cluster_metadata in burned-clusters.json schema_version: 3, alongside a global 1 work_id <-> 1 canonical provenance_family_id bijection rejecting any work split across multiple effective clusters). Two external sources or clusters are linked using separated personal-author and institutional-organization denominators when personal author overlap satisfies |A_i \cap A_j| / \min(|A_i|, |A_j|) \ge 0.50 (across cluster rosters or any per-work work_rosters entry) or when their normalized institutional_organizations sets share any technical committee or standards body (|O_i \cap O_j| \ge 1; pf_iso42001_pas8800 and pf_iso_automotive_ul4600 union via shared ISO/TC 22/SC 32), and connected components (transitive closure computed identically by compute_lineage_components across reconcile.py, apply.py, taxonomy_schema.py, and build_study_receipts.py) form a single effective_lineage_component_id across all strata and areas.
2. Paper–System–Benchmark–Checklist Lineage, Course Disambiguation, & Prospective 1-Work-Per-Unit Rule: A research paper, its companion open-source documentation, tutorials derived from it, and any checklist row summarizing that same work share one provenance_family_id (cluster_merges in taxonomy/method-study/burned-clusters.json: pf_cov_courses_2026_AIE -> pf_huyen_dmls, pf_cov_courses_2026_C868 -> pf_cmu_llmsys, pf_cov_courses_2026_DMLS -> pf_huyen_dmls, pf_cov_courses_2026_HAR -> pf_internal_mlsysbook, pf_cov_courses_2026_M594 -> pf_mit_han_tinyml, pf_cov_courses_2026_MLS26 -> pf_cov_mlsys_25_26_mlsys2026, pf_cov_courses_2026_NIST -> pf_nist_ai_rmf, pf_cov_courses_2026_S329 -> pf_huyen_dmls, pf_cov_courses_2026_S336 -> pf_stanford_cs336, pf_cov_courses_2026_UT378 -> pf_ut_cs378). Note the explicit course-code disambiguations in burned-clusters.json (courses_2026_code_legend): S329 in courses-and-standards-2026.yml is Stanford CS 329S (Machine Learning Systems Design, Chip Huyen, merging into pf_huyen_dmls), distinct from pf_stanford_cs329t (Stanford CS 329T Trustworthy Machine Learning, Koyejo & Hashimoto); and B294 in courses-and-standards-2026.yml is UC Berkeley CS 294-162 (Machine Learning Systems, Stoica & Gonzalez, which author-unions with pf_vllm_sosp23 / pf_sglang_neurips24 / pf_slora_mlsys24), distinct from pf_berkeley_aisafety (UC Berkeley CS 294-267 AI Safety, Alignment, and Control, Steinhardt / Song / Hendrycks, which author-unions with pf_cov_courses_2026_BAG and pf_amodei_concrete_safety). Historical Phase 3 bundling vs. prospective rule: in the executed Phase 3 P4+Q4 curation (preregistration.yaml), four curated Q4 prose units bundled paired foundational works (Q4-S3-SIMPLEX-CBF-01 pairing Sha 2001 Simplex and Ames et al. 2019 Control Barrier Functions; Q4-S3-NN-VERIFICATION-01 pairing Katz et al. CAV 2017 Reluplex and Liu et al. verification survey; Q4-S3-RAND-DRIVING-01 pairing Kalra & Paddock RAND RR-1478 and Corso et al. JAIR 2021; Q4-S5-ISO26262-21448-UL4600), Q4-S1-CONCRETE-SAFETY-01 placed Amodei et al. 2016 in S1, and Q4-COV-courses-and-standards-2026-191 inherited S2 from courses-and-standards-2026.yml before deduplicating into pf_nist_ai_rmf (1 effective lineage either way). Prospectively across all new area frames, every manifest unit represents at most 1 primary work (primary_work_count == 1 enforced on non-historical units by reconcile.py) assigned to its intrinsic source stratum (S1–S5) rather than the host checklist file's stratum.
3. Project-Internal Corpora (pf_internal_mlsysbook) & Editor-Affiliated Consortium Exclusion (editor_affiliated: true):
   • Prospective single internal lineage (pf_internal_mlsysbook, origin: internal, split: dev): Prospectively across all refresh campaigns, all project-internal sources (MLSysBook chapters, Harvard CS249r course materials, StaffML diagnostic banks, and single-lab papers where the taxonomy Editor is a primary or senior author) collapse into a single internal lineage pf_internal_mlsysbook (origin: internal, restricted exclusively to split: dev; reconcile.py also mechanically unions all origin: internal units into a single _effective_lineage). Any number of origin: internal units supplies 0 external lineages under Gate 2 and cannot satisfy Gate 5's external verbatim-quote check.
   • Editor-affiliated multi-institutional standards (editor_affiliated: true): Multi-institutional consortium benchmark or standard specifications where the taxonomy Editor is a primary/lead author (pf_mlperf_inference [P4-S5-MLPERF-INF-01, Reddi et al. ISCA 2020, and P4-S5-MLPERF-TINY-01, Banbury et al. NeurIPS D&B 2021]) remain S5 normative specifications (origin: external) for domain context, but carry editor_affiliated: true in sampling-manifest.json and taxonomy/method-study/burned-clusters.json. reconcile.py excludes editor_affiliated: true clusters from satisfying Gate 2's independent external-lineage floor and Gate 5's external verbatim-attestation floor (MLPerf itself also remains excluded from concepts and aliases under Gate 3).
   • Historical Phase 3 internal sub-unit IDs & StaffML holdout trace: In the executed Phase 3 P4+Q4 study (preregistration.yaml), internal units carried per-chapter (pf_mlsysbook_<vol>/<chapter>) and per-sample (pf_staffml_heldout_*) IDs (0 raw provenance_family_id clusters crossed splits at Phase 3; under the full 10 work-level cluster_merges audited in Phase 7, 3 canonical clusters — pf_stanford_cs336 across dev/held_out_test and pf_cov_mlsys_25_26_mlsys2026 plus pf_ut_cs378 across dev/reserve — spanned checklist and curated splits), and internal units were excluded from external generalization claims:
     1. All 7 held-out MLSysBook chapter units were excluded from Phase 6 coverage (COV evaluated only 79 origin: external S2/S3/S5 units across 10 raw clusters / 9 deduplicated external lineages after merging pf_cov_courses_2026_NIST -> pf_nist_ai_rmf, where 5 of the 6 non-systems-venues-2026 external units were exposed to 0.4-draft design inputs [3 via historical dev course/author overlap and 2 via held_out: false courses-and-standards-2026.yml], with dev_disjoint_76_external [36.8% vs. 10.5%] and unexposed_73_systems_venues_2026 [38.4% vs. 11.0% expressibility; 19.2% vs. 2.7% exact, diff = -16.4 pp] sensitivity slices recorded in study-receipts-summary.json).
     2. Direct audit of the 40 S4 StaffML held_out_test indexing units (38 in taxonomy/agreement/staffml-300-heldout-2026-10.yml + 2 in staffml-300-2026-10.yml [edge-1247 (Q4-S4-SAMPLE1-Q-160) and tinyml-1536 (Q4-S4-SAMPLE1-Q-249)], originally tagged at 0.2-draft [714/727 concepts]) against staffml/vault/data/taxonomy-tagging-2026-10.yaml confirms that 0 / 40 (0/38 + 0/2) appeared in proposed_concepts, subject_only_questions, or adjudication, while 13 / 40 (12/38 + 1/2) appeared in review_queue of the Oct 2026 whole-bank (10,716-question) tagging pass. Accordingly, those 38 items are reported solely as a tagging-rule calibration holdout (tagging_rule_holdout_38_staffml_2026_10 in study-receipts-summary.json), whereas only external 2026 systems-venues-2026 units were unexposed to 0.4-draft construction. In post-Phase-6 campaigns (bucket-a-campaign/sampling-manifest.json) and prospective refreshes, all MLSysBook, CS249r, and StaffML units reside in dev.

Mechanical enforcement (reconcile.py schema_version: 4, stage_reviewer_workspace.py, run_reviews.sh, & burned-clusters.json schema_version: 3):

• reconcile.py unconditionally requires <campaign>/sampling-manifest.json on every campaign; rejects non-finite numbers (NaN/Infinity, 1e999 float overflow in JSON, and .nan/.inf/1e999 in YAML), duplicate JSON/YAML keys, invalid calendar dates (datetime.date.fromisoformat), or duplicate unit_ids; enforces origin == "internal" => split == "dev", burned-clusters.json (schema_version: 3, pinning 98 IDs = all 97 raw Phase 3 cluster IDs plus 1 canonical internal merge target pf_internal_mlsysbook, alongside the exact 382 historical_phase3_units including initial_phase3_split [200 dev, 126 held_out_test, 56 reserve], current_exposure_split [382 dev], passage_kind, and passage_sha256 plus complete exposed_cluster_metadata and work_rosters, verified append-only against origin/dev, dev, and HEAD locally and against origin/dev in GitHub Actions taxonomy-validate-dev.yml), editor_affiliated exclusion (including excluding editor-affiliated units from Stage 6 release-floor calculations), global 1 work_id <-> 1 canonical provenance_family_id bijection and separated < 50% personal-author (min(|A_1|, |A_2|)) plus disjoint institutional_organizations (|O_1 \cap O_2| == 0 for independence; >= 1 shared organization unions components) overlap closure (compute_lineage_components), scope_note (\ge 15 chars across all 4 alias types), and word-bounded verbatim_quote substring verification against cited non-editor-affiliated external dev units with source_type: prose, passage_kind: primary_excerpt, and symlink-free primary_source_excerpt_path confined under <campaign>/sources/ (verified by SHA-256 across all primary_excerpt units, recording primary_source_sha256 and primary_source_locator; contextual discovery leads such as held_out: false checklists and taxonomy-tagging-dev-leads.yaml never satisfy Gate 2 warrant_sources or Gate 5 checked_dev_units). stage_reviewer_workspace.py parses all repo/staged YAML and JSON with safe_load_strict / _json_loads_strict and UTF-8 errors="strict" (decompressing .gz archives during holdout leak scanning), verifies byte-for-byte freshness of taxonomy/downloads/* against deterministic compilation (including --candidate-dir in Path A audit mode), and stages an external /tmp read-only workspace across review, crosscheck, audit, and guides modes containing only held_out: false checklists, split == "dev" manifest units/chapters/excerpts (plus all 382 current_exposure_split == "dev" historical_phase3_units in burned-clusters-dev-projection.json), and Markdown docs with non_dev_tokens (reported by category in non_dev_leak_scan_tokens_by_category across non-dev manifest units, held_out: true checklists, and taxonomy/agreement/*.yml item and suite IDs) redacted, while run_reviews.sh wraps all three CLIs (codex, agy, claude) in separate macOS Seatbelt profiles (macos_seatbelt_all_3_families_deny_repo_campaign_peer_read_and_workspace_write with pre-flight probe verification denying file-read*/file-write* across the entire GitHub parent directory [covering the worktree, main repo, all sibling MLSysBook* checkouts, AIConfigs, and all other local repositories], campaign dir, /tmp/tax-cand*, ~/.claude/{CLAUDE.md,rules,protected-checkouts.txt,projects}, ~/.codex, and peer reviewer I/O directories, plus process-exec* deny on curl, gh, wget, scp, ssh, and nc; parent CLI processes retain outbound HTTPS solely for hosted LLM inference APIs under tool-level web/shell disablement while pre-publication holdouts stay off public GitHub).

---

3. Split Allocation Protocol and Post-Phase-6 Cluster-Burn State

1. Prospective Refresh Targets (60% / 25% / 15%):
   • In periodic refreshes (.claude/workflows/taxonomy-refresh.md, taxonomy/METHOD.md), global external provenance_family_id clusters are assigned once before construction with target cluster proportions of 60% dev (visible to builders alongside internal dev corpora), 25% held_out_test (sealed external clusters for Step 5b evaluation), and 15% reserve (sequestered external clusters for replenishing burned held_out_test slices alongside fresh post-cutoff external sources). Multi-area clusters are assigned as whole clusters prior to per-area tallies.
2. Executed Phase 3 Allocation in the Two-Area Study (preregistration.yaml, seed = 20261103):
   • In the completed Phase 3 P4+Q4 study (382 units across 97 pre-deduplication provenance_family_id clusters [95 deduplicated clusters under the 2 Phase 3 merges pf_cov_courses_2026_DMLS -> pf_huyen_dmls and pf_cov_courses_2026_NIST -> pf_nist_ai_rmf, and 88 deduplicated canonical clusters (97 raw Phase 3 cluster IDs + 1 canonical internal merge target pf_internal_mlsysbook = 98 recorded IDs in burned-clusters.json, minus 10 work-level cluster_merges = 88 canonical clusters: 39 internal + 49 external; 41 effective lineage components = 1 internal + 40 external after < 50% personal-author and institutional_organizations overlap transitive closure) under the full 10 work-level cluster_merges audited in Phase 7]; 0 raw cross-split clusters at Phase 3, 3 canonical cross-split clusters under the 10 merges [pf_stanford_cs336, pf_cov_mlsys_25_26_mlsys2026, pf_ut_cs378]), build_phase3_preregistration.py combined pre-designated benchmark splits (fixed_split) with a deterministic cell-wise draw (seed = 20261103) over movable (area, stratum) cells of size n (1 cluster to held_out_test when n \ge 3, 1 cluster to reserve when n \ge 4, and the remaining n - \mathbb{I}(n \ge 3) - \mathbb{I}(n \ge 4) clusters to dev).
   • Because single-passage curated sources each formed a 1-unit dev cluster while held_out_test and reserve held larger venue/question-bank clusters, the achieved Phase 3 raw cluster split was 76 / 97 (78.4%) dev, 15 / 97 (15.5%) held_out_test, and 6 / 97 (6.2%) reserve, and the achieved unit split was 200 / 382 (52.4%) dev, 126 / 382 (33.0%) held_out_test, and 56 / 382 (14.7%) reserve:
     • Raw clusters by split (97 total in Phase 3, after classifying pf_cov_courses_2026_HAR as internal): 76 dev (47 P4-only, 23 Q4-only, 6 P4/Q4-shared; 44 external, 32 internal), 15 held_out_test (8 P4-only, 6 Q4-only, 1 P4/Q4-shared; 10 external [9 deduplicated external lineages], 5 internal), and 6 reserve (3 P4-only, 3 Q4-only; 4 external, 2 internal).
     • Units by split, area, and origin (382 total in Phase 3, after classifying HAR as internal): dev = 200 (142 P4 [94 external, 48 internal], 58 Q4 [14 external, 44 internal]); held_out_test = 126 (116 P4 [75 external, 41 internal], 10 Q4 [4 external, 6 internal]); reserve = 56 (47 P4 [46 external, 1 internal], 9 Q4 [2 external, 7 internal]).
3. Post-Phase-6 Global Cluster-Burn State (taxonomy/method-study/burned-clusters.json & bucket-a-campaign/sampling-manifest.json):
   • Inspecting the 15 Phase 6 held_out_test clusters (126 units) during the 2026-11-02 comparison burned them globally, including all 4 venue clusters (osdi2026, nsdi2026, eurosys2026, asplos2026) of taxonomy/coverage/systems-venues-2026.yml (now marked held_out: false across the entire repository).
   • Furthermore, auditing all 6 Phase 6 reserve clusters showed none remained unexposed to 0.4-draft (Vocab-Y) or repository docs: MLSys 2026 (pf_cov_mlsys_25_26_mlsys2026, 45 units from mlsys-papers-2025-2026.yml), pf_cov_courses_2026_USC (1 unit from courses-and-standards-2026.yml), and 8 MLSysBook chapter units came from held_out: false checklists or internal MLSysBook chapters; pf_ut_cs378 (P4-S2-UT378-01) merged with pf_cov_courses_2026_UT378 in courses-and-standards-2026.yml (held_out: false); and pf_iso42001_pas8800 (Q4-S5-ISO42001-PAS8800) overlapped 0.4-draft design source iso42001 (ml-systems-taxonomy.yml) and was named in sampling-frame.md.
   • Applying the 10 work-level cluster_merges across the exact 382 pre-registered Phase 3 units (with zero post-registration unit additions) and recording all exposed/burned clusters in the append-only ledger taxonomy/method-study/burned-clusters.json (schema_version: 3, 98 recorded IDs in burned_or_exposed_cluster_ids: all 97 raw Phase 3 cluster IDs plus 1 canonical internal merge target pf_internal_mlsysbook, collapsing under the 10 merges to 88 deduplicated canonical clusters: 39 internal + 49 external) yields 382 units across 88 deduplicated canonical clusters (39 internal + 49 external; 41 effective lineage components = 1 internal + 40 external) in bucket-a-campaign/sampling-manifest.json: 382 dev units (88 deduplicated clusters), 0 held_out_test units, and 0 unexposed external reserve units (unexposed_external_reserve_clusters: []). Replenishing held_out_test and reserve for the next structural refresh therefore requires freshly curated, single-work post-cutoff external clusters (such as MLSys 2027, 2027 systems venues, and newly published standards), never mlsys-papers-2025-2026.yml, systems-venues-2026.yml, or courses-and-standards-2026.yml.

---

4. Executed Two-Area Sampling Frame (P4 and Q4 — Exact 382-Unit Phase 3 Inventory)

In Phase 3 (2026-11-02), the 382-unit manifest (random_seed: 20261103) was committed in taxonomy/method-study/preregistration.yaml (enumerating S1–S5 source pools for the remaining 18 taxonomy areas remains open pre-1.0 work). Every item listed below corresponds directly to executed units in phase4-dev-sources.json and phase6-heldout-slice.json:

Area 1: P4 (serving) — Deployment & serving (9 subjects, 95 0.4-draft concepts)

Subjects (9): P4.1 (serving-architecture) Serving architectures · P4.2 (compound-ai) Compound AI systems & model routing · P4.3 (latency-goodput) Latency, throughput & goodput · P4.4 (request-scheduling) Batching, admission & autoscaling · P4.5 (llm-inference) LLM & generative inference · P4.6 (structured-generation) Decoding, sampling & structured output · P4.7 (test-time-compute) Test-time compute · P4.8 (retrieval-rag) Retrieval & vector search · P4.9 (on-device-deployment) On-device & edge deployment.

• S1 (Textbooks & Monographs — 50 dev, 5 held_out_test, 1 reserve in Phase 3):
  • P4-S1-DMLS-01 & P4-S1-DMLS-02 (dev): Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — Ch. 7–8 (batch vs. online prediction, dynamic batching, compression/export).
  • P4-S1-WSC-01 (dev): Barroso, Hölzle, & Ranganathan, The Datacenter as a Computer (3rd ed., 2018; pre-2021 foundational monograph) — Ch. 2 & 6 (tail latency, hedging, admission control).
  • 47 dev, 5 held_out_test (vol1/13_model_serving, excluded from Phase 6 COV), and 1 reserve (vol1/06_nn_architectures) MLSysBook Vol. I–IV chapter units (origin: internal; all promoted to dev post-Phase 6).
• S2 (University Courses — 33 dev, 2 held_out_test, 1 reserve in Phase 3):
  • Curated syllabi: P4-S2-CMU11868-01 & P4-S2-CMU11868-02 (CMU 11-868 Large Language Model Systems, Spring 2025, dev), P4-S2-MIT65940-01 (MIT 6.5940 TinyML and Efficient Deep Learning Computing, Fall 2024, dev), P4-S2-STANFORD-CS336-01 (Stanford CS336 / CS229S, 2025, Phase 6 held_out_test, now burned to dev), and P4-S2-UT378-01 (UT Austin CS 378, Fall 2024, Phase 3 reserve, now burned to dev via pf_cov_courses_2026_UT378 in courses-and-standards-2026.yml).
  • Checklist topic units from taxonomy/coverage/courses-and-standards-2026.yml (30 dev, 1 held_out_test [P4-COV-courses-and-standards-2026-212, now burned to dev]).
• S3 (Peer-Reviewed Venues — 55 dev, 73 held_out_test, 45 reserve in Phase 3):
  • 6 curated dev papers: P4-S3-ORCA-2022 (Yu et al., Orca, OSDI 2022), P4-S3-PAGED-2023 (Kwon et al., PagedAttention, SOSP 2023), P4-S3-SPECDEC-2023 (Leviathan, Kalman, & Matias, Speculative Decoding, ICML 2023), P4-S3-SARATHI-2024 (Agrawal et al., Sarathi-Serve, OSDI 2024), P4-S3-DISTSERVE-2024 (Zhong et al., DistServe, OSDI 2024), and P4-S3-SGLANG-2024 (Zheng et al., SGLang, NeurIPS 2024), plus 49 P4 checklist dev units from mlsys-papers-2025-2026.yml (MLSys 2025) and systems-venues-2025.yml.
  • 73 Phase 6 held_out_test units: P4-S3-SLORA-2024 (Sheng et al., S-LoRA, MLSys 2024) plus 72 P4 paper units from systems-venues-2026.yml (OSDI, NSDI, EuroSys, ASPLOS 2026, now burned to dev).
  • 45 Phase 6 reserve units from mlsys-papers-2025-2026.yml (MLSys 2026 — unexposed to Vocab-X, already held_out: false in 0.4-draft, now promoted to dev).
• S4 (Practitioner Artifacts & Diagnostic Scenarios — 2 dev, 36 held_out_test, 0 reserve in Phase 3):
  • 2 curated external dev guides: P4-S4-ONNX-ORT-01 (ONNX Runtime / Safetensors / Torch.export packaging & execution providers) and P4-S4-K8S-SERVING-01 (Kubernetes / Ray Serve / Triton rollouts, draining & backpressure).
  • 36 StaffML P4 scenarios from staffml-300-heldout-2026-10.yml (origin: internal, Phase 6 IDX, now burned to dev).
• S5 (Standards & Benchmark Specifications — 2 dev, 0 held_out_test, 0 reserve in Phase 3):
  • P4-S5-MLPERF-INF-01 (dev, pf_mlperf_inference, editor_affiliated: true): Reddi et al., MLPerf Inference Benchmark (ISCA 2020) & MLPerf Inference v4.1 LoadGen scenario rules (flagged editor_affiliated: true so it cannot count toward Gate 2/Gate 5 external-lineage independence).
  • P4-S5-MLPERF-TINY-01 (dev, pf_mlperf_inference, editor_affiliated: true): Banbury et al., MLPerf Tiny Benchmark (NeurIPS Datasets & Benchmarks 2021; shares MLPerf consortium and Editor authorship with pf_mlperf_inference).

Area 2: Q4 (safety-assurance) — Safety & assurance (4 subjects, 30 0.4-draft concepts)

Subjects (4): Q4.1 (hazards-operating-domain) Hazard analysis & operating domains · Q4.2 (runtime-enforcement) Runtime assurance & safety filters · Q4.3 (verification-validation) Safety verification & validation · Q4.4 (safety-cases) Safety cases, standards & certification.

• S1 (Textbooks & Monographs — 46 dev, 2 held_out_test, 7 reserve in Phase 3):
  • Q4-S1-LEVESON-STPA-01 (dev): Leveson, Engineering a Safer World: Systems Thinking Applied to Safety (MIT Press, 2011; foundational STPA monograph).
  • Q4-S1-CONCRETE-SAFETY-01 (dev): Amodei et al., Concrete Problems in AI Safety (2016; curated under S1 in Phase 3; prospective curation assigns papers to S3).
  • 44 dev, 2 held_out_test (vol4/04_nervous, excluded from Phase 6 COV), and 7 reserve (vol4/16_release) MLSysBook Vol. III–IV chapter units (origin: internal; all promoted to dev post-Phase 6).
• S2 (University Courses — 5 dev, 2 held_out_test, 1 reserve in Phase 3):
  • Q4-S2-STANFORD-CS329T-01 (Stanford CS329T Trustworthy Machine Learning, dev) plus 4 dev checklist units from courses-and-standards-2026.yml; Q4-S2-BERKELEY-CS294-267-01 (UC Berkeley CS294-267 AI Safety, Alignment, and Control) plus 1 NIST checklist unit (Q4-COV-courses-and-standards-2026-191, merged into pf_nist_ai_rmf) in Phase 6 held_out_test; and 1 USC checklist unit (Q4-COV-courses-and-standards-2026-208) in Phase 6 reserve (all burned/0.4-exposed units promoted to dev post-Phase 6).
• S3 (Peer-Reviewed Venues — 5 dev, 1 held_out_test, 0 reserve in Phase 3):
  • 3 curated dev prose units (historically bundling paired works in Phase 3; prospective curation uses 1 work per unit): Q4-S3-SIMPLEX-CBF-01 (Sha 2001 Simplex Architecture & Ames et al. 2019 Control Barrier Functions), Q4-S3-NN-VERIFICATION-01 (Katz et al. CAV 2017 Reluplex & Liu et al. neural network verification survey), and Q4-S3-RAND-DRIVING-01 (Kalra & Paddock RAND RR-1478 & Corso et al. JAIR 2021 black-box safety validation / adaptive stress testing survey), plus 2 dev venue-checklist units (Q4-COV-mlsys-papers-2025-2026-028 [MLSys 2025] and Q4-COV-systems-venues-2025-032 [SOSP 2025]).
  • 1 Phase 6 held_out_test unit: Q4-COV-systems-venues-2026-186 (ASPLOS 2026, now burned to dev).
• S4 (Practitioner Artifacts & Diagnostic Scenarios — 1 dev, 4 held_out_test, 0 reserve in Phase 3):
  • Q4-S4-FRONTIER-RSP-01 (dev): Frontier safety & deployment governance frameworks (Responsible Scaling / Preparedness / Frontier Safety capability thresholds & safeguard levels).
  • 4 StaffML Q4 scenarios (2 from staffml-300-heldout-2026-10.yml and 2 from staffml-300-2026-10.yml, origin: internal, Phase 6 IDX, now burned to dev).
• S5 (Standards & Assurance Guidance — 1 dev, 1 held_out_test, 1 reserve in Phase 3):
  • Q4-S5-ISO26262-21448-UL4600 (dev): ISO 26262:2018 (Functional Safety), ISO 21448:2022 (SOTIF), and UL 4600 / Goal Structuring Notation (GSN).
  • Q4-S5-NIST-RMF-600-1 (Phase 6 held_out_test, now burned to dev): NIST AI 100-1 (AI RMF 1.0) & NIST AI 600-1 (Generative AI Profile, 2024).
  • Q4-S5-ISO42001-PAS8800 (Phase 3 reserve, now burned to dev via 0.4-draft design source iso42001 and sampling-frame.md): ISO/IEC 42001:2023 & ISO/PAS 8800:2024 (Road Vehicles — Safety and Artificial Intelligence).