cs249r_book/taxonomy/method-study/sampling-frame.md

# Enumerated Sampling Frame and Split Hygiene for the ML Systems Taxonomy

This document specifies the five-stratum sampling frame, provenance-family clustering rules, and split allocation protocol (`dev`, `held_out_test`, and `reserve`) synthesized from `Method-A` (§3), `Method-B` (§3), and audited `Method-C` (§3.2), together with the executed Phase 3 allocation (`taxonomy/method-study/preregistration.yaml`) and post-Phase-6 global cluster-burn ledger (`taxonomy/method-study/burned-clusters.json`) for `P4` and `Q4`. Public verification uses the committed files in `taxonomy/method-study/` (`preregistration.yaml`, `burned-clusters.json`, `genesis-0.4-baseline.json.gz`, `study-receipts-summary.json`, `rubric.yaml`) and `taxonomy/term-provenance.yml`; raw maintainer review workspaces under `.claude/_reviews/` remain local and unversioned under the repository's private AI-configuration policy.

*Note: Sampling-frame strata `S1`–`S5` denote source populations and are distinct from Stack area codes `S1`–`S4`.*

---

## 1. Five Stratified Source Populations (`S1`–`S5`)

Every candidate concept and every coverage or indexing evaluation unit is drawn from an enumerated frame partitioned into five source populations (`Method-A` §3.1, `Method-B` §3.1). Source stratum records **where an excerpt comes from**; warrant type under ANSI/NISO Z39.19 §5.3.5.1–§5.3.5.3 is recorded separately so operational proxies are not conflated with direct user search logs or organizational term-preference mandates:

| Stratum ID | Stratum Name | Unit of Sampling | Inclusion Criteria & Warrant Role |
| :--- | :--- | :--- | :--- |
| **`S1`** | **Textbooks & Reference Monographs** | Chapter / major section (`1` work per unit prospectively) | Published or open-access textbooks covering ML systems architecture, distributed training/serving, hardware acceleration, or safety/reliability engineering (both external monographs and project-internal `MLSysBook` chapters). Provides **literary & curriculum warrant** (`Z39.19 §5.3.5.1`; internal `MLSysBook` chapters carry `origin: internal`). |
| **`S2`** | **University Courses** | Course syllabus (lecture module + assignment/lab scope; `1` course per unit prospectively) | Graduate or upper-division undergraduate ML systems courses at accredited universities with public lecture schedules and reading lists, stratified across hardware/compilers, systems/networking, and safe/reliable deployment (`Method-A` §3.2). Provides **external literary & pedagogical warrant** (except Harvard `CS249r`, which shares `MLSysBook` authorship and carries `origin: internal`). |
| **`S3`** | **Peer-Reviewed Systems & ML Venues** | Paper (`1` paper per unit prospectively: title, abstract, architecture/evaluation passage, or blind venue topic entry) | Peer-reviewed papers from systems conferences (`MLSys`, `OSDI`, `SOSP`, `NSDI`, `ASPLOS`, `ISCA`, `EuroSys`, `CAV`, `AISec`) and systems/safety tracks of ML/NLP venues (`NeurIPS`, `ICML`, `ICLR`, `EMNLP`, `FAccT`, `SATML`) within the six-calendar-year study window (`2021`–`2026` inclusive at study freeze) plus pre-registered foundational papers (`>500` citations in Google Scholar / OpenAlex or standard `S2` syllabus adoption). Provides **external literary warrant**. |
| **`S4`** | **Practitioner Artifacts, Postmortems & Diagnostic Scenarios** | Engineering design doc, incident postmortem, runtime spec, or diagnostic scenario | Public production engineering design guides, open-source serving/runtime documentation (`Triton`, `Ray Serve`, `ONNX Runtime` — used solely to extract underlying mechanisms, never product concepts), frontier deployment governance policies, and `StaffML` diagnostic scenarios (`origin: internal`). Provides **operational literary warrant** and (for `StaffML` tag counts across the `10,716`-question bank) an **assignment-frequency proxy**, not direct user search/request warrant (`Z39.19 §5.3.5.3`). |
| **`S5`** | **Standards, Assurance Guidance & Benchmark Specifications** | Standard clause / assurance guidance module / benchmark scenario specification | Published technical standards, public assurance guidance reports, and benchmark specifications (`NIST AI 100-1`, `NIST AI 600-1`, `ISO 26262`, `ISO 21448`, `ISO/IEC 42001`, `ISO/PAS 8800`, `UL 4600`, `OWASP Top 10 for LLM`, `MITRE ATLAS`, `MLPerf Inference/Training/Tiny`). Provides **normative literary warrant** (`Z39.19 §5.3.5.1`), not internal organizational preferred-term mandates (`Z39.19 §5.3.5.2`) unless term-form adoption is separately logged. |

### Cross-Stratum Scale Representation (`Method-C` §3.2, `Method-B` §3.2)
To prevent datacenter LLM serving from crowding out resource-constrained or classical ML systems, the frame requires representation across execution scales where applicable to an area:
- **Datacenter / Cloud Scale** (multi-node accelerator cluster serving, training, and fleet scheduling)
- **Enterprise / Single-Node / Workstation** (gateway/microservice orchestration, CPU/single-accelerator inference, retrieval and compound pipelines)
- **Edge / On-Device / Embedded (`TinyML`)** (MCU/NPU/mobile quantization, static arena memory planning, battery/thermal envelopes)

---

## 2. Global Provenance-Family Clustering (`provenance_family_id`)

Before stratification, split assignment, or warrant counting, every unit in `<campaign>/sampling-manifest.json` receives an explicit `unit_id`, `provenance_family_id`, `stratum` (`S1`–`S5`), `split` (`dev`, `held_out_test`, `reserve`), `origin` (`external` | `internal`), `editor_affiliated` (`true` | `false`), and `source_type` (`prose` | `heading`, with `passage_kind: primary_excerpt | curated_summary` on prose units and `authors` or `institutional_author: true` on external units) (`Method-A` §2.1 & §3.3, `Method-B` §2.2 & §3.4):

1. **Shared Authorship, Institutional Organization, & Global `work_id` Lineage (`compute_lineage_components` in `binder/cli/core/taxonomy_schema.py`)**: Let $A_i$ denote the normalized set of **all listed personal authors** of external source $i$ (`authors` array in `sampling-manifest.json` and per-work `work_rosters` in `exposed_cluster_metadata`, mandatory on every non-institutional `origin: external` unit; institutional standards without personal author rosters set `institutional_author: true` and `institutional_organization` / `institutional_organizations` on prospective units, seeded across all `49` external canonical clusters in `exposed_cluster_metadata` in `burned-clusters.json` `schema_version: 3`, alongside a global `1 work_id <-> 1 canonical provenance_family_id` bijection rejecting any work split across multiple effective clusters). Two external sources or clusters are linked using **separated personal-author and institutional-organization denominators** when personal author overlap satisfies $|A_i \cap A_j| / \min(|A_i|, |A_j|) \ge 0.50$ (across cluster rosters or any per-work `work_rosters` entry) **or** when their normalized `institutional_organizations` sets share any technical committee or standards body ($|O_i \cap O_j| \ge 1$; `pf_iso42001_pas8800` and `pf_iso_automotive_ul4600` union via shared `ISO/TC 22/SC 32`), and connected components (transitive closure computed identically by `compute_lineage_components` across `reconcile.py`, `apply.py`, `taxonomy_schema.py`, and `build_study_receipts.py`) form a single `effective_lineage_component_id` across all strata and areas.
2. **Paper–System–Benchmark–Checklist Lineage, Course Disambiguation, & Prospective 1-Work-Per-Unit Rule**: A research paper, its companion open-source documentation, tutorials derived from it, and any checklist row summarizing that same work share one `provenance_family_id` (`cluster_merges` in `taxonomy/method-study/burned-clusters.json`: `pf_cov_courses_2026_AIE -> pf_huyen_dmls`, `pf_cov_courses_2026_C868 -> pf_cmu_llmsys`, `pf_cov_courses_2026_DMLS -> pf_huyen_dmls`, `pf_cov_courses_2026_HAR -> pf_internal_mlsysbook`, `pf_cov_courses_2026_M594 -> pf_mit_han_tinyml`, `pf_cov_courses_2026_MLS26 -> pf_cov_mlsys_25_26_mlsys2026`, `pf_cov_courses_2026_NIST -> pf_nist_ai_rmf`, `pf_cov_courses_2026_S329 -> pf_huyen_dmls`, `pf_cov_courses_2026_S336 -> pf_stanford_cs336`, `pf_cov_courses_2026_UT378 -> pf_ut_cs378`). Note the explicit course-code disambiguations in `burned-clusters.json` (`courses_2026_code_legend`): `S329` in `courses-and-standards-2026.yml` is Stanford `CS 329S` (*Machine Learning Systems Design*, Chip Huyen, merging into `pf_huyen_dmls`), distinct from `pf_stanford_cs329t` (Stanford `CS 329T` *Trustworthy Machine Learning*, Koyejo & Hashimoto); and `B294` in `courses-and-standards-2026.yml` is UC Berkeley `CS 294-162` (*Machine Learning Systems*, Stoica & Gonzalez, which author-unions with `pf_vllm_sosp23` / `pf_sglang_neurips24` / `pf_slora_mlsys24`), distinct from `pf_berkeley_aisafety` (UC Berkeley `CS 294-267` *AI Safety, Alignment, and Control*, Steinhardt / Song / Hendrycks, which author-unions with `pf_cov_courses_2026_BAG` and `pf_amodei_concrete_safety`). **Historical Phase 3 bundling vs. prospective rule:** in the executed Phase 3 `P4`+`Q4` curation (`preregistration.yaml`), four curated `Q4` prose units bundled paired foundational works (`Q4-S3-SIMPLEX-CBF-01` pairing Sha 2001 *Simplex* and Ames et al. 2019 *Control Barrier Functions*; `Q4-S3-NN-VERIFICATION-01` pairing Katz et al. CAV 2017 *Reluplex* and Liu et al. verification survey; `Q4-S3-RAND-DRIVING-01` pairing Kalra & Paddock RAND RR-1478 and Corso et al. *JAIR* 2021; `Q4-S5-ISO26262-21448-UL4600`), `Q4-S1-CONCRETE-SAFETY-01` placed Amodei et al. 2016 in `S1`, and `Q4-COV-courses-and-standards-2026-191` inherited `S2` from `courses-and-standards-2026.yml` before deduplicating into `pf_nist_ai_rmf` (`1` effective lineage either way). **Prospectively across all new area frames**, every manifest unit represents **at most `1` primary work** (`primary_work_count == 1` enforced on non-historical units by `reconcile.py`) assigned to its intrinsic source stratum (`S1`–`S5`) rather than the host checklist file's stratum.
3. **Project-Internal Corpora (`pf_internal_mlsysbook`) & Editor-Affiliated Consortium Exclusion (`editor_affiliated: true`)**:
   - **Prospective single internal lineage (`pf_internal_mlsysbook`, `origin: internal`, `split: dev`)**: Prospectively across all refresh campaigns, all project-internal sources (`MLSysBook` chapters, `Harvard CS249r` course materials, `StaffML` diagnostic banks, and single-lab papers where the taxonomy Editor is a primary or senior author) collapse into a **single internal lineage `pf_internal_mlsysbook`** (`origin: internal`, restricted exclusively to `split: dev`; `reconcile.py` also mechanically unions all `origin: internal` units into a single `_effective_lineage`). Any number of `origin: internal` units supplies `0` external lineages under Gate 2 and cannot satisfy Gate 5's external verbatim-quote check.
   - **Editor-affiliated multi-institutional standards (`editor_affiliated: true`)**: Multi-institutional consortium benchmark or standard specifications where the taxonomy Editor is a primary/lead author (`pf_mlperf_inference` [`P4-S5-MLPERF-INF-01`, Reddi et al. ISCA 2020, and `P4-S5-MLPERF-TINY-01`, Banbury et al. NeurIPS D&B 2021]) remain `S5` normative specifications (`origin: external`) for domain context, but carry **`editor_affiliated: true`** in `sampling-manifest.json` and `taxonomy/method-study/burned-clusters.json`. `reconcile.py` excludes `editor_affiliated: true` clusters from satisfying Gate 2's independent external-lineage floor and Gate 5's external verbatim-attestation floor (`MLPerf` itself also remains excluded from concepts and aliases under Gate 3).
   - **Historical Phase 3 internal sub-unit IDs & StaffML holdout trace**: In the executed Phase 3 `P4`+`Q4` study (`preregistration.yaml`), internal units carried per-chapter (`pf_mlsysbook_<vol>/<chapter>`) and per-sample (`pf_staffml_heldout_*`) IDs (`0` raw `provenance_family_id` clusters crossed splits at Phase 3; under the full `10` work-level `cluster_merges` audited in Phase 7, `3` canonical clusters — `pf_stanford_cs336` across `dev`/`held_out_test` and `pf_cov_mlsys_25_26_mlsys2026` plus `pf_ut_cs378` across `dev`/`reserve` — spanned checklist and curated splits), and internal units were **excluded from external generalization claims**:
     1. All `7` held-out `MLSysBook` chapter units were excluded from Phase 6 coverage (`COV` evaluated only `79` `origin: external` `S2`/`S3`/`S5` units across `10` raw clusters / `9` deduplicated external lineages after merging `pf_cov_courses_2026_NIST -> pf_nist_ai_rmf`, where `5` of the `6` non-`systems-venues-2026` external units were exposed to `0.4-draft` design inputs [`3` via historical `dev` course/author overlap and `2` via `held_out: false` `courses-and-standards-2026.yml`], with `dev_disjoint_76_external` [`36.8%` vs. `10.5%`] and `unexposed_73_systems_venues_2026` [`38.4%` vs. `11.0%` expressibility; `19.2%` vs. `2.7%` exact, `diff = -16.4` pp] sensitivity slices recorded in `study-receipts-summary.json`).
     2. Direct audit of the `40` `S4` `StaffML` `held_out_test` indexing units (`38` in `taxonomy/agreement/staffml-300-heldout-2026-10.yml` + `2` in `staffml-300-2026-10.yml` [`edge-1247` (`Q4-S4-SAMPLE1-Q-160`) and `tinyml-1536` (`Q4-S4-SAMPLE1-Q-249`)], originally tagged at `0.2-draft` [`714`/`727` concepts]) against `staffml/vault/data/taxonomy-tagging-2026-10.yaml` confirms that **`0 / 40` (`0/38` + `0/2`) appeared in `proposed_concepts`, `subject_only_questions`, or `adjudication`**, while **`13 / 40` (`12/38` + `1/2`) appeared in `review_queue`** of the Oct 2026 whole-bank (`10,716`-question) tagging pass. Accordingly, those `38` items are reported solely as a **tagging-rule calibration holdout** (`tagging_rule_holdout_38_staffml_2026_10` in `study-receipts-summary.json`), whereas **only external 2026 `systems-venues-2026` units were unexposed to `0.4-draft` construction**. In post-Phase-6 campaigns (`bucket-a-campaign/sampling-manifest.json`) and prospective refreshes, all `MLSysBook`, `CS249r`, and `StaffML` units reside in `dev`.

**Mechanical enforcement (`reconcile.py` `schema_version: 4`, `stage_reviewer_workspace.py`, `run_reviews.sh`, & `burned-clusters.json` `schema_version: 3`)**:
- `reconcile.py` unconditionally requires `<campaign>/sampling-manifest.json` on every campaign; rejects non-finite numbers (`NaN`/`Infinity`, `1e999` float overflow in JSON, and `.nan`/`.inf`/`1e999` in YAML), duplicate JSON/YAML keys, invalid calendar dates (`datetime.date.fromisoformat`), or duplicate `unit_id`s; enforces `origin == "internal" => split == "dev"`, `burned-clusters.json` (`schema_version: 3`, pinning `98` IDs = all `97` raw Phase 3 cluster IDs plus `1` canonical internal merge target `pf_internal_mlsysbook`, alongside the exact `382` `historical_phase3_units` including `initial_phase3_split` [`200` `dev`, `126` `held_out_test`, `56` `reserve`], `current_exposure_split` [`382` `dev`], `passage_kind`, and `passage_sha256` plus complete `exposed_cluster_metadata` and `work_rosters`, verified append-only against `origin/dev`, `dev`, and `HEAD` locally and against `origin/dev` in GitHub Actions `taxonomy-validate-dev.yml`), `editor_affiliated` exclusion (including excluding editor-affiliated units from Stage 6 release-floor calculations), global `1 work_id <-> 1 canonical provenance_family_id` bijection and separated `< 50%` personal-author (`min(|A_1|, |A_2|)`) plus disjoint `institutional_organizations` (`|O_1 \cap O_2| == 0` for independence; `>= 1` shared organization unions components) overlap closure (`compute_lineage_components`), `scope_note` ($\ge 15$ chars across all 4 alias types), and word-bounded `verbatim_quote` substring verification against cited non-editor-affiliated external `dev` units with `source_type: prose`, `passage_kind: primary_excerpt`, and symlink-free `primary_source_excerpt_path` confined under `<campaign>/sources/` (verified by SHA-256 across all `primary_excerpt` units, recording `primary_source_sha256` and `primary_source_locator`; contextual discovery leads such as `held_out: false` checklists and `taxonomy-tagging-dev-leads.yaml` never satisfy Gate 2 `warrant_sources` or Gate 5 `checked_dev_units`). `stage_reviewer_workspace.py` parses all repo/staged YAML and JSON with `safe_load_strict` / `_json_loads_strict` and UTF-8 `errors="strict"` (decompressing `.gz` archives during holdout leak scanning), verifies byte-for-byte freshness of `taxonomy/downloads/*` against deterministic compilation (including `--candidate-dir` in Path A `audit` mode), and stages an external `/tmp` read-only workspace across `review`, `crosscheck`, `audit`, and `guides` modes containing only `held_out: false` checklists, `split == "dev"` manifest units/chapters/excerpts (plus all `382` `current_exposure_split == "dev"` `historical_phase3_units` in `burned-clusters-dev-projection.json`), and Markdown docs with `non_dev_tokens` (reported by category in `non_dev_leak_scan_tokens_by_category` across non-`dev` manifest units, `held_out: true` checklists, and `taxonomy/agreement/*.yml` item and suite IDs) redacted, while `run_reviews.sh` wraps all three CLIs (`codex`, `agy`, `claude`) in separate macOS Seatbelt profiles (`macos_seatbelt_all_3_families_deny_repo_campaign_peer_read_and_workspace_write` with pre-flight probe verification denying `file-read*`/`file-write*` across the entire GitHub parent directory [covering the worktree, main repo, all sibling `MLSysBook*` checkouts, `AIConfigs`, and all other local repositories], campaign dir, `/tmp/tax-cand*`, `~/.claude/{CLAUDE.md,rules,protected-checkouts.txt,projects}`, `~/.codex`, and peer reviewer I/O directories, plus `process-exec*` deny on `curl`, `gh`, `wget`, `scp`, `ssh`, and `nc`; parent CLI processes retain outbound HTTPS solely for hosted LLM inference APIs under tool-level web/shell disablement while pre-publication holdouts stay off public GitHub).

---

## 3. Split Allocation Protocol and Post-Phase-6 Cluster-Burn State

1. **Prospective Refresh Targets (`60%` / `25%` / `15%`)**:
   - In periodic refreshes (`.claude/workflows/taxonomy-refresh.md`, `taxonomy/METHOD.md`), global external `provenance_family_id` clusters are assigned once before construction with target cluster proportions of **`60%` `dev`** (visible to builders alongside internal `dev` corpora), **`25%` `held_out_test`** (sealed external clusters for Step 5b evaluation), and **`15%` `reserve`** (sequestered external clusters for replenishing burned `held_out_test` slices alongside fresh post-cutoff external sources). Multi-area clusters are assigned as whole clusters prior to per-area tallies.
2. **Executed Phase 3 Allocation in the Two-Area Study (`preregistration.yaml`, `seed = 20261103`)**:
   - In the completed Phase 3 `P4`+`Q4` study (`382` units across `97` pre-deduplication `provenance_family_id` clusters [`95` deduplicated clusters under the `2` Phase 3 merges `pf_cov_courses_2026_DMLS -> pf_huyen_dmls` and `pf_cov_courses_2026_NIST -> pf_nist_ai_rmf`, and **`88` deduplicated canonical clusters** (`97` raw Phase 3 cluster IDs + `1` canonical internal merge target `pf_internal_mlsysbook` = `98` recorded IDs in `burned-clusters.json`, minus `10` work-level `cluster_merges` = `88` canonical clusters: `39` internal + `49` external; **`41` effective lineage components = `1` internal + `40` external** after `< 50%` personal-author and `institutional_organizations` overlap transitive closure) under the full `10` work-level `cluster_merges` audited in Phase 7]; `0` raw cross-split clusters at Phase 3, `3` canonical cross-split clusters under the `10` merges [`pf_stanford_cs336`, `pf_cov_mlsys_25_26_mlsys2026`, `pf_ut_cs378`]), `build_phase3_preregistration.py` combined pre-designated benchmark splits (`fixed_split`) with a deterministic cell-wise draw (`seed = 20261103`) over movable `(area, stratum)` cells of size $n$ (`1` cluster to `held_out_test` when $n \ge 3$, `1` cluster to `reserve` when $n \ge 4$, and the remaining $n - \mathbb{I}(n \ge 3) - \mathbb{I}(n \ge 4)$ clusters to `dev`).
   - Because single-passage curated sources each formed a 1-unit `dev` cluster while `held_out_test` and `reserve` held larger venue/question-bank clusters, the **achieved Phase 3 raw cluster split was `76 / 97` (`78.4%`) `dev`, `15 / 97` (`15.5%`) `held_out_test`, and `6 / 97` (`6.2%`) `reserve`**, and the **achieved unit split was `200 / 382` (`52.4%`) `dev`, `126 / 382` (`33.0%`) `held_out_test`, and `56 / 382` (`14.7%`) `reserve`**:
     - **Raw clusters by split (`97` total in Phase 3, after classifying `pf_cov_courses_2026_HAR` as internal)**: `76` `dev` (`47` `P4`-only, `23` `Q4`-only, `6` `P4`/`Q4`-shared; `44` `external`, `32` `internal`), `15` `held_out_test` (`8` `P4`-only, `6` `Q4`-only, `1` `P4`/`Q4`-shared; `10` `external` [`9` deduplicated external lineages], `5` `internal`), and `6` `reserve` (`3` `P4`-only, `3` `Q4`-only; `4` `external`, `2` `internal`).
     - **Units by split, area, and origin (`382` total in Phase 3, after classifying `HAR` as internal)**: `dev` = `200` (`142` `P4` [`94` `external`, `48` `internal`], `58` `Q4` [`14` `external`, `44` `internal`]); `held_out_test` = `126` (`116` `P4` [`75` `external`, `41` `internal`], `10` `Q4` [`4` `external`, `6` `internal`]); `reserve` = `56` (`47` `P4` [`46` `external`, `1` `internal`], `9` `Q4` [`2` `external`, `7` `internal`]).
3. **Post-Phase-6 Global Cluster-Burn State (`taxonomy/method-study/burned-clusters.json` & `bucket-a-campaign/sampling-manifest.json`)**:
   - Inspecting the `15` Phase 6 `held_out_test` clusters (`126` units) during the `2026-11-02` comparison burned them globally, including **all `4` venue clusters (`osdi2026`, `nsdi2026`, `eurosys2026`, `asplos2026`) of `taxonomy/coverage/systems-venues-2026.yml`** (now marked `held_out: false` across the entire repository).
   - Furthermore, auditing all `6` Phase 6 `reserve` clusters showed none remained unexposed to `0.4-draft` (`Vocab-Y`) or repository docs: `MLSys 2026` (`pf_cov_mlsys_25_26_mlsys2026`, `45` units from `mlsys-papers-2025-2026.yml`), `pf_cov_courses_2026_USC` (`1` unit from `courses-and-standards-2026.yml`), and `8` `MLSysBook` chapter units came from `held_out: false` checklists or internal `MLSysBook` chapters; `pf_ut_cs378` (`P4-S2-UT378-01`) merged with `pf_cov_courses_2026_UT378` in `courses-and-standards-2026.yml` (`held_out: false`); and `pf_iso42001_pas8800` (`Q4-S5-ISO42001-PAS8800`) overlapped `0.4-draft` design source `iso42001` (`ml-systems-taxonomy.yml`) and was named in `sampling-frame.md`.
   - Applying the `10` work-level `cluster_merges` across the **exact `382` pre-registered Phase 3 units** (with zero post-registration unit additions) and recording all exposed/burned clusters in the append-only ledger `taxonomy/method-study/burned-clusters.json` (`schema_version: 3`, `98` recorded IDs in `burned_or_exposed_cluster_ids`: all `97` raw Phase 3 cluster IDs plus `1` canonical internal merge target `pf_internal_mlsysbook`, collapsing under the `10` merges to **`88` deduplicated canonical clusters: `39` internal + `49` external**) yields **`382` units across `88` deduplicated canonical clusters (`39` internal + `49` external; `41` effective lineage components = `1` internal + `40` external) in `bucket-a-campaign/sampling-manifest.json`: `382` `dev` units (`88` deduplicated clusters), `0` `held_out_test` units, and `0` unexposed external `reserve` units (`unexposed_external_reserve_clusters: []`)**. Replenishing `held_out_test` and `reserve` for the next structural refresh therefore requires freshly curated, single-work post-cutoff external clusters (such as `MLSys 2027`, `2027` systems venues, and newly published standards), never `mlsys-papers-2025-2026.yml`, `systems-venues-2026.yml`, or `courses-and-standards-2026.yml`.

---

## 4. Executed Two-Area Sampling Frame (`P4` and `Q4` — Exact `382`-Unit Phase 3 Inventory)

In Phase 3 (`2026-11-02`), the `382`-unit manifest (`random_seed: 20261103`) was committed in `taxonomy/method-study/preregistration.yaml` (enumerating `S1`–`S5` source pools for the remaining `18` taxonomy areas remains open pre-`1.0` work). Every item listed below corresponds directly to executed units in `phase4-dev-sources.json` and `phase6-heldout-slice.json`:

### Area 1: `P4` (`serving`) — Deployment & serving (`9` subjects, `95` `0.4-draft` concepts)
Subjects (`9`): `P4.1` (`serving-architecture`) Serving architectures · `P4.2` (`compound-ai`) Compound AI systems & model routing · `P4.3` (`latency-goodput`) Latency, throughput & goodput · `P4.4` (`request-scheduling`) Batching, admission & autoscaling · `P4.5` (`llm-inference`) LLM & generative inference · `P4.6` (`structured-generation`) Decoding, sampling & structured output · `P4.7` (`test-time-compute`) Test-time compute · `P4.8` (`retrieval-rag`) Retrieval & vector search · `P4.9` (`on-device-deployment`) On-device & edge deployment.

- **`S1` (Textbooks & Monographs — `50` `dev`, `5` `held_out_test`, `1` `reserve` in Phase 3)**:
  - `P4-S1-DMLS-01` & `P4-S1-DMLS-02` (`dev`): Huyen, *Designing Machine Learning Systems* (O'Reilly, 2022) — Ch. 7–8 (batch vs. online prediction, dynamic batching, compression/export).
  - `P4-S1-WSC-01` (`dev`): Barroso, Hölzle, & Ranganathan, *The Datacenter as a Computer* (3rd ed., 2018; pre-2021 foundational monograph) — Ch. 2 & 6 (tail latency, hedging, admission control).
  - `47` `dev`, `5` `held_out_test` (`vol1/13_model_serving`, excluded from Phase 6 `COV`), and `1` `reserve` (`vol1/06_nn_architectures`) `MLSysBook` Vol. I–IV chapter units (`origin: internal`; all promoted to `dev` post-Phase 6).
- **`S2` (University Courses — `33` `dev`, `2` `held_out_test`, `1` `reserve` in Phase 3)**:
  - Curated syllabi: `P4-S2-CMU11868-01` & `P4-S2-CMU11868-02` (CMU `11-868` *Large Language Model Systems*, Spring 2025, `dev`), `P4-S2-MIT65940-01` (MIT `6.5940` *TinyML and Efficient Deep Learning Computing*, Fall 2024, `dev`), `P4-S2-STANFORD-CS336-01` (Stanford `CS336` / `CS229S`, 2025, Phase 6 `held_out_test`, now burned to `dev`), and `P4-S2-UT378-01` (UT Austin `CS 378`, Fall 2024, Phase 3 `reserve`, now burned to `dev` via `pf_cov_courses_2026_UT378` in `courses-and-standards-2026.yml`).
  - Checklist topic units from `taxonomy/coverage/courses-and-standards-2026.yml` (`30` `dev`, `1` `held_out_test` [`P4-COV-courses-and-standards-2026-212`, now burned to `dev`]).
- **`S3` (Peer-Reviewed Venues — `55` `dev`, `73` `held_out_test`, `45` `reserve` in Phase 3)**:
  - `6` curated `dev` papers: `P4-S3-ORCA-2022` (Yu et al., *Orca*, OSDI 2022), `P4-S3-PAGED-2023` (Kwon et al., *PagedAttention*, SOSP 2023), `P4-S3-SPECDEC-2023` (Leviathan, Kalman, & Matias, *Speculative Decoding*, ICML 2023), `P4-S3-SARATHI-2024` (Agrawal et al., *Sarathi-Serve*, OSDI 2024), `P4-S3-DISTSERVE-2024` (Zhong et al., *DistServe*, OSDI 2024), and `P4-S3-SGLANG-2024` (Zheng et al., *SGLang*, NeurIPS 2024), plus `49` `P4` checklist `dev` units from `mlsys-papers-2025-2026.yml` (`MLSys 2025`) and `systems-venues-2025.yml`.
  - `73` Phase 6 `held_out_test` units: `P4-S3-SLORA-2024` (Sheng et al., *S-LoRA*, MLSys 2024) plus `72` `P4` paper units from `systems-venues-2026.yml` (`OSDI`, `NSDI`, `EuroSys`, `ASPLOS 2026`, now burned to `dev`).
  - `45` Phase 6 `reserve` units from `mlsys-papers-2025-2026.yml` (`MLSys 2026` — unexposed to `Vocab-X`, already `held_out: false` in `0.4-draft`, now promoted to `dev`).
- **`S4` (Practitioner Artifacts & Diagnostic Scenarios — `2` `dev`, `36` `held_out_test`, `0` `reserve` in Phase 3)**:
  - `2` curated external `dev` guides: `P4-S4-ONNX-ORT-01` (`ONNX Runtime` / `Safetensors` / `Torch.export` packaging & execution providers) and `P4-S4-K8S-SERVING-01` (Kubernetes / `Ray Serve` / `Triton` rollouts, draining & backpressure).
  - `36` `StaffML` `P4` scenarios from `staffml-300-heldout-2026-10.yml` (`origin: internal`, Phase 6 `IDX`, now burned to `dev`).
- **`S5` (Standards & Benchmark Specifications — `2` `dev`, `0` `held_out_test`, `0` `reserve` in Phase 3)**:
  - `P4-S5-MLPERF-INF-01` (`dev`, `pf_mlperf_inference`, `editor_affiliated: true`): Reddi et al., *MLPerf Inference Benchmark* (ISCA 2020) & MLPerf Inference v4.1 LoadGen scenario rules (flagged `editor_affiliated: true` so it cannot count toward Gate 2/Gate 5 external-lineage independence).
  - `P4-S5-MLPERF-TINY-01` (`dev`, `pf_mlperf_inference`, `editor_affiliated: true`): Banbury et al., *MLPerf Tiny Benchmark* (NeurIPS Datasets & Benchmarks 2021; shares MLPerf consortium and Editor authorship with `pf_mlperf_inference`).

### Area 2: `Q4` (`safety-assurance`) — Safety & assurance (`4` subjects, `30` `0.4-draft` concepts)
Subjects (`4`): `Q4.1` (`hazards-operating-domain`) Hazard analysis & operating domains · `Q4.2` (`runtime-enforcement`) Runtime assurance & safety filters · `Q4.3` (`verification-validation`) Safety verification & validation · `Q4.4` (`safety-cases`) Safety cases, standards & certification.

- **`S1` (Textbooks & Monographs — `46` `dev`, `2` `held_out_test`, `7` `reserve` in Phase 3)**:
  - `Q4-S1-LEVESON-STPA-01` (`dev`): Leveson, *Engineering a Safer World: Systems Thinking Applied to Safety* (MIT Press, 2011; foundational STPA monograph).
  - `Q4-S1-CONCRETE-SAFETY-01` (`dev`): Amodei et al., *Concrete Problems in AI Safety* (2016; curated under `S1` in Phase 3; prospective curation assigns papers to `S3`).
  - `44` `dev`, `2` `held_out_test` (`vol4/04_nervous`, excluded from Phase 6 `COV`), and `7` `reserve` (`vol4/16_release`) `MLSysBook` Vol. III–IV chapter units (`origin: internal`; all promoted to `dev` post-Phase 6).
- **`S2` (University Courses — `5` `dev`, `2` `held_out_test`, `1` `reserve` in Phase 3)**:
  - `Q4-S2-STANFORD-CS329T-01` (Stanford `CS329T` *Trustworthy Machine Learning*, `dev`) plus `4` `dev` checklist units from `courses-and-standards-2026.yml`; `Q4-S2-BERKELEY-CS294-267-01` (UC Berkeley `CS294-267` *AI Safety, Alignment, and Control*) plus `1` NIST checklist unit (`Q4-COV-courses-and-standards-2026-191`, merged into `pf_nist_ai_rmf`) in Phase 6 `held_out_test`; and `1` USC checklist unit (`Q4-COV-courses-and-standards-2026-208`) in Phase 6 `reserve` (all burned/`0.4`-exposed units promoted to `dev` post-Phase 6).
- **`S3` (Peer-Reviewed Venues — `5` `dev`, `1` `held_out_test`, `0` `reserve` in Phase 3)**:
  - `3` curated `dev` prose units (historically bundling paired works in Phase 3; prospective curation uses `1` work per unit): `Q4-S3-SIMPLEX-CBF-01` (Sha 2001 *Simplex Architecture* & Ames et al. 2019 *Control Barrier Functions*), `Q4-S3-NN-VERIFICATION-01` (Katz et al. CAV 2017 *Reluplex* & Liu et al. neural network verification survey), and `Q4-S3-RAND-DRIVING-01` (Kalra & Paddock RAND RR-1478 & Corso et al. *JAIR* 2021 black-box safety validation / adaptive stress testing survey), plus `2` `dev` venue-checklist units (`Q4-COV-mlsys-papers-2025-2026-028` [`MLSys 2025`] and `Q4-COV-systems-venues-2025-032` [`SOSP 2025`]).
  - `1` Phase 6 `held_out_test` unit: `Q4-COV-systems-venues-2026-186` (`ASPLOS 2026`, now burned to `dev`).
- **`S4` (Practitioner Artifacts & Diagnostic Scenarios — `1` `dev`, `4` `held_out_test`, `0` `reserve` in Phase 3)**:
  - `Q4-S4-FRONTIER-RSP-01` (`dev`): Frontier safety & deployment governance frameworks (Responsible Scaling / Preparedness / Frontier Safety capability thresholds & safeguard levels).
  - `4` `StaffML` `Q4` scenarios (`2` from `staffml-300-heldout-2026-10.yml` and `2` from `staffml-300-2026-10.yml`, `origin: internal`, Phase 6 `IDX`, now burned to `dev`).
- **`S5` (Standards & Assurance Guidance — `1` `dev`, `1` `held_out_test`, `1` `reserve` in Phase 3)**:
  - `Q4-S5-ISO26262-21448-UL4600` (`dev`): **ISO 26262:2018** (*Functional Safety*), **ISO 21448:2022** (*SOTIF*), and **UL 4600** / Goal Structuring Notation (`GSN`).
  - `Q4-S5-NIST-RMF-600-1` (Phase 6 `held_out_test`, now burned to `dev`): **NIST AI 100-1** (*AI RMF 1.0*) & **NIST AI 600-1** (*Generative AI Profile*, 2024).
  - `Q4-S5-ISO42001-PAS8800` (Phase 3 `reserve`, now burned to `dev` via `0.4-draft` design source `iso42001` and `sampling-frame.md`): **ISO/IEC 42001:2023** & **ISO/PAS 8800:2024** (*Road Vehicles — Safety and Artificial Intelligence*).