github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-11-09 10:39:41 -06:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity
Files
dev
cs249r_book/taxonomy/method-study/synthesized-method.md
T

35 KiB
Raw Permalink Blame History

Synthesized Candidate Method for the ML Systems Taxonomy

Authoritative enactment status (2026-11-02): taxonomy/METHOD.md and the refresh workflow (.claude/workflows/taxonomy-refresh.md, enforced mechanically by reconcile.py, apply.py, guides_apply.py, stage_reviewer_workspace.py, and ./binder/binder concepts check) constitute the authoritative enacted specification for 0.4-draft and periodic refreshes. This document records how the top-ranked mechanisms from the blind three-family method competition (Method-A by gpt-6-astra, Method-B by claude-opus-5-5, and audited scale-stratification elements of Method-C by gemini-4-argon-high) were synthesized into that enacted protocol, and flags prospective 1.0 schema/governance items where noted.

Every mechanism below cites the blind proposal from which it was drawn, its exact computed cross-family mean score (dim_means_cross, with all-judge mean dim_means_all in parentheses from study-receipts-summary.json) on rubric dimensions D1–D8 (taxonomy/method-study/rubric.yaml), and whether each quantitative rule comes from a published standard or a project-calibrated parameter ([P]). Citation provenance note: ANSI/NISO Z39.19-2005 (R2010) and W3C SKOS (2009) are publicly verifiable; ISO 25964-1:2011 / 25964-2:2013 clause references were screened from model reference knowledge without opening the paywalled ISO text (METHOD.md).

1. End-to-End Construction and Refresh Pipeline (Stage 1–Stage 10)

Source provenance: Synthesized from Method-B §1 (14-step DAG with explicit input/output artifacts, stopping rules, and failure recovery; D1 cross-family mean 4.75/5.0, all-judge 4.667/5.0) and Method-A §1 (8-stage pipeline with split discipline and pre-registration gates; D1 cross-family mean 4.00/5.0, all-judge 4.333/5.0). Stage identifiers use Stage 1–Stage 10 to avoid colliding with Stack area codes (S1–S4) or sampling-frame strata (S1–S5).

Stage Name Inputs Enacted Outputs / Artifacts Stopping Condition & Recovery
Stage 1 Charter, Scope & Stratified Split Allocation Domain boundary, enumerated 5-stratum source frame (S1–S5), burned-clusters.json taxonomy/METHOD.md, <campaign>/sampling-manifest.json (dev, held_out_test, reserve by provenance_family_id) Scope exclusions fixed; reconcile.py and stage_reviewer_workspace.py fail closed on cross-split clusters, internal non-dev splits, or burned-cluster reuse (Method-A §3.3).
Stage 2 Cross-Area Lookup & Candidate Discovery External /tmp staged dev workspace (stage_reviewer_workspace.py --mode review) codex-r1.yaml, gemini-r1.yaml, claude-r1.yaml with dev locators (workspace-isolation-receipt-review.json) Incremental refresh (METHOD.md Gate 1, Z39.19 §11.1.1): search all 20 areas in mlsys-taxonomy-concepts.csv before proposing new concepts; zero held_out: true or non-dev items in /tmp workspace. (Blank-slate rebuild mode, tested on Phase 4 Vocab-X, masks existing concepts during extraction.)
Stage 3 Warrant & Provenance-Lineage Gate Candidate clusters, sampling-manifest.json Structured warrant_sources on warranted items; single-lineage items routed to gaps: Passes enacted Gate 2 (§2.1: \ge 2 distinct dev provenance_family_id lineages with < 50% personal-author overlap on `min(
Stage 4 Product, Benchmark & SKU Exclusion Gate Warranted candidates & proposed aliases Filtered candidate set Passes enacted Gate 3 (§2.2: excludes products, multi-mechanism serving engines, benchmark suite families, and SKUs from concepts and multi-mechanism suites from aliases).
Stage 5 Compound Factoring & metric × workload Gate Multi-word candidates Pre-coordinated mechanisms vs. post-coordination / rejected metric × workload splits Passes enacted Gate 4 (§2.4: institutionalization across \ge 2 dev lineages; stage-metric allowlist enforced).
Stage 6 4-Part Differential Indexing Witness & Equivalence Gate Surviving candidates & entry-term proposals concepts_add (witness_justification), add_aliases (alias_metadata), or gaps: Passes enacted Gate 5 (§2.3: 6-key witness_justification for concepts; manifest-verified checked_dev_units + scope_note + word-bounded primary-excerpt external verbatim_quote across all 4 alias types; unwarranted distinct mechanisms route to gaps:, never aliases).
Stage 7 Single Primary Home, Definitions & Typed Relations Admitted concepts & updates patch.yaml (home, non-circular genus-differentia definition, definition_change_kind, edge_impact_note, checked_incident_relationships, 17-verb relationships) One home subject (home_rule), specializes passes 3-step all-and-some (BTG) test, part-of under skos:related (Z39.19 §8.3.3.2), post-patch degree \ge 2 across all active concepts.
Stage 8 3-Family Cross-Check, Reconciliation & Provenance Sidecar patch.yaml, three *-r2.yaml reviews, optional overrides.yaml final.yaml, reconcile-receipt.json (schema_version: 4), ml-systems-taxonomy.yml, consistency-chained taxonomy/term-provenance.yml (schema_version: 2), apply-receipt.json reconcile.py enforces $\ge 2$-family support on every kept operation and leaf field mutation (plus $\ge 2$-family extension_unchanged: true on definition updates) and non-overridable check_red_flags() (supporting --prepare-candidate + apply.py --emit-candidate self-consistent candidate bundles, run_reviews.sh audit --candidate-dir, --unseal-holdout immediate pre-evaluation cluster burn, and item-level Stage 6/7 receipt recomputation including seeded_gap_items and append-only evaluation_attempts); apply.py re-verifies in-process (including deterministic replay of both taxonomy and top-level provenance metadata from taxonomy/method-study/genesis-0.4-baseline.json.gz, canonical patch JSON digest binding, guides_replace support, and raw-bundle 1.0 verify_release_governance_receipt checks) and appends a SHA-256 consistency-chained campaign entry with replayed_patch to term-provenance.yml via sequential os.replace writes with rollback; 3-family publishability audit (--mode audit) passes with 0 blockers and 0 cuts (audit-receipt.json schema_version: 2).
Stage 9 Sealed Blind Coverage, Seeded-Gap & Indexing Evaluation Frozen candidate snapshot (apply.py --emit-candidate), unexposed external held_out_test clusters (reconcile.py --unseal-holdout) Coverage & agreement reports (evaluation-receipt.json, audit-receipt.json, study receipts) reconcile.py --unseal-holdout burns unsealed held_out_test clusters into burned-clusters.json (schema_version: 3) before evaluators run; two-judge blind coverage on non-editor-affiliated external units (consensus expressibility [both >= 0.5] + consensus exact single-concept [both == 1.0]) + seeded-gap sensitivity audit (§5.1) and dual-tagger agreement (\kappa, Hooper, Rolling, \alpha_{\text{MASI}}) coupled with leaf-specificity, utilization, and p_{\max} counter-metrics (§5.2) are bound to manifest evaluation_tracks and recomputed item-by-item in reconcile.py (rerun_index \in \{0, 1\}, verified against append-only evaluation_attempts in burned-clusters.json).
Stage 10 Compiler Verification, Changelog & Release Compiled artifacts, mlsysbook-alignment.yml, staffml consumer JSON Downloadable CSV/JSON/JSON Schema/GraphML/SKOS .ttl, CHANGELOG.md ./binder/binder concepts build && check && build --check (including validate_term_provenance_sidecar), coverage --check, agreement --check, pytest, and vault_cli check --strict all exit 0.

2. Concept Admission, Warrant, and Boundary Rules

Source provenance: Synthesized from Method-B §2 (D2 cross-family mean 4.50/5.0, all-judge 4.333/5.0) and Method-A §2 (D2 cross-family mean 4.50/5.0, all-judge 4.667/5.0).

2.1 Multi-Lineage Warrant and Heading-Only Ban (Enacted Gate 2)

Following ANSI/NISO Z39.19-2005 §5.3.5 (and conceptually aligned with ISO 25964-1 concept warrant):

  1. Two-Lineage dev Warrant Rule (0 single-source exceptions): Every new concept in concepts_add requires structured warrant_sources from at least two distinct provenance_family_id clusters in the dev split across S1–S5, including at least one non-editor-affiliated origin: external lineage (editor_affiliated: false; non-MLSysBook, non-Harvard CS249r, non-StaffML) and at least one source_type: prose unit containing explanatory text defining the mechanism (reconcile.py). While Method-B §2.1 explored a single-source curriculum/standard exception, the enacted protocol (METHOD.md Gate 2) allows zero single-source concept admissions: single-lineage mechanisms are recorded in gaps: until a second independent lineage appears.
  2. Authorship, Institutional Organization, & Global work_id Clustering: Two sources merge into a single effective provenance_family_id lineage (compute_lineage_components in binder/cli/core/taxonomy_schema.py, enforcing a global 1 work_id <-> 1 canonical provenance_family_id bijection) when:
    • Their listed personal author sets overlap by \ge 50\% (|A_i \cap A_j| / \min(|A_i|, |A_j|) \ge 0.50 across all listed authors) or their institutional_organizations share a technical committee or standards body (burned-clusters.json schema_version: 3, transitive closure shared across reconcile.py, apply.py, taxonomy_schema.py, and build_study_receipts.py; Method-A §2.1, [P]),
    • One work is a companion repository, documentation site, lecture adaptation, or tutorial authored by the creators of the other, or
    • Both belong to project-internal corpora (prospectively collapsed into a single internal lineage pf_internal_mlsysbook with origin: internal and split: dev, alongside historical Phase 3 per-chapter/per-sample pf_mlsysbook_* / pf_harvard_cs249r_* / pf_staffml_* IDs; multi-institutional consortium standards led by the Editor such as pf_mlperf_inference carry editor_affiliated: true and cannot satisfy the external-lineage independence floor).
  3. Heading-Only Warrant Ban & Contextual Discovery Exclusion: A syllabus or venue-checklist topic title (source_type: heading) may serve as the corroborating second lineage when paired with a source_type: prose source that defines the mechanism, but bare headings alone (0 prose sources) cannot mint a concept or justify extrapolated mechanisms, and contextual discovery leads (held_out: false checklists and taxonomy-tagging-dev-leads.yaml) never satisfy Gate 2 warrant_sources or Gate 5 checked_dev_units.

2.2 Named Entity, Product, Benchmark, and SKU Exclusion Gate (Enacted Gate 3)

Commercial products, multi-mechanism serving engines/frameworks, hardware SKUs, model releases, and benchmark suite families (vLLM, TensorRT-LLM, SGLang, DeepSpeed, PyTorch, H100, B200, MI300X, MLPerf [maintained by MLCommons], benchmark suite) are excluded as concepts (project scope policy inspired by Z39.19 §6.3.3 and §8.2; Method-A §2.2, Method-B §2.4):

  • Mechanism extraction: Extract the underlying architectural mechanism (vLLM \rightarrow mlsys.paged-kv and mlsys.continuous-batching; ZeRO \rightarrow mlsys.state-sharding).
  • Alias restriction: Only a named algorithm or single-mechanism implementation that is 1-to-1 synonymous with one mechanism and independently satisfies Gate 5's equivalence, homograph, and primary_excerpt attestation rules (primary_source_excerpt_path under <campaign>/sources/) may be added as a prospective alias (all 7 pre-sidecar 0.4-draft concepts / 8 legacy aliases lacking primary_excerpt sidecar attestation — assurance case on mlsys.safety-case, XGBoost on mlsys.tree-ensemble, grammar-guided decoding on mlsys.constrained-decoding, ZeRO and FSDP on mlsys.state-sharding, PagedAttention on mlsys.paged-kv, CUDA Graphs on mlsys.graph-replay, and CUDA streams on mlsys.device-streams — carry status pre-sidecar-legacy-queued-for-v1-audit in grandfathered_draft_alias_exceptions in term-provenance.yml). Multi-mechanism frameworks and benchmark suite families (vLLM, TensorRT-LLM, SGLang, MLPerf, benchmark suite) are excluded from both concepts and aliases (MULTI_MECHANISM_SUITE_ALIASES in reconcile.py).
  • No BTI (instance-of) relation: Individual named standards or open interchange formats enter only as general architectural mechanisms when warranted across \ge 2 lineages; BTI is excluded from the relation schema.

2.3 Operational 4-Part Concept-vs-Alias Witness & Z39.19 §8.2 Equivalence Gate (Enacted Gate 5)

To decide whether candidate C is a separate Concept, a typed alias on existing concept P, or a gaps: entry (Method-A §2.3, Method-B §1 Step 10 & §2.5, enacted in METHOD.md Gate 5):

  1. Criterion (a) — Distinct System Invariant / Envelope: Does C introduce a distinct system invariant, resource envelope, failure boundary, analytical causal model, or measurement semantics not already fixed by P?
  2. Criterion (b) — Independent Decision Consequence / Trade-Off: Does C carry an independent failure mode, trade-off, or engineering decision consequence not identical to P?
  3. Criterion (c) — Taxonomic / Causal Profile: Is C either a valid all-and-some (specializes) child of P with its own differentiating properties or an endpoint of a causal/associative edge not inherited from P?
  4. Criterion (d) — Structured Differential Indexing Witness & Two-Lineage Warrant: Can the proposer supply a 6-key witness_justification (compared_concept, criterion_a/b/c, and distinct scenario_a and scenario_b) and $\ge 2$-lineage dev warrant (Gate 2)?

Enacted three-way decision & tiebreak rule:

  • Separate Concept: Admitted when Criteria (a)–(d) and Gate 2 (\ge 2 dev lineages) all hold.
  • Unwarranted distinct mechanism \rightarrow gaps: (never aliases): When a candidate denotes a substantively distinct mechanism or methodology under Criteria (a)–(c) (for example, adaptive stress testing) but lacks two-lineage dev warrant, it routes to gaps:, never to aliases. Likewise, constituent execution phases (prefill phase vs. decode phase), distinct analytical methodologies (FMEA, FTA, STPA under Hazard analysis), planning artifacts (preregistered analysis plan), and method-produced artifacts (unsafe control action under STPA) are never aliased onto a parent.
  • Typed alias (synonym, quasi-synonym, acronym, or < 2-lineage upward-posting): Permitted only when <campaign>/sampling-manifest.json is verified, the surface term is word-bounded attested as an exact verbatim_quote substring in a cited non-editor-affiliated external dev unit with source_type: prose and passage_kind: primary_excerpt (with primary_source_excerpt_path under <campaign>/sources/ verified by SHA-256 on prospective units, never a curated_summary digest — which led to withdrawing candidate grammar-constrained decoding after checking arXiv:2312.07104), disambiguated against cross-area homographs (Z39.19 §5.3.1, §6.2.1), not globally tombstoned across other concepts (removed_aliases), and either denotes the same mechanism (synonym / acronym / Z39.19 §8.2.3 quasi-synonym) or is a valid all-and-some narrower variant (Z39.19 §8.2.4 upward-posting) with dev_lineages_requiring_separation \in \{0, 1\} (< 2 dev lineages). All four alias types require source (\ge 10 chars), manifest-verified checked_dev_units, scope_note (\ge 15 chars), and word-bounded primary-excerpt verbatim_quote in reconcile.py.

2.4 Compound Term Factoring vs. Pre-Coordination (Enacted Gate 4, Z39.19 §7)

A multi-word compound term C is admitted as a pre-coordinated concept only when (Method-A §2.3, Method-B §2.5):

  1. It names a lexicalized algorithm, protocol, or system mechanism with non-compositional behavior or its own failure mode/trade-off (speculative decoding, continuous batching, activation checkpointing), and
  2. It is institutionalized as a unit across \ge 2 distinct provenance_family_id lineages in dev ([P], aligned with Gate 2 in METHOD.md; Method-A originally proposed \ge 3 before calibration to $\ge 2`).

Generic metric × workload combinations (inference throughput, inference energy cost, training power) are factored into post-coordination rather than admitted, except for the explicit concept-scoped stage-specific structural metric allowlist (STAGE_METRIC_ALLOWED_TERMS in reconcile.py: SLO goodput on mlsys.slo-goodput, TTFT / time to first token on mlsys.ttft, TPOT / time per output token on mlsys.tpot, MFU / model FLOPs utilization on mlsys.model-flops-utilization, training goodput / ML goodput on mlsys.training-goodput, request throughput / QPS / queries per second / tokens per second on mlsys.request-throughput, latency percentiles / p50 / p99 / percentile latency on mlsys.latency-percentiles, ITL / inter-token latency on mlsys.inter-token-latency, MBU / model bandwidth utilization on mlsys.model-bandwidth-utilization).

3. Enumerated Sampling Frame, /tmp Staging, & Split Hygiene

Source provenance: Synthesized from Method-B §3 (D3 cross-family mean 4.75/5.0, all-judge 4.667/5.0), Method-A §3 (D3 cross-family mean 4.50/5.0, all-judge 4.667/5.0), and Method-C §3.2 (D3 cross-family mean 3.75/5.0, all-judge 4.00/5.0 — scale representation across cloud, enterprise/workstation, and edge/TinyML). Full details live in taxonomy/method-study/sampling-frame.md.

  1. Five Stratified Source Populations (S1–S5): S1 Textbooks & Reference Monographs, S2 University Courses, S3 Peer-Reviewed Systems & ML Venues, S4 Practitioner Artifacts & Diagnostic Scenarios, and S5 Standards, Assurance Guidance & Benchmark Specifications.
  2. Whole-Cluster Split Partition & External /tmp Workspace Staging (<campaign>/sampling-manifest.json):
    • Every unit declares unit_id, provenance_family_id, stratum (S1–S5), split (dev, held_out_test, reserve), origin (external | internal), editor_affiliated (true | false), and source_type (prose | heading).
    • Prospective refreshes target 60% dev, 25% held_out_test, and 15% reserve across external clusters (0 cross-split clusters; internal corpora collapse into pf_internal_mlsysbook in dev).
    • Across review, crosscheck, audit, and guides modes, stage_reviewer_workspace.py verifies byte-for-byte freshness of taxonomy/downloads/* against deterministic compilation (including --candidate-dir in Path A audit mode) and stages a read-only external directory in /tmp (out_dir rejected if inside repo_root or campaign_dir, with symlink-free sources/ confinement), redacting non-dev cluster/unit/agreement/held-out-checklist tokens (non_dev_leak_scan_tokens_by_category) from staged .md files and excluding held_out: true checklists, taxonomy/agreement/, and non-dev book chapters (workspace-isolation-receipt-<mode>.json, paired with macOS /usr/bin/sandbox-exec Seatbelt profiles [macos_seatbelt_all_3_families_deny_repo_campaign_peer_read_and_workspace_write with pre-flight probe verification across the entire GitHub parent directory, campaign dir, /tmp/tax-cand*, ~/.claude/{CLAUDE.md,rules,protected-checkouts.txt,projects}, ~/.codex, peer reviewer I/O dirs, and process-exec* deny on curl/gh/wget/scp/ssh/nc] wrapping all three CLIs [codex, agy, and claude -p]).
  3. Append-Only Global Cluster-Burn Ledger (taxonomy/method-study/burned-clusters.json schema_version: 3):
    • Once any unit in a held_out_test slice is unsealed via reconcile.py --unseal-holdout (or Stage 2 human packets via reconcile.py --unseal-stage2-human immediately before evaluation begins) or inspected to diagnose gaps, every unit sharing its provenance_family_id cluster across all areas is recorded in taxonomy/method-study/burned-clusters.json (schema_version: 3, verified append-only against origin/dev, dev, and HEAD locally plus server-side origin/dev CI in taxonomy-validate-dev.yml, currently pinning 98 IDs = 97 raw Phase 3 IDs + 1 canonical internal merge target pf_internal_mlsysbook, collapsing to 88 canonical clusters: 39 internal + 49 external and 41 effective lineage components = 1 internal + 40 external, with complete exposed_cluster_metadata, per-work work_rosters, append-only evaluation_attempts and stage2_human_attempts, and all 382 historical_phase3_units carrying initial_phase3_split [200/126/56] and current_exposure_split [382 dev]) and promoted to dev (held_out: false on venue checklists); reconcile.py and stage_reviewer_workspace.py fail closed if any burned cluster appears in held_out_test or reserve.

4. Structural Invariants, Relationships, and Lifecycle Governance

Source provenance: Synthesized from Method-B §4 (D4 cross-family mean 4.50/5.0, all-judge 4.667/5.0) and Method-A §4 (D4 cross-family mean 4.25/5.0, all-judge 4.50/5.0).

4.1 Polyhierarchy with Single Primary Home (home)

  • Every concept declares exactly one primary subject row (home), forming a clean partition across 101 subjects in 20 areas (Method-A §4.1, Method-B §4.2; post-move placement_examples home consistency enforced by reconcile.py). Cross-area relevance uses secondary_concepts with an explicit rationale.
  • Concept-to-concept specializes (rdfs:subPropertyOf skos:broader) forms a cycle-free DAG verified by the 3-step generic (BTG) all-and-some test (Z39.19 §8.3.1). Because ML system components participate in multiple assemblies, part-of is modeled as a cycle-checked associative relation exported under skos:related (Z39.19 §8.3.3.2), and BTI is not used.

4.2 Controlled 17-Verb Relationships & Extension-Preserving Definitions

  • Relationships use the closed 17-verb LinkML vocabulary (specializes + 16 associative/causal/evaluation/composition verbs under skos:related), disjoint from specializes transitive closure (SKOS S27 / Z39.19 §8.4), with mandatory predicate and conditions strings verified in both forward and inverse literal readings; every active concept must maintain post-patch incident degree \ge 2 (reconcile.py).
  • In-place concepts_update.definition edits require definition_change_kind: editorial-clarification, an explicit edge_impact_note, exact checked_incident_relationships matching all incident relationship IDs, and $\ge 2$-family extension_unchanged: true cross-check verification (reconcile.py & term-provenance.yml) verifying that every incident relationship remains literally true without shifting extension; extension-shifting changes require concepts_retire + concepts_add.

4.3 Non-Circular Genus–Differentia Definitions & Compiler Sidecar Gates

  • Every concept definition is a single non-circular genus-differentia sentence stating what kind of mechanism, structure, failure mode, or metric it is and what it changes in the system (Z39.19 §6.2.2; post-patch (effective_label, effective_defn) circularity enforced by reconcile.py).
  • ./binder/binder concepts check enforces LinkML schema conformance, monotonic SemVer ordering (supporting 0.x-draft, 1.x-draft maintenance branches, 1.0-rcN, and 1.x.y), SKOS S14 (prefLabel cardinality) plus global case-folded preferred/entry-term uniqueness across active and retired concepts (Z39.19 §5.3.1), SKOS S13 (prefLabel/altLabel disjointness), hierarchical DAG acyclicity, polarity consistency, and validate_term_provenance_sidecar (schema_version == 2, deterministic replay of both taxonomy state and top-level provenance metadata via replay_provenance_step from taxonomy/method-study/genesis-0.4-baseline.json.gz across every applied_campaigns entry across apply.py [including versioned guides_replace] and -draft-only guides_apply.py [which forbids version changes completely], final_yaml_canonical_json_sha256 binding, taxonomy_sha256, non_guides_taxonomy_sha256, sidecar_content_sha256 self-digest, global removed-alias tombstones, permanent retired-string reservation to replaced_by, governed attest_existing_aliases backfill, and raw-bundle verify_release_governance_receipt on non-draft/1.0 releases — detecting un-reconciled YAML or provenance tampering in-repo, paired with git history and CI diff review against origin/dev for validator code integrity).

4.4 Permanent Identities, Zero-Extension-Shift Parent Rule, and Retirement Schema

Concept IDs (mlsys.*) are permanent and never reused (Method-A §4.6, Method-B §4.7–4.8, METHOD.md, .claude/rules/taxonomy/content.md):

  • Label refinement (unchanged extension): Update label and retain the former label in aliases (with alias_metadata).
  • Zero-extension-shift parent ID vs. concept split / narrowing rule:
    • Adding a specializes child while preserving the parent's extension (0% extension shift on parent): When a narrower sub-mechanism is minted as a new specializes child while the broader parent's own extension and every incident relationship remain unchanged (definition_change_kind: editorial-clarification), keep the existing mlsys.* ID on the broader parent, mint a new mlsys.* ID for the narrower child via concepts_add, and re-review downstream StaffML and mlsysbook-alignment.yml tags to migrate items specifically teaching the narrower child.
    • Any narrowing, broadening, or splitting of an existing concept's extension: Whenever a concept's own extension narrows, broadens, or splits into peer mechanisms (so existing references to the legacy ID no longer denote the same extension; Method-A §4.6 had explored an < 20% tolerance, which the enacted protocol tightened to zero extension shift in METHOD.md Gate 4 and §4.2 above), retire the legacy mlsys.* ID (concepts_retire, optionally setting migrate_hierarchy_and_views: true and pointing replaced_by to the primary redirect in 0.x-draft; former labels and aliases remain permanently reserved to that replaced_by target), mint new mlsys.* IDs via concepts_add, and retag all downstream StaffML and mlsysbook-alignment.yml consumers before closing the campaign.
  • Enacted 0.4-draft RetiredConcept vs. prospective 1.0 fields: In 0.4-draft (mlsys-taxonomy.linkml.yml and apply.py), RetiredConcept persists id, former label, former aliases, transitive-resolved active replaced_by, version, and reason. Adding machine-typed resolution_class (auto, broaden, review, split) and multivalued split_into slots is recorded as a prospective 1.0 LinkML schema extension for external board review (candidate-experts.md).

5. Evaluation Protocol, Empirical Calibration Findings, and Anti-Gaming Controls

Source provenance: Synthesized from Method-B §5–6 (D5 cross-family mean 4.75/5.0 [all-judge 4.667], D6 4.75/5.0 [4.667], D7 4.75/5.0 [4.667]) and Method-A §5–6 (D5 4.00/5.0 [4.167], D6 4.25/5.0 [4.333], D7 4.00/5.0 [4.167]).

5.1 Two-Pass Blind Coverage, Expressibility vs. Exact Estimators, & Seeded-Gap Limitation

To test vocabulary completeness on unexposed origin: external held_out_test clusters (Method-A §5.6, Method-B §5.3):

  • Pass 1 (Vocabulary-Blind Topic Extraction): Extract central mechanisms from external held-out units in an isolated directory with no access to the taxonomy.
  • Pass 2 (Two-Family Expressibility Scoring — Two Distinct Consensus Estimators): Two distinct model families score each unit (Exact = 1.0 single concept/alias, Post-coord = 0.5 conjunction of 2–3 concepts without specificity loss, Missing = 0.0), co-reporting three distinct binary metrics plus 10,000-resample deduplicated-lineage cluster-bootstrap 95% CIs:
    1. Consensus expressibility (both judges >= 0.5, i.e., Exact or Post-coord in both judges): 36.7% (29/79) on Vocab-Y vs. 10.1% (8/79) on Vocab-X in Phase 6 (diff = -26.6 pp; 9-lineage deduplicated cluster-bootstrap 95% CI [-45.1, -6.8] pp, 10-raw-cluster CI [-44.9, -6.7] pp; 36.8% [28/76] vs. 10.5% [8/76] on dev_disjoint_76_external; 38.4% [28/73] vs. 11.0% [8/73] on unexposed_73_systems_venues_2026). Headline Step 5b non-regression gate: paired cluster-bootstrap 95% lower bound >= -5.0 pp on shared unexposed external units ([P]).
    2. Consensus exact single-concept coverage (both judges == 1.0): 19.0% (15/79) on Vocab-Y vs. 2.5% (2/79) on Vocab-X in Phase 6 (diff = -16.5 pp across all 79 units; 19.2% [14/73] vs. 2.7% [2/73], diff = -16.4 pp from 2/73 - 14/73 = -16.44 pp on unexposed_73_systems_venues_2026; co-reported alongside consensus expressibility so post-coordination reliance is transparent).
    3. Any-judge expressibility (either judge >= 0.5): 72.2% (57/79) on Vocab-Y vs. 36.7% (29/79) on Vocab-X, paired with binary coverage judge \kappa (0.366 vs. 0.325).
  • Seeded-Gap Sensitivity Audit & Empirical Phase 6 Limitation (Method-A §5.6): Method-A proposed removing k concepts (k = 10 in Phase 6) and requiring Pass 2 to degrade \ge 85\% ([P]) of baseline-covered judgments citing those concepts. Empirical Phase 6 result (study-receipts-summary.json): unrestricted LLM judges degraded only 24 / 40 = 60.0% of affected judgments on Vocab-Y and 16 / 29 = 55.2% on Vocab-X — falling short of the \ge 85\% target because judges substituted broader parent/neighbor concepts or 0.5 post-coordination. Consequently, taxonomy-refresh.md Step 5b reports seeded-gap sensitivity as a mandatory diagnostic audit during 0.x-draft until negative-example prompt calibration achieves \ge 85.0\% sensitivity prior to 1.0.

5.2 Multi-Rater Indexing Agreement + Specificity Counter-Metrics

On held-out scenarios indexed independently by two blind taggers (1 primary subject, 0–3 concepts):

  1. Primary subject agreement: Cohen's \kappa on single-pair LLM runs (release floor >= 0.70), and both orientation-invariant Krippendorff's nominal \alpha_{\text{subject}} \ge 0.65 and pair-stratified \bar{\kappa}_{\text{subject}} \ge 0.65 on the 6-pair (12-human) pre-1.0 study.
  2. Concept-set consistency: Both conditional (both non-empty) and all-item (J(\emptyset,\emptyset)=1, J(A,\emptyset)=0) Hooper (1965) and Rolling (1981) overlap (LLM floors >= 40% / >= 57% conditional, >= 35% / >= 50% all-item), paired with empty-assignment rate (\le 20\%).
  3. Chance-corrected set agreement: Krippendorff's $\alpha_{\text{MASI}}$ using Passonneau's (2006) MASI distance d_{\text{MASI}}(A,B) = 1 - J(A,B)\cdot M(A,B) (LLM floor >= 0.40).
  4. Anti-Gaming Specificity Counter-Metrics (Method-B §6.7, Method-C §6.4):
    • Leaf-concept specificity ratio: Share of assigned concept tags whose concept has 0 incoming specializes children (in Phase 6 IDX, Vocab-X scored higher than Vocab-Y: 86.1% [186/216] vs. 80.9% [148/183]).
    • Max single-concept share (p_{\max}): \le 15.0\% (Vocab-X achieved lower concentration than Vocab-Y: 6.94% vs. 9.84%).
    • Effective vocabulary utilization: Distinct concepts assigned divided by active area concepts (Vocab-X = 55 / 111 = 49.5% vs. Vocab-Y = 44 / 125 = 35.2%).

6. Traceability Matrix: Pre-Study Baseline vs. Synthesized Design vs. Enacted METHOD.md

Rubric scores below are exact cross-family means (dim_means_cross, excluding self-ratings; totals in study-receipts-summary.json: Method-B 37.25/40 cross / 36.50/40 all, Method-A 34.00/40 cross / 35.33/40 all, Method-C 25.75/40 cross / 29.83/40 all).

Mechanism Pre-study 0.4-draft METHOD.md (before Phase 7) Enacted Post-Study METHOD.md & reconcile.py Blind Proposal & Exact Cross-Family Score
1. Sampling frame & split hygiene Checklist-level held_out flag only; no provenance_family_id clustering or reserve pool 5-stratum frame (S1–S5), whole-cluster dev/held_out_test/reserve in sampling-manifest.json, external /tmp staging (stage_reviewer_workspace.py), append-only burned-clusters.json Method-B §3 (D3 = 4.75), Method-A §3 (D3 = 4.50), Method-C §3.2 (D3 = 3.75)
2. Two-lineage warrant & prose check Two sources without authorship clustering or internal/external separation \ge 2 distinct dev provenance_family_id clusters (< 50% author-overlap closure), \ge 1 non-editor-affiliated origin: external, \ge 1 source_type: prose, 0 single-source exceptions Method-A §2.1 (D2 = 4.50), Method-B §2.1–2.2 (D2 = 4.50)
3. Concept-vs-alias & compound gates Unstructured editorial judgment; MLPerf / benchmark suite / preregistered analysis plan leaked into aliases 6-key witness_justification, manifest-checked alias_metadata + scope_note + word-bounded primary-excerpt external verbatim_quote across all 4 alias types (upward-posting < 2 dev lineages), $\ge 2$-lineage compound gate, unwarranted concepts \rightarrow gaps: Method-B §1 & §2.5 (D2 = 4.50), Method-A §2.3 (D2 = 4.50)
4. Structural & provenance gates 17 LinkML verbs + SKOS S27; no committed provenance sidecar binding ml-systems-taxonomy.yml Adds definition_change_kind: editorial-clarification (0 extension shift on in-place edits), edge_impact_note, post-patch degree \ge 2, former label/aliases on RetiredConcept, and consistency-chained taxonomy/term-provenance.yml (schema_version: 2) verified inside binder concepts check Method-B §4 (D4 = 4.50), Method-A §4.3 & §4.6 (D4 = 4.25)
5. Coverage & seeded-gap audit Single-judge checklist status without seeded-gap sensitivity calibration Two-family blind scoring (both >= 0.5 expressibility + both == 1.0 exact) + 9-lineage deduplicated cluster-bootstrap CIs + mandatory seeded-gap sensitivity audit (60.0% / 55.2% vs. >= 85% target) Method-B §5.3 (D5 = 4.75, D6 = 4.75), Method-A §5.6 (D5 = 4.00, D6 = 4.25)
6. Agreement & specificity coupling Subject \kappa and conditional Hooper/Rolling on 0.2-draft StaffML samples Adds all-item Hooper/Rolling, \alpha_{\text{MASI}}, empty-set ceiling (\le 20\%), leaf-concept ratio (0 specializes children), p_{\max} \le 15\%, and 240-item / 40-anchor 12-human 1.0 protocol Method-B §5.2 & §6.7 (D5 = 4.75, D6 = 4.75), Method-A §5.4 (D5 = 4.00), Method-C §6.4 (D6 = 3.00)
Reference in New Issue View Git Blame Copy Permalink