github-starred/cs249r_book
2
0
Fork 0
mirror of https://github.com/harvard-edge/cs249r_book.git synced 2026-11-09 10:39:41 -06:00
Code Issues 276 Packages Projects Releases 26 Wiki Activity
Files
dev
cs249r_book/taxonomy/method-study/candidate-experts.md
T

58 KiB
Raw Permalink Blame History

External Editorial Board Charter and Human Evaluation Protocol (Z39.19 §11.1.3.1, §11.2.2, §11.3.1.1)

Status: public governance charter and role specification for version 0.4-draft (2026-11-02). This document defines seat qualifications, area coverage obligations, appointment and conflict-of-interest rules, and pre-registered human evaluation specifications only; no external board has yet been convened, no individual has been contacted or asked to endorse the ML Systems Taxonomy, and unconsented personal names are intentionally excluded from the public repository.

1. Purpose and Historical Context

Under ANSI/NISO Z39.19-2005 (R2010) §11.1.3.1 (Committee Approach), §11.2.2 (Evaluation Criteria), and §11.3.1.1 (Addition of Terms), a controlled vocabulary intended for community reuse combines empirical corpus extraction with review by domain subject specialists and controlled-vocabulary specialists.

In 0.4-draft, construction and verification used language-model passes under deterministic reconciliation and LinkML compilation gates, with final decisions made by a single human Editor (Vijay Janapa Reddi): specifically, the 0.4-draft refresh ran Round 1 as seven parallel lenses in one model family (claude-fable-5-1) followed by a three-family Round 2 cross-check and independent audit (gpt-6.1-sol, gemini-4-argon-high, claude-opus-5-5), while three-family blind discovery and reconciliation were evaluated on the pre-registered Phase 3–7 two-area study (P4 and Q4) and codified for future refreshes.

To transition from an editorial draft (0.x-draft) to a citable community release (1.0), the project requires review by a 7-seat external editorial board (13 mandatory dedicated sub-area specialists covering 12 areas [Q4a+Q4b in Q4] plus Chair/Co-Chair/Alternate coverage of the remaining 8 areas [F1, F2, S1–S3, P4, T1, T2] across all 20 taxonomy areas / 101 subjects / 878 concepts) and a fixed 12-indexer (6-pair) dual-human inter-indexer consistency study.

2. Board Structure, Voting Seats, and Full 20-Area Specialist Coverage

Note: Sampling-frame strata S1–S5 (S1 textbooks/monographs, S2 university courses, S3 peer-reviewed systems/ML venues, S4 practitioner artifacts and StaffML scenarios, S5 standards and assurance frameworks) are distinct from Stack area codes S1–S4. Subject and concept counts below are computed directly from taxonomy/ml-systems-taxonomy.yml (12 + 22 + 34 + 23 + 10 = 101 subjects; 94 + 206 + 301 + 200 + 77 = 878 primary concepts).

The board comprises 7 voting seats (5 domain group seats + 2 vocabulary/SKOS seats). Every voting seat has designated unconflicted Alternate coverage from a distinct institution (16 distinct natural-person (seat, role) standing-roster slots total: 6 single-seat Chairs + 2 T1/T2 Co-Chairs + 8 Alternates), plus 13 mandatory dedicated sub-area specialist reviewers (29 standing + specialist positions total: 13 dedicated specialists covering 12 areas [F3, S4, P1, P2, P3, P5, Q1, Q2, Q3, Q4a+Q4b, Q5, Q6] alongside Chair/Co-Chair/Alternate coverage of the remaining 8 areas [F1, F2, S1, S2, S3, P4, T1, T2]) so that all 20 areas (101 subjects) receive dedicated expert review before 1.0 sign-off. Distinct-person and anti-self-appointment rule (46 minimum distinct natural persons): the independent administrative convenor (1), all 3 ex-officio nominating-panel members (3), the external academic steering advisor (1), all 16 standing roster members (16), all 13 mandatory sub-area specialists (13), all 12 human indexers (12; 1 + 3 + 1 + 16 + 13 + 12 = 46 distinct individuals minimum), plus any temporary appointing-committee proxy voters and any ad-hoc escalation voters must be distinct natural persons (0 cross-seat or cross-role reuse across person_id, normalized_full_name, and orcid_or_uri; neither the administrative convenor nor any member of the 3-scholar ex-officio nominating panel may be nominated or appointed to the steering-advisor role, V1, V2, any standing Chair/Co-Chair/Alternate seat, any of the 13 sub-area specialist roles, any ad-hoc item escalation panel, or any of the 12 human indexer roles in the release cycle they convene or nominate):

Voting Seat Taxonomy Scope (ml-systems-taxonomy.yml counts) Seat Composition (Chairs, Alternates & Specialists) Required Qualifications & Stratum Grounding Mandatory Sub-Area Specialist Coverage (Prerequisite Before 1.0 Sign-Off)
Seat F (Foundations, 1 vote) F1 Workloads & requirements · F2 Models & architectures · F3 Quantitative system models (12 subjects, 94 primary concepts, 115 incl. secondary) 1 Chair + 1 Alternate from distinct institutions, plus 1 F3 specialist reviewer Published research and graduate teaching (S1–S3) in ML workloads, foundation-model architectures, and analytical performance/scaling models. Dedicated queueing, roofline, and analytical performance-modeling specialist required for F3 alongside F1/F2 architecture coverage.
Seat S (Stack, 1 vote) S1 Hardware & accelerators · S2 Frameworks, compilers & runtimes · S3 Parallelism & distributed execution · S4 Datacenter infrastructure & fleet (22 subjects, 206 primary concepts, 225 incl. secondary) 1 Chair + 1 Alternate from distinct institutions, plus 1 S4 specialist reviewer Graduate systems courses (S2) and peer-reviewed compiler, accelerator, or distributed-execution systems (S3). Dedicated datacenter networking, cluster scheduling, and power/thermal fleet infrastructure specialist required for S4 alongside S1–S3 hardware/compiler/parallelism coverage.
Seat P (Lifecycle, 1 vote) P1 Data engineering (6 subj.) · P2 Training & adaptation (7 subj.) · P3 Model evaluation & testing (5 subj.) · P4 Deployment & serving (9 subj.) · P5 MLOps & operations (7 subj.) (34 subjects, 301 primary concepts, 336 incl. secondary) 1 Chair + 1 Alternate from distinct institutions, plus 4 targeted sub-area specialists Peer-reviewed serving, training, data-pipeline, and evaluation systems (S3) plus production deployment/MLOps artifacts (S4). Because P spans 34 subjects across 5 distinct engineering disciplines, release sign-off requires Chair/Alternate P4 serving coverage plus 4 dedicated sub-area specialists: (i) P1 data engineering, curation & storage pipelines, (ii) P2 distributed pretraining & RL post-training, (iii) P3 offline/online evaluation protocols & statistical testing, and (iv) P5 MLOps, CI/CD, observability & fleet incident operations.
Seat Q (Qualities, 1 vote) Q1 Performance, efficiency & benchmarking · Q2 Reliability & fault tolerance · Q3 Security & privacy · Q4 Safety & assurance · Q5 Responsible AI · Q6 Sustainability & cost (23 subjects, 200 primary concepts, 240 incl. secondary) 1 Chair + 1 Alternate from distinct institutions, plus 7 targeted sub-area specialists Standards participation (S5) and peer-reviewed research (S3) across system benchmarking/efficiency, formal/runtime assurance, security/privacy, and governance. Release sign-off requires 7 distinct specialist reports: (i) Q1 benchmarking & compression efficiency, (ii) Q2 fault-tolerant distributed systems, (iii) Q3 ML security & differential privacy, (iv) Q4a formal & runtime safety verification, (v) Q4b system-theoretic hazard analysis & safety cases (STPA, FMEA, FTA, UL 4600, SOTIF), (vi) Q5 algorithmic fairness, sociotechnical evaluation & AI governance, and (vii) Q6 datacenter energy, operational/embodied carbon & compute cost accounting.
Seat T (System types, 1 shared vote) T1 Agentic systems (5 subj., 25 primary concepts) · T2 Embodied & physical systems (5 subj., 52 primary concepts) (10 subjects, 77 primary concepts, 106 incl. secondary) 2 Joint Co-Chairs (1 T1, 1 T2) + 2 Specialty Alternates (1 T1 Alternate, 1 T2 Alternate) across distinct institutions T1: agentic harnesses, tool-use security, and multi-agent evaluation (S2–S4). T2: robotics, autonomous cyber-physical control, sensing, and WCET (S2, S3, S5). When either Co-Chair recuses, their specialty Alternate (T1 or T2) acts for that half of Seat T. If both T1 Co-Chair and T1 Alternate are conflicted on a T1 item (or both T2 Co-Chair and T2 Alternate are conflicted on a T2 item), Seat T's item vote escalates to the deterministic 3-voter panel in §3.2 rather than permitting the unconflicted half of T to decide outside its specialty. Both T1 and T2 holders must concur for Seat T to cast an approve release vote; if they split on release sign-off, Seat T records abstain (so >= 5 of 7 approval must come from the other seats) and any item rejected by its T1 or T2 Co-Chair/Alternate is blocked from 1.0 under the binding board-rejection exclusion rule.
Seat V1 (Controlled Vocabulary & LIS, 1 vote) ANSI/NISO Z39.19, ISO 25964-1/-2, faceted classification, equivalence/compound rules, and human indexer study 1 Chair + 1 Alternate from distinct institutions Scholarship or standards-committee experience in knowledge organization systems (KOS), thesaurus construction (Z39.19 / ISO 25964), and inter-indexer consistency evaluation. Affirmative sign-off from V1 (via Chair, or Alternate on recusal) is mandatory on taxonomy/METHOD.md clause conformance and on the pre-registered human inter-indexer protocol before data collection and before 1.0 release.
Seat V2 (SKOS / Semantic Web, 1 vote) W3C SKOS Reference, OWL typed relation extensions, FAIR vocabulary publication, and external crosswalks 1 Chair + 1 Alternate from distinct institutions Authorship or editorial maintenance of published W3C SKOS/OWL vocabularies, RDF validation pipelines, and ISO 25964-to-SKOS mappings. Affirmative sign-off from V2 (via Chair, or Alternate on recusal) is mandatory on mlsys-taxonomy.ttl, mlsys-taxonomy-relations.ttl, mlsys-taxonomy.dcat.ttl, and external SKOS crosswalks before 1.0 release.

3. Appointment Independence, Recusal, Quorum, and Binding Board-Rejection Exclusion Rule

  1. Independent appointment chain, non-Editor administrative convenor, and standing-roster vs. active-voting co-authorship caps:
    • Deterministic initial convenor selection, ex-officio nominating panel, & open-nomination fallback: Outreach to form the ex-officio nominating panel is administered by an independent non-Editor administrative convenor selected deterministically — without Editor discretion — by inviting, in reverse-chronological order of most recently completed chair or co-chair term across the fixed standards-committee precedence sequence (1) NISO Z39.19 / Information Discovery & Interchange Topic Committee -> (2) ISO/TC 46/SC 9/WG 8 (ISO 25964) -> (3) W3C SKOS Working Group -> (4) DCMI Usage Board (with any same-date term-completion tie broken by ascending SHA-256(NIST Randomness Beacon pulse at outreach announcement || candidate ORCID/URI)), the first willing scholar from an institution disjoint from the Editor's institution who has zero co-authorship or joint grant/consortium leadership with the Editor in the trailing 5 years (neither the Editor nor any member of the Editor's lab or institution plays any role in selecting, ranking, contacting, or filtering the convenor or the nominating panel; if all four historical rosters are exhausted after 21 calendar days, a public 14-day call for active ISKO / ASIS&T SIG/CR officers meeting the identical independence criteria is resolved by the same NIST Randomness Beacon hash tiebreak). Mandatory pre-outreach publication: the administrative convenor's signed COI declaration, eligibility check, and selection trace must be committed to the public governance archive before any nominating-panel invitation is sent. Before any board invitations issue, that convenor forms an independent 3-scholar nominating panel ex officio by inviting, in deterministic reverse-chronological order of service, the most recent willing program or general chairs of (i) MLSys (fallback: EuroSys / NSDI), (ii) OSDI or SOSP (fallback: ASPLOS / ISCA), and (iii) an active NISO / DCMI / W3C knowledge-organization or SKOS working group (fallback: ISKO / ASIS&T SIG/CR chairs), each a distinct natural person disjoint from the convenor, from a distinct institution disjoint from the Editor's institution, and having zero co-authorship or joint grant/consortium leadership with the Editor in the trailing 5 years. Deterministic open-nomination fallback: if any of the 3 nominating seats remains unfilled after 30 calendar days across both primary and fallback venue lists, the administrative convenor opens a public 21-day community/self-nomination call for scholars meeting the seat's domain and independence criteria and selects among verified-eligible nominees via a pre-announced public randomness beacon (SHA-256 of the first NIST Randomness Beacon pulse after the 21-day window closes). That 3-scholar nominating panel appoints by \ge 2/3 vote an external academic steering advisor (tenured faculty member or standards-body working-group chair disjoint from the convenor and nominating panel, from an institution disjoint from the Editor's institution, with zero co-authorship or joint grant/consortium leadership with the Editor in the trailing 5 years).
    • Independent V1/V2 and domain Chair nomination, candidate-specific recusal, & appointing-committee vacancy backfill: Every appointing voter (including the steering advisor, V1/V2 Chairs and Alternates, and any nominating-panel or temporary-proxy substitute) is bound by the identical COI disclosure and candidate-specific recusal rules in §3.2 (and may never vote on their own appointment or on a candidate sharing any of their institutions or trailing-5-year co-author networks). Whenever the steering advisor is conflicted or unavailable on a candidate appointment or ad-hoc voter selection, an unconflicted member of the 3-scholar ex-officio nominating panel (selected in deterministic seat order (iii) -> (i) -> (ii)) substitutes for the steering advisor. To seat V1 and V2 Chairs and Alternates before V1/V2 themselves exist, each of the 4 V1/V2 positions (V1:chair, V1:alternate, V2:chair, V2:alternate) requires a recorded >= 3-voter appointment vote (\ge 2/3 affirmative votes across \ge 3 eligible unconflicted distinct-person voters: the steering_advisor plus at least 2 unconflicted nominating_panel members, or all 3 unconflicted nominating_panel members if the steering_advisor is conflicted on that candidate; all 4 appointees must be distinct natural persons disjoint from the convenor and nominating panel, from 4 distinct non-Editor institutions, with zero co-authorship with the Editor in the trailing 5 years; the Editor may surface scope context in writing but holds no appointment or veto power). Once V1 and V2 are seated, the 3-voter appointing committee {V1 Chair, V2 Chair, external academic steering advisor} appoints all domain Chairs, T1/T2 Co-Chairs, Alternates, the 13 mandatory sub-area specialists, any ad-hoc escalation voters (adhoc_voters), and the 12 human indexers by a \ge 2/3 vote (>= 2 of 3 eligible unconflicted voters concurring across >= 3 eligible unconflicted distinct-person voters). Deterministic appointing-committee vacancy backfill & 3-voter halt floor: if V1 Chair is conflicted or vacant on a candidate, V1 Alternate substitutes; if both V1 Chair and V1 Alternate are conflicted, replacement proceeds in deterministic order (a) unconflicted V2 Alternate (if not already serving on the 3-voter committee) -> (b) unconflicted member of the reconvened 3-scholar ex-officio nominating panel in seat order (iii) -> (i) -> (ii) not already substituting on that vote (symmetric order if both V2 Chair and V2 Alternate are conflicted: unconflicted V1 Alternate -> unconflicted nominating-panel member in order (iii) -> (i) -> (ii)). Mandatory 3-voter appointment floor: every appointment vote (steering_advisor, all 16 standing roster slots, all 13 specialists, any substitute_specialists, all 12 indexers, and any adhoc_voters) requires at least 3 eligible unconflicted distinct-person voters (len(voters) >= 3, approve / len(voters) >= 2/3); if fewer than 3 unconflicted voters are available after exhausting those substitutions, all appointments halt until the ex-officio nominating panel (>= 2 of 3 unconflicted members) seats an unconflicted temporary_proxies voter (non-Editor institution, zero Editor co-authorship in the trailing 5 years), paired with mandatory verification that distinct-person, institutional, and co-authorship caps hold before review packets issue.
    • Standing-roster caps, non-Editor single-institution diversity caps, & dynamic active-voting caps: Every seat pairs a Chair (or T1/T2 Co-Chair) and Alternate with disjoint institutions sets (0 shared primary or secondary institutions within any seat pair, and 0 shared institutions across T1:cochair, T1:alternate, T2:cochair, T2:alternate):
      • Across the 16-person standing roster (6 Chairs + 2 T1/T2 Co-Chairs + 8 Alternates): at most 2 standing-roster members (<= 1 Chair/Co-Chair and <= 1 Alternate, and 0 on V1/V2) may share a primary or secondary institution with the Editor; at most 2 standing-roster members (<= 1 Chair/Co-Chair) may share any single non-Editor institution; and at most 3 domain standing-roster members (<= 2 Chairs/Co-Chairs and <= 1 Alternate, and 0 on V1/V2) may have co-authored a publication or benchmark specification with the Editor in the trailing 5 years.
      • Across the 13 mandatory sub-area specialists and 12 human indexers: at most 1 specialist (<= 3 from any single non-Editor institution) and 0 human indexers (<= 2 from any single non-Editor institution, and <= 1 per indexing pair so paired raters never share an institution) may share a primary or secondary institution with the Editor; at most 3 specialists and 0 human indexers may have co-authored with the Editor in the trailing 5 years; and 0 human indexers may be current or former PhD/postdoc advisees, lab members, or direct reports of the Editor (editor_advisee_or_lab_or_report == false).
      • On every active vote — re-verified dynamically on full-board item votes (including reduced-N_{\text{unconflicted}} votes), 1.0 release sign-off votes (8 active voting individuals across F, S, P, Q, T1, T2, V1, V2), and 3-voter escalation panels, counting any Alternate or ad-hoc voter acting in a voting capacity and counting Seat T's T1 and T2 holders individually — at most 1 active voting holder may share a primary or secondary institution with the Editor, at most 2 active voting holders (<= 1 on a 3-voter escalation panel) may share any single non-Editor institution, and at most 2 active domain voting holders (<= 1 on a 3-voter escalation panel, 0 on V1/V2) may have co-authored a publication or benchmark specification with the Editor in the trailing 5 years. If substituting an Alternate would breach an active-voting cap, on an item vote that Alternate is skipped in favor of the unconflicted 3-voter panel below (or next eligible circular seat), and on a 1.0 release sign-off vote (or when N_{\text{unconflicted}} would drop below 5) {V1 Chair, V2 Chair, external academic steering advisor} appoint by \ge 2/3 vote an unconflicted ad-hoc external voter with zero Editor co-authorship and a non-Editor institution to represent that seat so 7/7 quorum and active-voting caps hold simultaneously.
  2. Mandatory conflict-of-interest disclosure, specialist substitution, and deterministic 3-voter escalation:
    • Before reviewing a packet, each Chair, Co-Chair, Alternate, and sub-area specialist discloses (a) co-authorship or joint consortium/working-group leadership with the Editor within the trailing 5 years, (b) authorship or leadership of frameworks, papers, or standards tied to candidate mechanisms, and (c) commercial vendor affiliations.
    • Any participant who authored a cited warrant source, led an associated framework/standard, or co-authored the topic with the Editor recuses from evaluating or voting on the admission, preferred label, definition, or aliasing of that item (multi-mechanism suites and benchmark consortia such as vLLM and MLPerf remain categorically excluded under Gate 3). If any of the 13 mandatory sub-area specialists (F3, S4, P1, P2, P3, P5, Q1, Q2, Q3, Q4a, Q4b, Q5, Q6) is conflicted on an item in their sub-area, {V1 Chair, V2 Chair, external academic steering advisor} appoint an unconflicted substitute specialist (distinct natural person) for those items so no sub-area item goes unreviewed.
    • One vote per seat & unconflicted substitution: When a Chair recuses on an item, voting authority for that seat transfers to that seat's unconflicted Alternate (in Seat T, from T1 Co-Chair to T1 Alternate, or T2 Co-Chair to T2 Alternate); each of the 7 seats casts at most one vote.
    • Deterministic 3-voter unconflicted escalation panel: If both Chair and Alternate on a domain seat X \in \{F, S, P, Q\} are conflicted on an item — or in Seat T, if both T1 Co-Chair and T1 Alternate are conflicted on a T1-scoped item, both T2 Co-Chair and T2 Alternate are conflicted on a T2-scoped item, or a cross-T1/T2 item leaves either half without an unconflicted holder — V1 convenes a deterministic 3-voter unconflicted panel (satisfying the active-voting caps of <= 1 Editor-institution voter and <= 1 trailing-5-year Editor co-author across the 3 panelists) comprising:
      1. V1 Chair (or V1 Alternate if V1 Chair is conflicted, or V2 Chair / V2 Alternate if both V1 holders are conflicted; if all 4 vocabulary holders across V1 and V2 are conflicted on that item, the external academic steering advisor appoints an ad-hoc unconflicted external KOS/SKOS specialist voter from a non-Editor institution with zero Editor co-authorship in the trailing 5 years before the panel convenes), plus
      2. The first 2 unconflicted domain seats scanning forward from X in the fixed circular order F -> S -> P -> Q -> T -> F whose participation preserves the panel's active-voting caps, represented by each seat's unconflicted Chair (or unconflicted Alternate if its Chair is conflicted; when Seat T serves as an escalation panelist, unconflicted T1 and T2 holders cast a single joint vote if they concur, or the scan advances to the next unconflicted domain seat in circular order if either half of T is conflicted or split). Each panelist casts one vote (>= 2 of 3 required to approve, zero 1–1 ties). Fallback halt rule: if fewer than 2 eligible unconflicted domain seats exist across the board on an item (or if both V1 holders are conflicted on a V1-scoped standards question, or both V2 holders on a V2-scoped SKOS question), {V1 Chair, V2 Chair, external academic steering advisor} (via the unconflicted members plus steering advisor) must appoint unconflicted ad-hoc external specialist voter(s) (distinct natural persons, non-Editor institution, zero Editor co-authorship in the trailing 5 years) before that item can be approved; otherwise the item is withheld from 1.0.
    • Recusal denominator in full-board item votes: If a full-board item vote has N_{\text{unconflicted}} unconflicted seats satisfying the active-voting caps (after Alternate or ad-hoc substitution), valid action requires N_{\text{unconflicted}} \ge 5 and affirmative votes from at least \lceil (N_{\text{unconflicted}} + 1) / 2 \rceil unconflicted seats (an explicit abstain by an unconflicted seat stays in the denominator).
  3. Full-domain 1.0 quorum and binding board-rejection exclusion rule:
    • 1.0 Quorum: Release sign-off requires all 7 seats participating (each via its unconflicted Chair or Alternate — ensuring all 5 domain group seats F, S, P, Q, T and both V1 and V2 are represented) plus completion of all mandatory sub-area specialist reports in §2.
    • Release approval threshold: A 1.0 citable release requires >= 5 of 7 seats voting approve, including affirmative votes from both V1 and V2 within their respective scopes.
    • Binding board-rejection exclusion rule: While the Editor holds repository maintenance authority over 0.x-draft experimental branches, no concept, alias, relationship, lifecycle schema change, or conformance claim rejected by a board majority (or rejected within scope by V1, V2, or its responsible T1/T2 holder) may ship inside a 1.0 release. Rejected items must either be revised and pass renewed review by the rejecting seat(s)/majority or be excluded/retired before 1.0 tags are cut, with all dispositions recorded in taxonomy/CHANGELOG.md.

4. Scoped Work Packages and Pre-Registered Dual-Human Indexing Design

  1. Domain Group Packets (F, S, P, Q, and T1+T2, plus mandatory sub-area specialists):
    • Subject & Concept Inventory (10–34 subjects and 77–301 primary concepts [106–336 including secondary placements] per group in 0.4-draft): audit missing must-have mechanisms backed by \ge 2 independent provenance_family_id lineages (>= 1 non-editor-affiliated origin: external), product/suite leaks (Gate 3), and Home-Rule placements.
    • Split Hierarchy Queue (16 edges across the taxonomy): adjudicate each specializes edge where prior model audits split using the 3-step all-and-some test (Z39.19 §8.3.1).
    • Zero-Warrant Queue (27 concepts across the taxonomy): verify external literary warrant in S1–S5 or recommend retirement (concepts_retire with replaced_by) or typed aliasing.
  2. Vocabulary & SKOS Specialist Packets (V1 and V2):
    • V1 (Z39.19 / ISO 25964 Packet): verify taxonomy/METHOD.md clause mappings against primary standard texts (specifically auditing all Z39.19 and ISO 25964-1/-2 pinpoints); review the 94 legacy compound concept labels matching punctuation-safe \b(?:and|versus|vs)\b\.? (95 including \bor\b) and 16 legacy compound non-preferred entry terms (aliases, 110 combined across all 2,129 preferred labels and aliases) against Z39.19 §7; audit legacy aliases item-by-item across all 1,251 non-preferred entry terms (2,129 total preferred labels + entry terms, recorded in v1_legacy_alias_dispositions inside the content-addressed governance_bundle_relpath verified by verify_release_governance_receipt, including all 7 pre-sidecar 0.4-draft concepts / 8 legacy aliases tracked under grandfathered_draft_alias_exceptions in taxonomy/term-provenance.yml with status pre-sidecar-legacy-queued-for-v1-audit — broader assurance case on mlsys.safety-case, implementation name XGBoost on mlsys.tree-ensemble, unattested grammar-guided decoding on mlsys.constrained-decoding, named algorithm/mechanism aliases ZeRO and FSDP on mlsys.state-sharding, PagedAttention on mlsys.paged-kv, CUDA Graphs on mlsys.graph-replay, and CUDA streams on mlsys.device-streams, each of which must either acquire full Gate 5 primary_excerpt attestation or be removed with tombstones prior to 1.0) against Z39.19 §8.2 and Gate 3; review prospective 1.0 RetiredConcept schema slots (resolution_class, split_into); and approve the Stage 2 human indexing pre-registration before recruitment.
    • V2 (SKOS / RDF / Crosswalk Packet): verify mlsys-taxonomy.ttl, mlsys-taxonomy-relations.ttl, mlsys-taxonomy.dcat.ttl, URI dereferencing, conditional-edge RDF caveats, and SKOS crosswalks (ACM CCS, CS2023, SFIA 9, ESCO).
  3. Dual-Human Inter-Indexer Consistency Study (Z39.19 §11.2.2, V1-approved pre-registration required before recruitment):
    • Fixed 12-indexer (6-pair) staffing & Editor-independence exclusion: Two independent human indexers (PhD candidates, postdocs, or staff ML systems engineers who did not author the taxonomy, are distinct natural persons disjoint from all 29 standing/specialist board roles, the convenor, nominating panel, steering advisor, and any ad-hoc escalation voters, are not current or former PhD/postdoc advisees, lab members, or institutional direct reports of the Editor, do not share an institution with the Editor, and have zero co-authorship with the Editor within the trailing 5 years [60 months], matching §2/§3's editor_coauthor_5yr exclusion window) are recruited per expertise stratum — F (3 areas), S (4 areas), P (5 areas), Q (6 areas), T1 (1 area), and T2 (1 area) — totaling 6 × 2 = 12 distinct indexers (0 overlap across pairs). The 6-pair design is fixed (changing pair count would alter the 40 × 6 cross-group anchor incidence matrix and 8-rater statistics below, and is disallowed without a new V1-approved pre-registration prior to recruitment).
    • Area-balanced allocation & uniform effective_lineage_component_id independence unit (30 Stage 1 calibration + 240 unique Stage 2 items = 12 per area across all 20 areas):
      • Global effective_lineage_component_id assignment: Before any Stage 1 or Stage 2 sampling, compute_lineage_components (binder/cli/core/taxonomy_schema.py) resolves all transitive cluster_merges, enforces the global 1 work_id <-> 1 canonical provenance_family_id bijection, and computes connected components using separated personal-author overlap (|A_i \cap A_j| / \min(|A_i|, |A_j|) \ge 0.50 across cluster rosters and per-work work_rosters) or shared institutional_organizations (|O_i \cap O_j| \ge 1) across the full candidate pool plus burned-clusters.json (schema_version: 3), assigning every external unit a deterministic effective_lineage_component_id (eff_lineage:<lexicographically_smallest_canonical_pf>) (for example, in the 382-unit Phase 3–7 study, 88 canonical provenance_family_id clusters = 39 internal + 49 external collapse to 41 effective lineage components = 1 internal + 40 external). All three independence boundaries below — Stage 1/Stage 2/burned-dev split disjointness, the within-area Stage 2 sampling cap, and the 10,000-resample cluster bootstraps — operate uniformly on effective_lineage_component_id, and study receipts report both canonical provenance_family_id count (M_{\text{canon}}) and effective_lineage_component_id count (M_{\text{eff}}) separately.
      1. Stage 1 (Calibration, 30 items = 5 per pair, component-disjoint from Stage 2 and burned dev): Drawn from unexposed non-editor-affiliated origin: external sources completely disjoint by effective_lineage_component_id from Stage 2's evaluation components and disjoint from every effective_lineage_component_id containing any burned cluster in burned-clusters.json. Indexers practice applying tagging_rules (taxonomy/README.md across all 101 subjects), reconcile rule misunderstandings with V1, and freeze the indexing guide before Stage 2 packets are unsealed. Stage 1 items and effective_lineage_component_ids are excluded from Stage 2 scoring.
      2. Stage 2 Primary Allocation (12 unique items/area \times 20 areas = 240 unique items, <= 1 item per effective_lineage_component_id per area, M_{\text{canon}} \ge 120, M_{\text{eff}} \ge 120, max_areas_per_component <= 4): Drawn from unexposed non-editor-affiliated origin: external post-cutoff / reserve sources (completely disjoint by effective_lineage_component_id from Stage 1 and from all pre-existing clusters in taxonomy/method-study/burned-clusters.json, requiring at least M_{\text{canon}} \ge 120 canonical external clusters and M_{\text{eff}} \ge 120 distinct external effective_lineage_component_ids across the 240 items, capping each component's cross-area span at max_areas_per_component <= 4 distinct areas, and sampling at most 1 item per effective_lineage_component_id within any single area so every one of the 20 areas spans 12 distinct effective_lineage_component_ids [<= 1/12 (~8.33%) exact within-area share per component]; if any area has < 12 eligible unexposed external effective_lineage_component_ids, M_{\text{eff}} < 120, or any component spans > 4 areas prior to unsealing, reconcile.py --unseal-stage2-human halts before unsealing until additional independent external works are curated into the candidate pool). Allocating by area first (12 allocated items in every one of the 20 areas) exceeds the >= 10 allocated items/area design rule everywhere (including P's 5 areas and Q's 6 areas) and provides a 4-item buffer per area above the separate >= 8 jointly non-empty (both_non_empty) per-area evaluation floor (80% non-empty \times 12 = 9.6 \ge 8). Primary-pair item counts sum to 240: F = 36 (3×12), S = 48 (4×12), P = 60 (5×12), Q = 72 (6×12), T1 = 12 (1×12), T2 = 12 (1×12).
      3. Deterministic Item-Level 40-Anchor Cross-Group Routing (2 anchors/area = 40 anchor items \times 3 secondary pairs = 120 secondary evaluations; 360 total pair-evaluations): Within each of the 20 areas, the first 2 stratified items sorted by (area_code, unit_id) are designated as anchors (40 anchor items total: Q = 12, P = 10, S = 8, F = 6, T1 = 2, T2 = 2). Within each source group g (k = 0 \dots n_g - 1 sorted by (area_code, unit_id)), item k is routed deterministically to the 3 distinct non-primary pairs having the largest remaining quota in group $g$'s target row vector, breaking ties by canonical pair order [F, S, P, Q, T1, T2], producing the exact item-level 3-receiver schedule below (row sum = 3 per anchor item, column sum = 20 incoming anchor items for every one of the 6 pairs):
        • Q (12 anchors, row sums {F: 8, S: 7, P: 7, T1: 7, T2: 7}): k=0:[F,S,P], k=1:[F,T1,T2], k=2:[F,S,P], k=3:[F,T1,T2], k=4:[S,P,T1], k=5:[F,S,T2], k=6:[P,T1,T2], k=7:[F,S,P], k=8:[F,T1,T2], k=9:[S,P,T1], k=10:[F,S,T2], k=11:[P,T1,T2] (36 secondary evaluations).
        • P (10 anchors, row sums {F: 6, S: 6, Q: 6, T1: 6, T2: 6}): k=0:[F,S,Q], k=1:[F,T1,T2], k=2:[S,Q,T1], k=3:[F,S,T2], k=4:[Q,T1,T2], k=5:[F,S,Q], k=6:[F,T1,T2], k=7:[S,Q,T1], k=8:[F,S,T2], k=9:[Q,T1,T2] (30 secondary evaluations).
        • S (8 anchors, row sums {F: 4, P: 5, Q: 5, T1: 5, T2: 5}): k=0:[P,Q,T1], k=1:[F,P,T2], k=2:[Q,T1,T2], k=3:[F,P,Q], k=4:[F,T1,T2], k=5:[P,Q,T1], k=6:[F,P,T2], k=7:[Q,T1,T2] (24 secondary evaluations).
        • F (6 anchors, row sums {S: 5, P: 5, Q: 5, T1: 1, T2: 2}): k=0..3:[S,P,Q], k=4:[S,P,T2], k=5:[Q,T1,T2] (18 secondary evaluations).
        • T1 (2 anchors, row sums {F: 1, S: 1, P: 2, Q: 2, T2: 0}): k=0:[F,P,Q], k=1:[S,P,Q] (6 secondary evaluations).
        • T2 (2 anchors, row sums {F: 1, S: 1, P: 1, Q: 2, T1: 1}): k=0:[F,S,Q], k=1:[P,Q,T1] (6 secondary evaluations). Group and area headers are masked on all item packets, items are shuffled within each pair's packet (total evaluations per pair: F = 56, S = 68, P = 80, Q = 92, T1 = 32, T2 = 32; sum = 360), and indexers choose freely from all 101 subjects and 878 concepts (1 primary subject, 0–3 concepts).
    • Executable weighted multi-pair estimators, separated 240-item (seed = 20270101) & 40-anchor (seed = 20270102) cluster bootstraps, mandatory unit-test fixtures, and raw-bundle 1.0 gate:
      • Headline 240-item primary-pair cluster bootstrap (seed = 20270101) & separate 40-anchor pooled cluster bootstrap (seed = 20270102): Headline release-gate point estimates evaluate the fixed 240 primary-pair item assignments (N = 240, r_i = 2 raters per item from pre-designated primary pair P_g = (r_{g,1}, r_{g,2}), 12 items/area \times 20 areas across 12 distinct effective_lineage_component_ids per area). Because an effective_lineage_component_id can span multiple areas across different groups while each area contains only 2 anchor items (so requiring every area's 2 anchors to survive inside the 240-item primary bootstrap would reject \approx 1 - (1 - (1 - 2/M)^M)^{20} of draws even on 240 independent components), primary-pair and 40-anchor uncertainty intervals use two separate pre-registered 10,000-resample cluster bootstraps:
        1. Primary 240-item area-post-stratified bootstrap (seed = 20270101): Over the M_{\text{eff}} \ge 120 distinct external effective_lineage_component_ids E_1, \dots, E_{M_{\text{eff}}} spanning the 240 Stage 2 items (max_areas_per_component <= 4, 12 distinct components in every area), each resample draws (k_1, \dots, k_{M_{\text{eff}}}) \sim \text{Multinomial}(M_{\text{eff}}; 1/M_{\text{eff}}, \dots, 1/M_{\text{eff}}) (\sum_{m=1}^{M_{\text{eff}}} k_m = M_{\text{eff}}), assigns each item i \in E_{m(i)} raw multiplicity w_i^* = k_{m(i)}, post-stratifies within area a(i) \in \{1,\dots,20\} to normalized weight \tilde{w}_i^* = \frac{1/20}{\sum_{j \in \text{area } a(i)} w_j^*} w_i^* (\sum_{i=1}^{240} \tilde{w}_i^* = 1), and scales to sample-size item weight $W_i = N,\tilde{w}_i^ = 240,\tilde{w}_i^*$* (\sum_{i=1}^{240} W_i = 240, which equals W_i \equiv 1 at the point estimate k_m \equiv 1, preserving exact 1/20 area weights and n_g / 240 group weights N_g = \sum_{i \in I_g} W_i = n_g in every valid resample). Primary degenerate-resample rule (no anchor-survival condition): if a draw has \sum_{j \in \text{area } a} w_j^* = 0 in any area a \in \{1,\dots,20\}, fewer than 2 active resampled components (|\{m : k_m > 0\}| < 2), or weighted conditional non-empty denominator \sum_{i \in \mathcal{B}} W_i = 0 where \mathcal{B} = \{i : A_{i1} \ne \emptyset \land A_{i2} \ne \emptyset\}, increment degenerate_resample_count and redraw from seed = 20270101 until 10,000 valid resamples complete (degenerate_resample_count <= 100; at the worst-case boundary M_{\text{eff}} = 120 with 12 distinct components per area, the exact single-area empty-cell probability is (1 - 12/120)^{120} = 0.9^{120} \approx 3.23 \times 10^{-6} and the 20-area union bound is \le 20 \times 0.9^{120} \approx 6.46 \times 10^{-5}, or \approx 0.65 expected redraws per 10,000 resamples; at M_{\text{eff}} = 240, (1 - 12/240)^{240} \approx 4.50 \times 10^{-6} per area). Across the 10,000 valid draws sorted ascending as d_0 \le d_1 \le \dots \le d_{9999} for each of the 7 primary Stage 2 metrics (alpha_subject, kappa_bar_subject, cond_hooper, all_hooper, cond_rolling, all_rolling, alpha_masi), the two-sided percentile 95% CI is [d_250, d_9749] (250 draws in each tail).
        2. Separate 40-anchor 8-rater pooled cluster bootstrap (seed = 20270102): Over the M_{\text{eff,anchor}} distinct external effective_lineage_component_ids spanning the 40 anchor items, each resample draws (k_1^{\text{anchor}}, \dots, k_{M_{\text{eff,anchor}}}^{\text{anchor}}) \sim \text{Multinomial}(M_{\text{eff,anchor}}; 1/M_{\text{eff,anchor}}, \dots, 1/M_{\text{eff,anchor}}), sets w_i^* = k_{m(i)}^{\text{anchor}} for i \in \{1,\dots,40\}, and scales globally across all 40 anchors to $W_{i,\text{anchor}} = 40,\frac{w_i^}{\sum_{j=1}^{40} w_j^}$ (\sum_{i=1}^{40} W_{i,\text{anchor}} = 40, avoiding $2$-anchor per-area zero-cell rejection), incrementing anchor_degenerate_resample_count and redrawing from seed = 20270102 only if \sum_{j=1}^{40} w_j^* = 0 or |\{m : k_m^{\text{anchor}} > 0\}| < 2 (anchor_degenerate_resample_count <= 100), reporting [d_250, d_9749] percentile 95% CIs across all 3 40-anchor 8-rater metrics (alpha_subject_anchor, alpha_masi_anchor, kappa_fleiss_anchor).
      • Exact weighted Krippendorff's \alpha (101-subject nominal \alpha_{\text{subject}} and set-valued \alpha_{\text{MASI}}): Let \delta(v, v') \in [0, 1] denote the linear disagreement metric entering Krippendorff's coincidence sum directly (never squared a second time): \delta_{\text{nom}}(c, k) = \mathbb{I}[c \ne k] on 101-subject IDs for \alpha_{\text{subject}}, and Passonneau's linear MASI distance \delta_{\text{MASI}}(A, B) = 1 - J(A, B)\,M(A, B) on concept sets (including \emptyset, with \delta_{\text{MASI}}(\emptyset, \emptyset) = 0 and \delta_{\text{MASI}}(A, \emptyset) = 1 for A \ne \emptyset, matching build_study_receipts.py and reconcile.py) for \alpha_{\text{MASI}}. Over items i \in \{1,\dots,N\} with r_i \ge 2 raters (r_i = 2 on the N = 240 primary-pair items with W_i; r_i = 8 on the N = 40 anchor items with pooled W_{i,\text{anchor}} = 40\,w_i^* / \sum_{j=1}^{40} w_j^*), define weighted pairable value count N_{\text{val}} = \sum_{i=1}^N W_i\,r_i (N_{\text{val}} = 480 on the 240 paired items; N_{\text{val}} = 320 on the 40 8-rater anchors), weighted coincidence entry o_{ck} = \sum_{i=1}^N W_i \frac{n_{ic}(n_{ik} - \mathbb{I}[c = k])}{r_i - 1} where n_{ic} = \sum_{u=1}^{r_i} \mathbb{I}[v_{iu} = c], value marginal n_c = \sum_k o_{ck} = \sum_{i=1}^N W_i\,n_{ic} (\sum_c n_c = N_{\text{val}}), weighted observed disagreement D_o = \frac{1}{N_{\text{val}}} \sum_{c \ne k} o_{ck}\,\delta(c, k) = \frac{1}{N_{\text{val}}} \sum_{i=1}^N \frac{W_i}{r_i - 1} \sum_{1 \le u \ne u' \le r_i} \delta(v_{iu}, v_{iu'}), and finite-sample expected disagreement D_e = \frac{1}{N_{\text{val}}(N_{\text{val}} - 1)} \sum_{c \ne k} n_c\,n_k\,\delta(c, k), yielding \alpha = 1 - D_o / D_e. Pair-stratified Cohen's $\bar{\kappa}_{\text{subject}}$: within each group g \in \{F, S, P, Q, T1, T2\} (N_g = \sum_{i \in I_g} W_i = n_g), p_{o,g} = \frac{1}{N_g}\sum_{i \in I_g} W_i\,\mathbb{I}[v_{i1} = v_{i2}], p_{e,g} = \sum_s \bigl(\frac{1}{N_g}\sum_{i \in I_g} W_i\,\mathbb{I}[v_{i1}=s]\bigr)\bigl(\frac{1}{N_g}\sum_{i \in I_g} W_i\,\mathbb{I}[v_{i2}=s]\bigr), \kappa_g = (p_{o,g} - p_{e,g}) / (1 - p_{e,g}), and \bar{\kappa}_{\text{subject}} = \sum_g (N_g / N)\,\kappa_g. Degenerate-marginal convention: if D_e = 0 (or p_{e,g} = 1, or \bar{P}_e = 1 in Fleiss' \kappa), define \alpha = 1.0 (\kappa = 1.0) when D_o = 0 (p_{o,g} = 1) and 0.0 when D_o > 0, flagging any zero-variance stratum explicitly in the study report.
      • Cross-pair 40-anchor 8-rater reliability & weighted overlap estimators: Across the 40 anchor items (4 pairs = r = 8 human indexers per item, weighted by pooled W_{i,\text{anchor}} with \sum_{i=1}^{40} W_{i,\text{anchor}} = 40), report 8-rater Krippendorff's \alpha_{\text{subject,anchor}} and \alpha_{\text{MASI,anchor}} via the r_i = 8 formula above alongside weighted 8-rater Fleiss' \kappa_{\text{Fleiss}} = (\bar{P}_o - \bar{P}_e)/(1 - \bar{P}_e) where \bar{P}_o = \frac{1}{40}\sum_{i=1}^{40} W_{i,\text{anchor}} \frac{1}{56}\sum_s n_{is}(n_{is}-1) and \bar{P}_e = \sum_s \bigl(\frac{1}{320}\sum_{i=1}^{40} W_{i,\text{anchor}} n_{is}\bigr)^2. On the 240 primary-pair items, with h_i = |A_{i1} \cap A_{i2}| / |A_{i1} \cup A_{i2}| and q_i = 2|A_{i1} \cap A_{i2}| / (|A_{i1}| + |A_{i2}|) on \mathcal{B} = \{i : A_{i1} \ne \emptyset \land A_{i2} \ne \emptyset\} (extended to all 240 items via h_i = q_i = 1 when A_{i1} = A_{i2} = \emptyset and 0 when exactly one is empty), conditional Hooper and Rolling are (\sum_{i \in \mathcal{B}} W_i h_i) / (\sum_{i \in \mathcal{B}} W_i) and (\sum_{i \in \mathcal{B}} W_i q_i) / (\sum_{i \in \mathcal{B}} W_i), and all-item Hooper and Rolling divide by \sum_{i=1}^{240} W_i = 240 (co-reported alongside equal-area 20-area macro-averages). Leaf-concept specificity ratio (leaf_specificity) is the share of assigned concept tags whose concept has 0 incoming specializes children in ml-systems-taxonomy.yml. Five mandatory reference-implementation unit-test fixtures (run_stage2_estimator_self_tests in binder/cli/core/taxonomy_schema.py) must pass before Stage 2 unsealing and on every governance verification: (1) orientation-swap invariance (swapping (v_{i1}, v_{i2}) on any item subset leaves \alpha_{\text{subject}}, \alpha_{\text{MASI}}, Hooper, Rolling, and \kappa_{\text{Fleiss}} invariant to < 10^{-12}), (2) uniform-weight identity (W_i \equiv 1 equals unweighted \alpha_{\text{subject}}, \alpha_{\text{MASI}}, \bar{\kappa}_{\text{subject}}, Hooper, and Rolling to < 10^{-12}), (3) integer cluster-replication equivalence (unstratified integer weights W_i = k_i \in \mathbb{Z}_{\ge 0} with \sum_{i=1}^N k_i = N equal unweighted \alpha_{\text{subject}} and \alpha_{\text{MASI}} on the physically duplicated $N$-item dataset to < 10^{-12}), (4) zero-variance boundary (D_e = 0, D_o = 0 \implies \alpha = 1.0; p_{e,g} = 1, p_{o,g} = 1 \implies \kappa_g = 1.0), and (5) worst-case multi-area boundary completion (M_{\text{eff}} = 120, max_areas_per_component = 4, 12 distinct components/area completes 10,000 valid resamples from seed = 20270101 with degenerate_resample_count <= 100 and verifies (1 - 12/120)^{120} < 10^{-5}).
      • Pre-registered 1.0 human consistency floors, raw-bundle recomputation, & append-only single-recalibration cap (enforced mechanically by verify_release_governance_receipt in binder/cli/core/taxonomy_schema.py): verify_release_governance_receipt loads and SHA-256-verifies the content-addressed governance_bundle_relpath under taxonomy/method-study/ (checking distinct-person rosters across convenor [1], 3 nominating_panel members, steering_advisor [1], 16 standing_roster Chairs/Co-Chairs/Alternates, 13 sub_area_specialists, any temporary_proxies, substitute_specialists, or adhoc_voters, and 12 human_indexers [>= 46 distinct non-Editor individuals]; Editor and non-Editor single-institution diversity caps, 5-year COI sub-caps, and editor_advisee_or_lab_or_report == false on indexers; charter_attestation selection traces and >= 3-voter appointment_votes; vote_record.seat_votes across all 7 seats supporting "approve", "reject", and "abstain" with mandatory t1_voter_person_id, t2_voter_person_id, t1_vote, and t2_vote on Seat T; content-addressed reports across all 13 sub_area_specialists [plus any substitute_specialist_reports] and 8 chair_area_signoffs, vote_record.item_level_votes, v1_standards_verification, v2_skos_verification, and queue_dispositions resolving every zero_warrant_concepts [>= 27] and split_hierarchy_edges [>= 16] queue item; item-by-item v1_legacy_alias_dispositions covering all 7 grandfathered concepts and every unattested active legacy entry term; calibrated seeded_gap_audit sensitivity \ge 85.0\% over \ge 10 ablated concepts across all eligible positive probes; 30 component-disjoint Stage 1 calibration items [plus 12 recalibration_slice_items on recalibration_rerun_index == 1]; 240 Stage 2 primary items = 12 per area across 12 distinct effective_lineage_component_ids per area with M_{\text{canon}} \ge 120, M_{\text{eff}} \ge 120, and max_areas_per_component <= 4, verified under strict_prospective_work_bijection=True against candidate_pool_manifest and pre-burned first_burned_by_stage2_attempt_id entries in burned-clusters.json; exact 40-anchor 8-rater schedule; and finite numeric domain checks rejecting NaN/Infinity and 1e999 float overflow) and recomputes all Stage 2 metrics (leaf_specificity, zero_variance_strata, anchor_zero_variance_flags) and all 10 [d_250, d_9749] percentile 95% CIs across both 10,000-resample bootstraps (seed = 20270101 and seed = 20270102) directly from raw item-level ratings: orientation-invariant 101-subject \alpha_{\text{subject}} \ge 0.65 and pair-stratified \bar{\kappa}_{\text{subject}} \ge 0.65, conditional Hooper \ge 45.0\% (all-item Hooper \ge 40.0\%), conditional Rolling \ge 60.0\% (all-item Rolling \ge 55.0\%), pooled \alpha_{\text{MASI}} \ge 0.45, empty-assignment rate \le 20.0\% (\ge 192 of 240 items jointly non-empty and \ge 8 of 12 items jointly non-empty in every one of the 20 areas), and max single-concept share p_{\max} \le 15.0\%. Attempt counts are pre-registered and enforced before packet exposure via reconcile.py --unseal-stage2-human against the append-only stage2_human_attempts ledger in burned-clusters.json (keyed by (cycle_id, target_major_minor), blocking both campaign-rename and RC-tag resets): if any floor or per-area minimum fails, 1.0 release halts, offending subject scope notes or tagging rules are adjudicated on a separate effective_lineage_component_id-disjoint 12-item calibration slice, and Stage 2 may be rerun at most once (recalibration_rerun_index \in \{0, 1\}, max 1 recalibration rerun per (cycle_id, target_major_minor)) on fresh unexposed external effective_lineage_component_ids disjoint from Attempt 0; if the rerun still misses any floor, 1.0 halts for a full taxonomy structural revision cycle rather than repeated re-testing.

5. Normative 1.0 Governance Receipt and Raw Bundle Contract (verify_release_governance_receipt)

Non-draft releases (status: normative-release, version without -draft, such as 1.0-rc1 or 1.0) fail closed inside apply.py and ./binder/binder concepts check (validate_term_provenance_sidecar -> verify_release_governance_receipt in binder/cli/core/taxonomy_schema.py) unless:

  1. grandfathered_draft_alias_exceptions in taxonomy/term-provenance.yml is empty ({}), and model["status"] == "normative-release".
  2. release_governance_receipt (schema_version: 1) binds cycle_id, target_version, candidate_taxonomy_sha256, candidate_non_guides_taxonomy_sha256, governance_bundle_relpath (confined symlink-free under taxonomy/method-study/), and governance_bundle_sha256, plus top-level cross-checked summaries board_signoff (seated_voting_members == 7, 5 <= affirmative_votes <= 7, v1_approved: true, v2_approved: true, sub_area_specialist_reports == 13, active_editor_institution_voters <= 1, active_editor_coauthor_voters <= 2, charter_sha256, vote_record_sha256) and human_study_stage2 (study_bundle_sha256 == governance_bundle_sha256, n_items == 240, n_raters == 12, canonical_provenance_clusters >= 120, effective_lineage_components >= 120, max_areas_per_component <= 4, bootstrap_resamples == 10000, all 7 primary metrics + empty_rate <= 0.20 + min_area_both_non_empty >= 8 + p_max <= 0.15 + leaf_specificity + seeded_gap_sensitivity >= 0.85, all 3 anchor metrics, zero_variance_strata, anchor_zero_variance_flags, primary_bootstrap_95_ci, anchor_bootstrap_95_ci, degenerate_resample_count <= 100, anchor_degenerate_resample_count <= 100, recalibration_rerun_index \in \{0, 1\}).
  3. The raw JSON bundle at governance_bundle_relpath (schema_version: 1, cycle_id, target_version, candidate_taxonomy_sha256, candidate_non_guides_taxonomy_sha256, editor_person_id) contains and passes full recomputation across:
    • charter_attestation: charter_relpath == "taxonomy/method-study/candidate-experts.md", live charter_file_sha256, convenor_selection_trace (precedence_committee in {"NISO_Z39_19", "ISO_TC46_SC9_WG8", "W3C_SKOS", "DCMI_USAGE_BOARD", "ISKO_ASIST_FALLBACK"}, signed_coi_declaration_sha256, pre_outreach_published_at <= panel_outreach_started_at), 3-entry nominating_panel_selection_trace (venue_slot across MLSys, OSDI_SOSP, KOS_SKOS), and signed_by_person_ids.
    • Distinct natural-person rosters (>= 46 non-Editor individuals disjoint from editor_person_id, each carrying person_id, institution, optional secondary_institutions, five_year_coi_attested: true, editor_institution, editor_coauthor_5yr, and editor_advisee_or_lab_or_report) across convenor (1, voting: false), nominating_panel (3, voting: false), steering_advisor (1, voting: false), optional temporary_proxies, standing_roster (16 (seat, role) slots), sub_area_specialists (13 specialist_slot entries), optional substitute_specialists, optional adhoc_voters, and human_indexers (12 indexers across 6 pairs F, S, P, Q, T1, T2, pair_coauthored_5yr: false, editor_advisee_or_lab_or_report: false).
    • appointment_votes: recorded >= 3-voter, >= 2/3-supermajority unconflicted appointment votes (appointee_person_id, voters with voter_person_id, vote, and coauthored_5yr_with_appointee: false, disjoint institutions, zero self-votes) covering steering_advisor, any temporary_proxies, all 16 standing_roster slots (V1/V2 requiring >= 2 nominating_panel voters alongside steering_advisor/panel; domain seats, specialists, substitute specialists, indexers, and ad-hoc voters seated from {V1, V2, steering_advisor, nominating_panel, temporary_proxies}), all 13 sub_area_specialists, any substitute_specialists, any adhoc_voters, and all 12 human_indexers.
    • vote_record: seat_votes across all 7 seats (F, S, P, Q, T, V1, V2, recused: false, vote in {"approve", "reject", "abstain"}), with mandatory t1_voter_person_id, t2_voter_person_id, t1_vote, and t2_vote on Seat T (seat_t_t1_concurred, seat_t_t2_concurred), enforcing <= 1 Editor-institution, <= 2 single-institution, and <= 2 Editor-coauthor active-voting caps across all 8 active voting holders, optional item_level_votes (escalation_3voter / full_board_item) and dissent_records, and unresolved_binding_rejections == [].
    • Content-addressed specialist, chair, standards, SKOS, and queue artifacts: 13 sub_area_specialists JSON reports (report_relpath, report_sha256, reviewed_concept_ids, plus any substitute_specialist_reports covering recused_concept_ids) and 8 chair_area_signoffs (F1, F2, S1, S2, S3, P4, T1, T2) jointly covering every active concept across all 20 areas; v1_standards_verification (primary_standard_texts_verified: true, editor_coi_dispositions_confirmed: true, compound_term_audit_completed: true, checked_standards); v2_skos_verification (ttl_and_dcat_verified: true, checked_crosswalks covering ACM CCS, CS2023, SFIA 9, ESCO); and queue_dispositions adjudicating all >= 16 split_hierarchy_edges and >= 27 zero_warrant_concepts queue items.
    • v1_legacy_alias_dispositions: item-by-item V1 dispositions (uphold, remove, reclassify-or-split, v1_signed_off: true, >= 15-char rationale) covering all 7 grandfathered concepts / 8 aliases and every active legacy entry term not yet attested in term-provenance.yml.
    • seeded_gap_audit: calibrated two-judge (judge_ids) ablation sensitivity >= 85.0% across >= 10 active ablated_concept_ids over the complete cross-product baseline_coverage_judgments (>= 10 units \times judges) and complete seeded_gap_items denominator.
    • stage2_study: protocol_approval (v1_approver_person_id, SHA-256-bound frozen_tagging_guide_relpath and candidate_pool_manifest_relpath verified under strict_prospective_work_bijection=True, ordered UTC timestamps protocol_approved_at <= stage1_completed_at <= guide_frozen_at <= unsealed_at <= collection_completed_at), recalibration_rerun_index, stage1_calibration_items (30 items, 5/pair), optional recalibration_slice_items (12 items on recalibration_rerun_index == 1 paired with prior_failed_bundle_relpath / prior_failed_bundle_sha256), and stage2_items (240 items = 12/area across 12 distinct effective_lineage_component_ids/area, M_{\text{canon}} \ge 120, M_{\text{eff}} \ge 120, max_areas_per_component <= 4, carrying primary_evaluation and secondary_evaluations on the 40 anchors), matching human_study_stage2 recomputed point estimates, leaf_specificity, zero_variance_strata, anchor_zero_variance_flags, and all 10 [d_250, d_9749] percentile 95% CIs to <= 1e-9.

6. Pre-Unseal Stage 2 Human Ledger Gate (reconcile.py --unseal-stage2-human)

Before any human indexer sees Stage 1 or Stage 2 packets, the study coordinator must execute:

python3 .claude/workflows/taxonomy-refresh/reconcile.py --unseal-stage2-human <stage2-plan.json>

Under repo_apply_lock(), unseal_stage2_human_attempt validates <stage2-plan.json> (schema_version: 1, attempt_id, cycle_id, target_major_minor, recalibration_rerun_index \in \{0, 1\}, unsealed_at [with optional protocol_approved_at <= stage1_completed_at <= guide_frozen_at <= unsealed_at], candidate_pool_manifest_relpath, frozen_tagging_guide_relpath, stage1_unit_ids [30], stage2_unit_ids [240], and recalibration_unit_ids [0 on Attempt 0, 12 on Attempt 1]):

  1. Enforces at most 2 attempts (recalibration_rerun_index 0 and 1) for any (cycle_id, target_major_minor) pair across burned-clusters.json, requiring Attempt 1 to be completely disjoint by effective_lineage_component_id from Attempt 0.
  2. Runs compute_lineage_components(..., strict_prospective_work_bijection=True) over the full candidate_pool_manifest plus burned-clusters.json, verifying 1 work_id <-> 1 canonical provenance_family_id bijection, zero overlap with pre-existing burned components, pairwise component disjointness across Stage 1, recalibration, and Stage 2, M_{\text{canon}} \ge 120, M_{\text{eff}} \ge 120, max_areas_per_component <= 4, and 12 distinct components per area.
  3. Atomically appends the attempt record to stage2_human_attempts, adds all Stage 1, recalibration, and Stage 2 canonical clusters to burned_or_exposed_cluster_ids, stamps first_burned_by_stage2_attempt_id = attempt_id in exposed_cluster_metadata (permitting verify_release_governance_receipt to distinguish pre-existing burned clusters from clusters burned by attempt_id itself while permanently burning aborted attempts), and writes <stage2-plan>.unseal-receipt.json.
Reference in New Issue View Git Blame Copy Permalink