resolve_precision('fp32') yields key 'fp32', which was absent from the H100/H200 precision_flops map (stored as 'fp32_cuda' = 67 TFLOP/s), so Engine.solve fell through to peak_flops (989, the FP16 dense rate) -- a ~15x FP32 overestimate reachable by no precision string.
- Rename precision_flops key fp32_cuda -> fp32 on H100 and H200 (value 67 TFLOP/s unchanged)
- Propagate the rename to every consumer, value-preserving: 2 LEGO cells
(compute_infrastructure, appendix_assumptions), 2 mlsysim_constants audit
manifests, and the migrate-constants map
- Add test_hardware guards: precision-vocabulary check (no canonical precision
aliased away) + H100/H200 fp32 -> 67 regression
Add native Binder checks for LEGO formatting/unit/prose contracts, extend mlsysim formatter helpers, and normalize Vol1/Vol2 LEGO output strings through typed formatters.
Validate precision-sensitive prose rendering across both volumes and promote direct math/string assembly to fmt_math, fmt_display_math, fmt_text, and domain-specific helpers.
Hardware specs already get sanity-bounds protection in
test_physics_bounds.py (catching an 80 TB/s typo for 80 GB/s), but
registered model specs had no equivalent guard: a transposed parameter
count (124M typed as 421M) or a decimal slip (124 typed as 12.4) in a
registry YAML would bind silently.
Two layers now protect every TransformerWorkload:
- test_transformer_attention_geometry: universal invariant that
hidden_dim splits evenly into heads with a 32-256 head dim and is a
multiple of 64 (catches transposed hidden_dim/heads across all 13
registered transformers).
- test_transformer_curated_specs: a curated, web-verified spec table
(seeded with GPT-2 Small, Radford 2019) that fails CI when a registry
value diverges >1% from its verified figure. Extend the table whenever
a new model spec is registered.
Wire GPT2_Small into the two chapter-side checks so its specs cannot be
hardcoded into a LEGO cell: CANONICAL in audit_mlsysim_drift.py and
HARDCODED_REGISTRY in book_check_registry_sources.py.
Verified the guard fires: transposing 124M to 421M fails the curated
test, and swapping hidden_dim/heads fails the geometry invariant.
NVIDIA quotes NVLink (and Google quotes TPU ICI) as bidirectional totals,
while PCIe and fabric entries store per-direction rates — two conventions
under one field name, undeclared. IOInterconnect now carries an explicit
direction field ('per_direction' default; 'bidirectional_total' tagged on
all NVLink/ICI YAML entries) and a bandwidth_per_direction accessor.
Systems.Nodes intra_node_bw now feeds the per-direction rate (H100 450,
A100 300, B200 900 GB/s) into the collective beta terms — previously the
bidirectional total made every intra-node-bound allreduce/TP latency ~2x
optimistic. Stored datasheet figures are unchanged, so spec displays keep
the familiar marketing numbers.
Distributed golden re-pinned to the corrected physics (tp comm 267->535 ms,
scaling efficiency 0.809->0.761 on the Llama3-8B/Research_256 case); new
convention test pins the tagging and accessor; drift auditor gains
per-direction entries.
Audit 2026-06-09, findings_provenance.md M1/M2. Ruling: VJ 2026-06-10.
Systems.NetworkEnergy.Per1Kb had zero consumers and its implied 1,000
pJ/byte contradicted Hardware.Tech.Movement.Network's canonical 10,000
pJ/byte by 10x — a competing constant waiting to be picked up. Datacenter
network movement energy lives in Hardware.Tech.Movement. Also cross-pin
cal.REFERENCE_HARDWARE_TFLOPS (989.0, a bare code float) to
Hardware.Cloud.H100.compute.peak_flops so the duplicate cannot drift from
the registry silently.
Audit 2026-06-09, findings_discipline.md.
calc_fleet_tco now uses the explicit 365-day year (8,760 h) matching
DAYS_PER_YEAR and every book LEGO cell, instead of Pint's Julian year
(8,766 h) — the test tolerance could not distinguish the two; now pinned at
rel 1e-6 with the 10x-wrong comment arithmetic corrected. Docstring states
the device-level scope (full CapEx up front, no PUE/maintenance) and points
to EconomicsModel for facility-level TCO; negative-input guards added.
EconomicsModel docstring documents amortization_years/-multiplier. EUR unit
alias commented as a 1:1 parity alias. Engine: non-Transformer training
fallback now accounts weights + gradients + Adam state (12 B/param) instead
of 3x weights (~2.7x understated); offload PCIe fallback reads
cal.FALLBACK_PCIE_BANDWIDTH_GB_S instead of an inline literal; memory
traffic computed once for both the latency term and the bottleneck
classifier; efficiency-derates-compute-only and weights-only inference
memory conventions documented. Golden pins added for mfu/hfu/energy
(ResNet-50/A100), batch-32 intensity, and the new training fallback.
Audit 2026-06-09, findings_formulas_economics_engine.md A1-A7, B2-B4, B6, B8-B9.
Full suite green.
Known-answer tests for calc_transformer_decode_flops (2P rule, 8e9 params ->
1.6e10 flop) and calc_checkpoint_size (default ZeRO 14 B/param -> 112 GB for
8B), pinning DecodeConstant=2.0 and the calibration byte conventions — all
previously had zero coverage, so a 2x YAML edit passed the suite while
silently changing vol1 conclusion and appendix_dam numbers. Input contracts:
calc_effective_flops and dTime now reject MFU/efficiency outside [0,1]
(returned 1.5x peak silently before); KV-cache functions validate dimensions
like their activation-memory sibling. Docstrings: decode-FLOPs long-context
caveat (2P excludes attention-over-KV terms), roofline ridge-point tie-break,
ms-basis epsilon note, BYTES_ADAM_STATE moments-only comment distinguishing
it from the 12/14 B master-copy conventions in calibration.
Audit 2026-06-09, findings_formulas_performance_memory.md P1-1..3, P2-1..4, P2-7.
Full suite green (789 tests).
Known-answer tests for calc_queue_latency_mmc (independent direct-summation
Erlang-C reference: C(2,1.5)=0.642857, p50=0.502629s, p99=8.326675s, plus
the M/M/1 reduction) — previously only qualitative assertions, so a dropped
(1-rho) or factor-of-2 wait rate passed CI. First package tests for
statistics.py (PSI known answer + symmetry, sample-size known answer +
inverse-square scaling, propagation factor) — previously pinned only by book
render guards. Contract guards: calc_failure_probability rejects negative
durations (returned -0.105 before), calc_mtbf_cluster rejects n=0 (bare
ZeroDivisionError before), Young-Daly warns when delta >= 2*MTBF (returned
physically impossible tau < delta silently). calc_mtbf_cluster docstring now
states the exponential assumption and correlation_factor direction; PSI
docstring defers thresholds to Ops.Monitoring instead of carrying a second
band convention; two-proportion guards baseline_rate/detectable_lift; M/M/c
quantiles normalize -0.0; TailLatencyModel documents its two quantile
approximations.
Audit 2026-06-09, findings_formulas_reliability_serving.md P1-1..3, P2-1..6.
Document the per-GPU-NIC assumption in hierarchical allreduce (correct only
when the g concurrent inter-node rings each have a dedicated NIC), the
validity regime of the ring/tree crossover estimate (large-N; ~40% high at
N=8; undefined at N=2), the pedagogical 1.2/1.05 double-binary-tree factors,
the links-across-cut semantics of bisection num_ports, the corrected
oversubscription docstring formula, and the deliberate omission of Thakur's
reduction term (module docstring). Add nonnegative message/latency validation
to ring/tree/all-to-all collectives for parity with point-to-point. Add a
known-answer test pinning every constant factor of hierarchical allreduce —
the one collective whose factors no test pinned (the pre-2026-06-06
inflated-intra-term bug would have passed the existing qualitative test).
Audit 2026-06-09, findings_formulas_communication.md F1-F7, F9.
MFU now counts only model FLOPs (excluding recomputation); HFU is computed
from actual hardware ops instead of a fabricated 1.1x ratio. hfu == mfu
without recomputation, 4/3 x mfu under full recompute. Recompute now pairs
full-recompute memory accounting with the 4x ops multiplier instead of
charging selective-checkpoint memory against full-recompute FLOPs. Energy
utilization switches to HFU (identical in all currently rendered paths).
Adds overhead_dominated flag for profiles where dispatch + layer tax
exceeds both roofline ceilings (ResNet-50/A100/b1 case). No rendered book
number changes: .hfu and activation_recomputation are unused in chapters.
Audit 2026-06-09, findings_formulas_economics_engine.md B1/B5/B7/B10.
- Migrated manual parameter footprint calculations to mlsysim.physics.memory_from_params
- Migrated manual KV cache calculations to mlsysim.physics.memory.calc_kv_cache_size
- Migrated manual throughput conversions to mlsysim.physics.quantities.token_throughput
- Added calc_binomial_failure_probability to mlsysim.physics.reliability and used it in robust_ai
- Ensured all mathematical logic in book chapters uses single source of truth
calc_point_to_point_time's expected value assumed KB=1024 bytes, but pint's
KB is decimal (1000). calc_hop_latency's assertion compared a pint Quantity
against pytest.approx of a Quantity, which silently drops units. Both were
test bugs; the implementations were correct.
No-backward-compat sweep, part 2. The module had been reduced to a
units-only re-export ('retired, do not add values'); per policy it is
now gone and every consumer reads the real home:
- 32 internal modules, the package __init__, and core/__init__ import
core.units (physics modules keep their own physics/constants)
- 43 book chapters' LEGO cells migrated (star imports, named imports,
and the appendix attribute style 'constants.VIDEO_*' -> units.*);
lego_focal_verify confirms identical pass/flag state to pre-sweep
(8 pre-existing structural flags, A/B-verified unchanged)
- tutorial/tools/test imports migrated (incl. binder test_units.py);
book tests' legacy-symbol scanner treats a missing constants.py as
'nothing defined'
- test_constants_allowlist.py rewritten as a deletion pin: the module
(and any shim for it) must never return; core.units must keep the
measurement surface (ureg, Q_, GB, BYTES_FP16, PRECISION_MAP, ...)
- test_mlsysim_registry_parity.py deleted: it compared registry values
against legacy constants 'before deletion' — that migration is
complete (registry gate list updated)
- Docs prose (DATA_MODEL, architecture, contributing) states the
deletion instead of describing a retired shim
749 passed; binder registry gate green; star import intact.
No-backward-compat policy: mlsysim.engine.solvers is the canonical
implementation package and mlsysim.solvers the public path — the
middle re-export module served no purpose once both ends derive
mechanically. All 16 internal/test import sites repointed; the module-
exports test now pins the shim's ABSENCE (ModuleNotFoundError) plus
exact public/engine surface identity; docstrings, CLI design doc, and
docs links updated. 751 passed.
- frozen=True on all registry pydantic types (hardware, models,
systems, infrastructure, datasets, platforms): registry entries are
shared singletons read by every LEGO cell in a render — a stray
assignment in any cell previously corrupted every downstream cell
silently; it now raises ValidationError at the assignment site
- The freeze immediately caught the one real mutation site:
SensitivitySolver's deepcopy-then-assign perturbation now builds
modified copies via model_copy(update=...) instead
- mlsysim zoo: the CATEGORY argument --help always marked optional now
actually is — omitting it renders both registries and exits 0
(previously errored); contract test re-pinned, cli-reference updated
751 passed; training/ml_systems/distributed_training chapters execute
clean against frozen registries.
- fmt_int: nonzero values that round to 0 now raise (the allow_zero
removal left fmt_int as a silent-zero path: fmt_int(0.4) rendered
'0'); rounding switched from banker's to half-away-from-zero so
adjacent halves agree (2.5 -> 3, was 2)
- Auto-scale counts promote at the rounding boundary:
fmt_params(999_999_999) -> '1B' (was '1000.0M'),
fmt_params(999_999_999_999) -> '1T' (was '1000B'); top scale falls
back to commas ('2,000T')
- fmt_unit now routes through the shared _UNIT_LABEL_NORMALIZATION
table (its divergent local map missed EFLOPs/KFLOPs rates and bare
work units), so fmt_val/fmt_unit table columns always match fmt_qty
prose suffixes. Visible alignment effects: 'Gbps' -> 'Gb/s',
'B' -> 'bytes' in a handful of table cells (now consistent with
adjacent fmt_qty prose)
- fmt_pp(-1) -> '-1 percentage point' (singular checked on magnitude)
- fmt_usd(-500) -> '-$500' (sign before currency glyph; approx and
scale/per variants handled)
20 new boundary tests; fmt suite 175 passed, full suite 751 passed.
Per-layer activation bytes now follow the cited paper's Sec. 4.1 bounds
(FP16 bytes baked into the constants, other precisions scale by
bytes/2): none = 34*s*b*h + 5*a*s^2*b, selective = 34*s*b*h,
full = 2*s*b*h.
The previous model used 34/10/2 as precision-free coefficients AND
multiplied by precision_bytes — double-counting FP16 width at every
strategy; its selective coefficient (10) matched no published source;
and the quadratic attention term (dominant at long sequence) was
missing because n_heads was accepted then discarded.
- strategy='none' now requires n_heads (the 5*a*s^2*b term needs it);
unknown strategy strings raise instead of silently meaning 'none'
- TrainingMemoryModel and TransformerWorkload.training_memory pass the
model's head count through
- Golden Llama3-8B/H100 pin re-derived: activations 21.47 -> 36.51 GB,
total 150.76 -> 165.79 GB (still infeasible on one H100)
- Formula tests re-pinned with the paper's expected values; new tests
for the n_heads requirement, unknown-strategy rejection, and
precision scaling
Decision: user-approved 2026-06-06 ('match Korthikanti exactly').
Book training.qmd call sites updated in the follow-up commit.
- calc_hierarchical_allreduce_time now implements the 3-phase algorithm
its docstring documents (intra reduce-scatter -> inter allreduce of
M/g -> intra all-gather). The old code ran a FULL intra-node
allreduce of M (already equal to RS+AG) plus a second intra allreduce
of M/g labeled 'broadcast', inflating the intra-node term by (1+1/g)
(+12.5% at g=8)
- DistributedModel: n_nodes for the hierarchical path now ceils instead
of floors (12 ranks on 8-GPU nodes span 2 nodes; flooring modeled
zero inter-node traffic)
- calc_all_to_all_time: zero-bandwidth guard (parity with ring/tree) and
honest docstring (M is the per-node buffer, not bytes transmitted)
- calc_pipeline_bubble: rejects n_stages/n_microbatches/v_stages < 1
(0/0 and silent 100%-bubble degenerate paths); boundary test updated
to pin the loud behavior
- physics/__all__: export SPEED_OF_LIGHT_FIBER_KM_S so LEGO cells using
'from mlsysim import *' can see it
Vijay Janapa Reddi