A bold lead-in paragraph (e.g. **Step 2**: ...) immediately followed by a
list item with no blank line between them is parsed by Pandoc as one
paragraph, so the list marker can render inline. Add a markup check
(check markup --scope list-spacing) that flags this, extend the
format lists fixer to insert the blank line for bold lead-ins (not just
colon-terminated lead-ins), and cover both with tests.
Adds 'binder layout purpose <pdf> --vol1|--vol2' subcommand and folds the
check into the high-level 'binder layout --vol1/--vol2' planner so a Purpose
spilling past its opener page is caught with the other layout scans and fails
the layout pass. Delegates to book/tools/audit/check_purpose_overflow.py.
Some servers and doi.org (DataCite DOIs, e.g. NIST) answer HEAD with 404 but
GET with 200. Retry GET before declaring a locator dead. Cuts false positives
from 33 to 17 true dead 404s; timeouts/DNS now warn (could-not-confirm) rather
than error.
binder check bib --scope links resolves every doi/url in the shared bib
(MIT Press: confirm all URLs). HTTP 404/410 = error (dead); timeout/DNS/5xx
= warning (could-not-confirm); 403/401 = warning (bot-block). Network-bound,
default=False so it is not a pre-commit gate. Found 21 dead 404 locators.
Two new default-on, errors-only bib check scopes (native in Binder):
- key-year: key year vs entry year (>1yr gap = silent citation corruption).
Errors-only subset of key-content, so the intentional surname WARNINGs stay
opt-in and do not block commits. Skips standards-numbered keys.
- duplicate-keys: every citekey must be unique (bibtex-tidy only warned;
the openai2023gpt4 collision slipped through). Both auto-join the curated
'binder check bib' set -> wired to the book-check-bib pre-commit hook.
The key-content regex ^([a-z]+)(\d{4}) read the first 4 digits of a standard
number as the entry year (iso14040 -> 1404), firing 5 false-positive ERRORs on
ISO/IEC/IEEE/NIST entries. Add a STANDARDS_KEY_PREFIXES skip. Verified it skips
exactly 7 genuine standards entries and no real author.
Add native Binder checks for LEGO formatting/unit/prose contracts, extend mlsysim formatter helpers, and normalize Vol1/Vol2 LEGO output strings through typed formatters.
Validate precision-sensitive prose rendering across both volumes and promote direct math/string assembly to fmt_math, fmt_display_math, fmt_text, and domain-specific helpers.
Five war-stories (PaLM loss spikes, AlexNet two-GPU split, Google TPU,
Discord Rust rewrite, Facebook inference tax) have a third beat that is
the engineering *response* to the constraint, not the damage from a
disaster. Folding that fix into Failure mode buried the most useful beat
for a student (what the engineers actually did) and the book's core move
(a constraint forcing an architecture). Restore it as a distinct
**Resolution** beat. Disaster war-stories, whose third beat is damage,
stay folded to Context / Failure mode / Systems lesson.
Teach the callout-schema check to accept both war-story archetypes:
Context / Failure mode / Systems lesson, or with an optional Resolution.
New 'markup --scope callout-schema' enforces a uniform bold-label
structure per callout type: every .callout-war-story must be
Context / Failure mode / Systems lesson and every .callout-definition a
single triple-bold term head. Runs in pre-commit and *-validate-dev CI
like the sibling 'callouts' scope. Registered default=False until vol2
narrative callouts are normalized to the same schema (17 still flagged
there); vol1 passes clean.
Adds 'binder check cli --scope binder-canonical' (default=yes): parses
.pre-commit-config.yaml and fails if any local hook scoped to book/quarto/
contents/ calls a raw script instead of ./book/binder. Repo-wide guards, CI
hygiene, third-party hooks, and other subprojects are not book content and
are not inspected; an explicit ALLOWLIST covers any justified exception.
This makes Binder-as-single-front-door a mechanically enforced invariant
instead of a convention — the day someone wires an off-Binder book hook (as
book-check-lego-units once was), the commit fails with an actionable fix.
Runs via the existing always-run book-check-cli-contract hook; no new hook.
Also hardens the cli-contract 'check cli help' case to assert on stable
tokens (group header + scope names) rather than the note phrase, which Rich
wraps once a group gains a scope. Extends it to cover the new scope.
These §10.7 proper nouns must stay capitalized in sentence-style headings
(§10.3.1). Without them, heading-case lowercased 'GPUDirect' → 'gpudirect',
conflicting with the spelling-dict check that capitalizes it. Adds them to the
ACRONYMS preserve-set so the two MIT Press checks agree.
Body prose forbids contractions as a deliberate register override of the copy
editor's permissive default (mit-press-editorial.md §10.11). Two improvements:
- LLM-actionable: emit suggestion '<contraction> → <full form>' from an
expansion map (can't → cannot, it's → it is, …), case-preserving at sentence
start.
- §10.11 exception: mask inline code and double-quoted spans before matching so
contractions in quoted speech / direct dialogue are not flagged.
Automated fixers consume 'binder check <group> --json', so every issue must
carry the exact fix. Populate ValidationIssue.suggestion as '<wrong> → <right>'
for both percent checks:
- percent_in_table: '50 percent → 50%'
- percent_in_prose: '98.5% → 98.5 percent'; range hits get a judgment note
('30–50% → (range) X to Y percent') instead of a misleading per-value fix.
percent_prose.Hit now carries an explicit replacement + is_range flag.
Documented the contract (file/line/code/suggestion/context) in the
ValidateCommand 'adding a check' docstring so future checks inherit it.
The _run_mitpress_* prefix already marks checks that trace to the MIT Press
copyedit (em-dash, slash, vs., e.g./i.e., percent-in-captions, etc.). Extend
it to six more that derive from the copy editor's style sheet / AU query
categories but were not yet labeled, so provenance is visible in the check
registry:
_run_abbreviation_first_use -> _run_mitpress_abbreviation_first_use (Cat F + abbrev list)
_run_concept_term_capitalization -> _run_mitpress_concept_term_capitalization (Cat G + glossary cap)
_run_percent_in_prose -> _run_mitpress_percent_in_prose (style sheet: spell out %)
_run_percent_in_tables -> _run_mitpress_percent_in_tables (style sheet: % kept in tables)
_run_latin_running_text -> _run_mitpress_latin_running_text (spell out e.g./i.e./viz.)
_run_compound_prefix -> _run_mitpress_compound_prefix (close up non-/re-/pre-)
Scope names (CLI surface) are unchanged, so pre-commit, CI, and the CLI
contract are unaffected; only the internal method identifiers move.
New 'percent-in-prose' scope (numbers group, default=True) flags the %
symbol where body prose should spell out 'percent', the symmetric inverse of
percent-in-tables. Per MIT Press AU response Category H / Chicago §9.18,
running prose (body, caption sentences, alt text) uses the word; % is for
tables, equations, code, and in-figure data.
Detection (cli/checks/percent_prose.py) reuses the pipe-table walk to exclude
tables and exempts every legitimate non-prose context: code, math, Quarto
attributes ({width=100%}), link/image targets and URLs, <style> CSS, HTML
comments, caption/alt lines, and quoted material (a % inside double quotes is
a verbatim quotation). No auto-fixer: prose ranges need judgment.
Tables invert the prose convention and use the % symbol, not the word
'percent'. This adds enforcement on both sides, sharing one implementation
in cli/checks/percent_tables.py (matching the cli/checks/currency_style.py
pattern) so the fixer fixes exactly what the check flags:
- check: 'percent-in-tables' scope in the numbers group (validate.py),
default=True → runs in book-check-numbers pre-commit + validate-dev
CI automatically.
- fix: native 'percent-tables' target in the format command
(./book/binder format percent-tables), wired as the
book-format-percent-tables pre-commit hook ahead of prettify so
columns realign after the substitution shortens cells.
Detection reuses the header+separator pipe-table walk shared by the other
table checks, gated by a number-token before 'percent' so header labels and
'percentage points / pp' are never touched.
Adds 'binder layout overlaps <pdf>': a true-geometry PyMuPDF scan for
element-on-element margin overlaps and footer/header overflow that the
anchor-based 'margins' scan is blind to. Exits non-zero on any overlap
(always a bug); --strict also gates on overflow.
Also wires the same scan into verify_volume_pdf as a non-blocking warning,
so every validated PDF build reports 'Margin geometry: N overlaps, M
overflow'. Gracefully skips when PyMuPDF is absent (e.g. minimal CI).
Each margin overflow now ships a ready-to-apply recipe: the computed
in-block \vspace*{-Xmm} nudge (seed = ceil(over_pts*0.45)+2 mm, where
0.45 pre-absorbs the ~18% a \vspace* loses inside a marginnote), the
resolved .column-margin block addressed by SVG/PNG asset name and line
(robust against the scanner's prose-line mis-anchoring), or a footnote
[offset], or "needs eyes" when no block sits near (body table / mis-
anchored). Realizes the computable half of auto-layout.md §2.4 inside
binder so the figure-overflow sweep stops being hand-measured.
Additive: appends one CSV column, no flag/contract change.
The book split on the first axis's name: the vol2 introduction's
formal definition list, the C3 diagnostic appendix, the DAM
projection, the fleet quick-reference, and quizzes said 'Computation
(C1)' while the c3-taxonomy figure, the part principles, the
displacement-of-overhead invariant, and all prose triads (4:0) said
'Compute'. Unified on Compute everywhere the word names the axis
(~30 sites), including index subentries (!computation -> !compute),
the Compute-bound regime label, and the MFU underbrace. Generic uses
(prefill computation, Communication-Computation overlap/ratio, SMPC,
MoE stage labels, code identifiers) untouched; anchors unchanged.
Both heading-case checkers' axis allowlists learn 'Compute' so the
'(Compute)' case-study heading label passes as a framework label.
Introduce 13 LaTeX-typeset algorithm blocks across 8 chapters (Adam update,
gradient checkpointing, FlashAttention, backpropagation, mini-batch SGD,
reverse-mode AD, PTQ calibration, tiled GEMM, ring all-reduce, 1F1B pipeline,
continuous-batching scheduler, speculative decoding, power-of-two routing),
each integrated into existing prose with a motivating lead-in and a
systems-consequence follow-up.
Extension (book/quarto/_extensions/mlsysbook-ext/pseudocode):
- Vendored under the owned mlsysbook-ext/ namespace (moved from leovan/,
version -mlsysbook-custom) and wired into the html, pdf, and epub filters.
- MIT Press cross-ref casing: @algo- -> 'algorithm N' mid-sentence,
@Algo- -> 'Algorithm N' at sentence start.
- Right-aligned italic triangle comments (CLRS column convention).
- Static EPUB renderer (upstream supports only HTML via pseudocode.js and PDF
via algpseudocodex; EPUB previously emitted raw \begin{algorithm}).
- Uses the algo- prefix: native @alg- collides with Quarto's algorithm theorem
environment and FATALs the build.
Ref tooling (book/cli/commands): teach validate.py, bib.py, and info.py the
algo-/Algo- crossref prefix (LABEL_REF_PATTERN, EXCLUDED_CITATION_PREFIXES,
Algorithm label-def patterns, and a chunk-option label harvest that runs inside
the pseudocode fence) so check refs and check labels resolve @algo-/@Algo-
instead of misreading them as missing citations.
Verified rendering in HTML, PDF, and EPUB; check refs and check labels pass
book-wide.
Resolved 36 conflicts by taking dev's side (mlsysim-deferred-polish
import migrations) and deterministically re-applying this branch's
mechanical transformations: LEGO cell reformat, dead #| label removal,
and the two #| output: false hoists.
_collapse_blank_lines toggled code-block state on every ``` line,
but commented-out figures (<!-- ... -->) often contain partial code
blocks with unbalanced fences. One stray fence inverted the toggle for
the rest of the file, so real python cells were treated as prose and
their blank lines collapsed (observed in security_privacy.qmd, which
desynced at its first commented-out tikz block).
Skip fence tracking and blank collapsing inside HTML comment regions;
commented-out content passes through verbatim.
No-backward-compat sweep, part 2. The module had been reduced to a
units-only re-export ('retired, do not add values'); per policy it is
now gone and every consumer reads the real home:
- 32 internal modules, the package __init__, and core/__init__ import
core.units (physics modules keep their own physics/constants)
- 43 book chapters' LEGO cells migrated (star imports, named imports,
and the appendix attribute style 'constants.VIDEO_*' -> units.*);
lego_focal_verify confirms identical pass/flag state to pre-sweep
(8 pre-existing structural flags, A/B-verified unchanged)
- tutorial/tools/test imports migrated (incl. binder test_units.py);
book tests' legacy-symbol scanner treats a missing constants.py as
'nothing defined'
- test_constants_allowlist.py rewritten as a deletion pin: the module
(and any shim for it) must never return; core.units must keep the
measurement surface (ureg, Q_, GB, BYTES_FP16, PRECISION_MAP, ...)
- test_mlsysim_registry_parity.py deleted: it compared registry values
against legacy constants 'before deletion' — that migration is
complete (registry gate list updated)
- Docs prose (DATA_MODEL, architecture, contributing) states the
deletion instead of describing a retired shim
749 passed; binder registry gate green; star import intact.
The book-format-python hook's fence regex matched display blocks
(```python, ```{.python}) but skipped executable ```{python} cells,
so hand-aligned assignment columns accumulated unchecked in LEGO code
(931 aligned lines across 22 files).
Add a second formatting profile for executable cells:
- Black at 150 chars (keeps one-line fmt_* exports; the display-block
70-char profile would explode them into multiline calls)
- --skip-string-normalization (preserves the corpus quote-style mix)
- #| option lines shielded with a sentinel (Black rewrites '#|' to
'# |', which breaks Quarto cell options)
- no comment wrapping (LEGO box-drawing headers must not rewrap)
- blank-line runs collapsed to one inside cells, keeping cells compact
and making output a fixed point for book-format-blanks, whose fence
toggle can desync on commented-out partial code blocks
Apply the one-time mechanical reformat across both volumes. Black
verifies AST equivalence, so all cells are semantically unchanged;
binder math/code checks and the LEGO test suites pass.
A plain-prose colon introduces a single clause, which MIT Press lowercases
(Chicago 17th 6.61 + 8.180), so the cross-ref prefix is mid-sentence: @sec-.
A colon after a bold run-in label (**Setup**:) opens a sentence and keeps the
capital. Previously every colon forced a capital, matching the engineering-
textbook habit the copyedit rejected.
- check refs --scope xref-case: split the colon branch (bold-label lead-in =
sentence start; plain-prose colon = mid-sentence)
- lowercase the 7 prose-colon cross-refs the corrected check now flags
