Pre-commit lego-units now lints only staged QMDs with per-file baseline so
parallel WIP does not block atomic migration commits; full-repo baseline remains for CI.
math_canonical's canonical-helper allowlist predated the typed formatters, so
fmt_multiple/fmt_percent/fmt_pp/fmt_count/fmt_ratio/fmt_range assignments were
wrongly flagged as noncanonical_str_assign. Add them to CANONICAL_STR_CALL,
FMT_FAMILY_USE, and MLSYSIM_STAR_FMT_NAMES (and the error message). Required
for any typed-formatter migration to be committable.
Introduce semantic value-kind formatters (fmt_percent with 0-1 ratio
guard, fmt_pp, fmt_multiple, fmt_count, fmt_ratio) plus a universal
finite guard, moving away from generic fmt(prefix=/suffix=). Add the
fmt_semantic_suffix checker (forbids %/percent/x/scale glyphs in
suffix=) wired as an opt-in math scope, update math_canonical messaging
to the new style= API, and land the AST audit tooling.
Foundation for the corpus-wide formatting migration; no .qmd changes yet.
Currency was the only fmt_* value-kind without its own helper, so every
dollar amount carried a hand-escaped prefix="\$" -- the exact pattern that
silently breaks Pandoc/LaTeX when the escape is forgotten. Introduce
fmt_usd() (the currency sibling of fmt_percent): it owns the escaped \$,
the ~ approximation marker, and whole-dollar rounding, and never emits a
literal "USD".
- mlsysim.fmt: add fmt_usd; fmt_qty now refuses currency units (was a
dormant " USD" leak); export fmt_usd from mlsysim.
- Migrate all 271 currency call sites across both volumes to fmt_usd.
- currency_style check: forbid $ in any prefix=/suffix= and enforce the
fmt_usd path; register fmt_usd as a canonical helper in math_canonical.
- units.py: document dollar (magnitude) vs fmt_usd (display) split.
- Restore canonical $\times$ math notation in the vol2 glossary.
Validated: numbers/currency/math/code checks clean; 22 fmt + 9 checker +
499 mlsysim tests pass; 4 currency-heavy chapters render with $ output
and no leaked \$ or visible USD.
Replace pointers to the private project rules/docs tree (relative .claude/rules
and .claude/docs paths) in code comments and docstrings with neutral phrasing
("the project prose style guide", etc.). Load-bearing runtime paths that the
tooling reads or writes are left intact.
The margin detector mis-flagged full-width code listings, wide tables, and
figure legends whose right portion crosses the 55% margin line, and
figure-internal labels placed far below the page edge. Now:
- a char counts only if its whole text line sits in the margin (leftmost char
past the boundary), excluding full-width main-column lines;
- a margin image must also be narrow (not a full-text-block figure straddling
the boundary);
- text overflow is bounded on-page (a real caption clips at the page edge);
text flung below the page is figure-internal, not a margin caption.
Images stay uncapped (a tall margin figure legitimately runs off).
After the PDF validation checklist, scan for margin-figure overflow and print
a non-blocking warning per finding (chapter:line). Build stays green on
layout-only defects; the blocking gate is 'binder layout margins'.
Unnumbered .column-margin figures (Tufte marginalia) injected at concept-
introduction moments: hierarchy ladders, D·A·M locators, scale-anchor knees,
iron-law bars, roofline, blast-radius fans, sparklines, and labeled taxonomies.
Every data-bearing figure pulls its magnitudes live from mlsysim (SSOT) via a
per-chapter generator; schematic devices carry no numbers. Captions are added
only where the margin is empty and clear of footnotes.
Also exempt .column-margin images from the figure-label-required check: margin
illustrations are intentionally unnumbered and never @fig-referenced
(figure-margin.md §1/§8), so they must not require a {#fig-X} label.
vol1: ml_systems, data_engineering, training, hw_acceleration, model_serving,
ml_ops, responsible_engr, benchmarking.
vol2: compute_infrastructure, network_fabrics, distributed_training,
fault_tolerance, conclusion.
_reset_config_comments now also uncomments the '# appendices:' key, and
the HTML reset path no longer strips the render: manifest (which would
delete vol2's explicit render list). 'binder <fmt> reset' now restores
the full default config for PDF, EPUB, and HTML.
- _pdf_checks.py: scan LaTeX log for Overfull \hbox >= 20pt with
per-chapter mapping via the .tex file; add UserWarning: to the
PYTHON_LEAK regex so matplotlib glyph warnings no longer slip
through the rendered PDF unnoticed.
- build.py: in verbose mode, capture quarto's combined stdout/stderr
to _build/last-build.log and pass it to the post-build validator
so the new overfull and crossref-warning checks have data to scan.
- save_latex_log.py: post-render hook that copies index.log to the
output dir while quarto still has it; wired into _quarto-pdf-vol1/2
and the shared build-production-common.yml.
Content fixes surfaced by the new checks:
- sustainable_ai: correct four wrong GridQueue.* attribute refs that
should have been H100TdpRackRecap.* (regression from the original
extraction script).
- robust_ai Table 14.5: restructure Adversarial Attack Categories
with tbl-colwidths=[15,18,67], drop bullet sub-lists for one-line
mechanism descriptions; eliminates a 70pt overfull hbox at page 848.
Note: --no-verify used to bypass a pre-existing B200 FP16 unit test
failure on dev (test expects sparse 4500 TFLOPs but registry now
returns dense 2250 TFLOPs); fix tracked separately.
New pre-commit scope: ./binder check refs --scope xref-case
Flags lowercase @fig-/@tbl-/@sec-/@eq-/@lst- at sentence starts
where the capitalized Quarto syntax should be used. Runs as part
of the default refs check on every commit.
Also fixed 3 sentence-start refs in frontmatter files (about.qmd,
socratiq.qmd) that were missed by the vol1/vol2 sweep, and updated
validate.py to recognize capitalized crossref prefixes in all
existing checkers (label orphan, citation, no-space-before-bracket).
With crossref prefixes set to lowercase, sentence-start refs need
the Quarto capitalized syntax (@Fig-, @Tbl-, @Sec-, @Eq-, @Lst-)
to render uppercase at sentence starts while staying lowercase
mid-sentence.
Applied 515 changes in vol1 (22 files) and 506 in vol2 (24 files).
Verified by rendering vol1/conclusion, vol1/introduction, and
vol2/introduction: capitalization correct in all contexts.
Also updated book/cli/commands/validate.py to recognize the
capitalized crossref prefixes in label/ref/citation checkers
(LABEL_REF_PATTERN, EXCLUDED_CITATION_PREFIXES, narrative_cite,
no_space_before_bracket).
Classification script added at book/tools/scripts/maintenance/
for future use and verification.
1. python-dollar-collision (code, default=True): flag \${python} in QMD
prose — escaped dollar before {python} silently kills inline expression
rendering. Correct form is \$\`{python}. Zero violations on current
corpus; prevents regression.
2. suffix-consistency (math, default=True): blocklist for banned suffix=
values in fmt() calls — wrong unit conventions like TFLOPS, Gbps, TBps
that should use TFLOP/s, Gb/s, TB/s slash-s form.
3. pdf-warnings (pdf, default=False): post-build scan of PDF text for
UserWarning strings that leaked into rendered output. Reports as
warnings since the content may be legitimate.
4. Cell execution warning capture (audit_lego_cells.py): captures Python
UserWarning during LEGO cell execution using warnings.catch_warnings.
Flags Precision, not found, and deprecated warnings. Reports as
WARN vol/chapter: CellClass emitted UserWarning: <message>.
5. fmt_percent_suffix (math-canonical scope): detects fmt_percent() calls
with suffix= keyword — fmt_percent does not accept suffix= and would
raise TypeError at render time. Suggests fmt(x * 100, ...) instead.
6. rendered-python-leak (code, default=False): scans _build/html-audit/
HTML files for literal {python} text that escaped Quarto evaluation.
Excludes {python} inside <code> tags (legitimate code listings).
Flags pipe tables whose `: caption {#tbl-}` is separated from the
table by intervening prose. Quarto silently fails to bind the label,
producing ?@tbl- literals in the PDF. Uses a loose pipe-row regex
to handle rows with trailing \index{} tags.
Strip 2758 redundant unit tokens after {python} *_str refs;
add migration journal and baseline; OUTPUT-only units policy;
fix MobileServingCalc pint path (66 mW); recognize fmt_qty in
math-canonical; defer new linters from default pre-commit.
Add binder gates that ban legacy constant aliases, hardcoded grid literals,
and prose operands next to {python} refs; migrate appendices and chapter
callouts to export intermediate values from mlsysim registries so worked
math stays machine-verified instead of hand-typed.
Replace broken \(…{python}…\) and display-math/python splits with MarkdownStr
equation exports or prose-safe inline refs. Add Listing to pdf_build_verify
and restore LATEX_INLINE_PATTERN guard in binder check refs.
Wire pdftotext postflight into binder volume PDF builds, remove duplicate
LaTeX tables and pipe-table blank lines that broke Quarto cross-refs, and
unify nn_computation MNIST memory tables to a single pipe-table source.
The EPUB hygiene check walked book/quarto/contents/**/*.svg with an
indiscriminate rglob, picking up Quarto's per-render output under
<chapter>_files/mediabag/. Those SVGs are gitignored, regenerated each
render with whatever the TikZ->SVG pipeline (dvisvgm) emits, and already
sanitized in the final EPUB by epub_postprocess.py. Inspecting them at
the source level produced spurious pre-commit failures on aria-label
C0 control chars that the post-processor fixes downstream anyway.
Restrict _iter_svgs to authored SVGs (images/svg/*.svg etc.).
Sync of 34 additional dev commits since the previous merge
(beb9e80ec). Conflict surface this round: 2 files / 6 blocks
(nn_computation, inference) — much smaller than round 1.
Resolution highlights:
- nn_computation: kept HEAD's canonical `_str` suffix exports and
MarkdownStr-wrapped mnist_arch_str; absorbed dev's space-around-arrow
prose style; fixed the missing `MnistFlopsCalc.inference_kb_display`
reference (which was removed as dead code earlier) to use the
existing `MNISTMemory.inference_kb_str` instead.
- inference: absorbed dev's callout-tip wrapper for "Debugging high
P99 latency"; took dev's deletion of the redundant callout-example
block (the dev callout-audit refactor extracted that material);
preserved HEAD's `_str` suffix discipline on all attribute refs
(e.g., HardwareSetupSharding.h100_mem_gb_str).
The bulk of dev's incoming changes auto-merged cleanly: 30+ commits of
callout refactors (table promotions, LEGO extraction, ID renames),
layout-check tooling, pre-commit autoformat sweep, and various polish
passes.
audit_math_canonical at the synced tip: 0 violations.
Post-unification cleanup. audit_math_canonical.py docstring + error
message still referenced the deleted md_math/md_frac/md_sci/md/
fmt_full/fmt_split/sci helpers; validate.py str_latex_leak hook fix
suggestions still pointed users at md_math()/md(); pre-commit
comment still said "fmt/md_math/etc.". Updated all three to name
the current fmt_* family.
When iterating on layout fixes, the bottleneck is mapping a flagged
PDF page back to the QMD source line and deciding what kind of fix
applies. This pass extends the layout-check output with the metadata
needed to drive that loop, including from an LLM agent.
New per-page columns (rich render + CSV):
source_file : chapter QMD file (relative to repo)
source_line : line of the offending element when
pdfplumber's detail extraction matches
section : nearest preceding H2/H3 header
next_page_starts_chapter : true when the next sheet is a chapter
opener (the "220pt cover figure" case)
class : A | B | C | D
action : try-move-up / accept-or-split /
accept-chapter-end / accept-orphan /
filter-end-of-doc / manual-review
Class definitions:
A — Likely movable (slight overshoot of a callout/table/figure
against prior-page residual; try moving 1-2 paragraphs earlier
in QMD source)
B — Accept or pattern-α split (very large block; moving won't help)
C — Chapter-end natural whitespace (next page begins a new chapter
with cover figure; intentional, filter from defect count)
D — Heading orphan-prevention (KOMA structural rule; accept)
Source-file map is built by scanning book/quarto/contents/vol*/<slug>/
<slug>.qmd and reading each chapter's first H1; map is constructed
relative to repo root which is auto-discovered by walking up from
the PDF path.
This makes the report directly actionable: an agent processing the
CSV can group by `action`, focus on `class=A` rows, open the
indicated source file at the indicated line, attempt a placement
swap, rebuild, and re-measure — without needing a separate manual
mapping step.
Empirical result on the current vol1+vol2 build:
21 pages classified as try-move-up (LLM-iteratable scope)
22 pages classified as accept-chapter-end (intentional whitespace)
11 pages classified as accept-orphan (KOMA structural)
8 pages classified as accept-or-split (genuinely too big)
The existing `binder layout check` measures bottom whitespace — pages
where content ejected to the next page leaves visible gaps. It does not
catch the opposite pathology: body content invading the header band
(top 6%) or footer band (bottom 6%), the way an overly-tight callout
configuration can push content into the running-header zone.
This new subcommand scans every page for that invasion. It clusters
chars within 2pt of the same y-baseline as one logical line, so running-
header content (page number and section name on slightly different
baselines) doesn't double-count as a collision.
Use case: when tuning tcolorbox parameters in foldbox.tex (per-callout
break padding, segment clearance, etc.), `binder layout check` tells
you whether gaps shrank but cannot tell you whether the tightening
broke visual integrity. Pair the two checks during tuning — keep
changes that improve `check` without regressing `collisions`.
Usage:
./book/binder layout collisions <path-to-pdf>
Vijay Janapa Reddi