The book-format-python hook's fence regex matched display blocks
(```python, ```{.python}) but skipped executable ```{python} cells,
so hand-aligned assignment columns accumulated unchecked in LEGO code
(931 aligned lines across 22 files).
Add a second formatting profile for executable cells:
- Black at 150 chars (keeps one-line fmt_* exports; the display-block
70-char profile would explode them into multiline calls)
- --skip-string-normalization (preserves the corpus quote-style mix)
- #| option lines shielded with a sentinel (Black rewrites '#|' to
'# |', which breaks Quarto cell options)
- no comment wrapping (LEGO box-drawing headers must not rewrap)
- blank-line runs collapsed to one inside cells, keeping cells compact
and making output a fixed point for book-format-blanks, whose fence
toggle can desync on commented-out partial code blocks
Apply the one-time mechanical reformat across both volumes. Black
verifies AST equivalence, so all cells are semantically unchanged;
binder math/code checks and the LEGO test suites pass.
Replace pointers to the private project rules/docs tree (relative .claude/rules
and .claude/docs paths) in code comments and docstrings with neutral phrasing
("the project prose style guide", etc.). Load-bearing runtime paths that the
tooling reads or writes are left intact.
Builds per-chapter review packets that compare concept-map entries
against the existing glossary headwords, classifying each missing
candidate by category and priority so editorial decisions
(add / already covered / too composite / methodology-formula-application
/ rename / needs review) can be made deliberately.
Outputs CSV + per-chapter Markdown packets + per-volume summary JSON
under .mlsysbook/glossary-pass/latest/. Pure stdlib + PyYAML; no LLM
calls. Pairs with .claude/rules/glossary.md.
Every other vol1 chapter folder name matches its qmd slug; this aligns the
last outlier so folder == qmd stem across the volume. Also matches the slides
folder name (slides/vol1/10_model_compression).
- git mv contents/vol1/optimizations → contents/vol1/model_compression
- Update path refs in 4 quarto configs (html, pdf, pdf-copyedit, epub)
- Update path refs in index_prune_candidates.yml, build_locator_bins.py,
format_tables.py, learning_objectives_bolding_parallel.sh
- Update chapter-id refs (vol1/optimizations → vol1/model_compression) in
vol2 quiz integration strings (ops_scale, robust_ai, sustainable_ai)
- Update CHAPTER_DIRS, READING_ORDER, and stale "outlier" docstrings in
fix_abbreviations.py, build_prior_vocab.py, build_audit_context.py,
generate_quizzes.py (fallback logic kept as defensive code)
- Rename _audit/optimizations_audit.json → model_compression_audit.json
and fix its stale source_file/chapter fields
- Update vol1/README.md chapter table
Script spells out single digits 1-9 in body prose per MIT Press
style. Comprehensive exclusions for: units, ranges, versions,
decimals, math, code, tables, cross-refs, labeled items, and
technical terms (M/M/1, GPT-3, etc.). 556 replacements already
applied by agent; script is idempotent.
The pipe table prettifier re-spaces table cells, making em dash
close-up in tables futile. Updated script to skip all table lines
(not just separator lines). This resolves the 19 "remaining" em
dashes that were all inside table cells.
Adds publisher fields to @inproceedings and journal fields to @article
entries based on known venue mappings. Covers NeurIPS (Curran Associates),
ICML/AISTATS (PMLR), ICLR (OpenReview.net), OSDI/NSDI (USENIX),
MLSys, ACM venues, IEEE venues, and arXiv preprints.
Lowercases concept terms in body prose: Iron Law → iron law,
Degradation Equation → degradation equation, Verification Gap →
verification gap, Bitter Lesson → bitter lesson, ML Node → ML node,
and wall terms (Data/Compute/Memory/Power Wall).
Preserves capitalization in: bold definitions, section headers,
display math, inline math, \index{} entries, callout titles,
sentence starts, and the "Iron Law of Processor Performance" (H&P
reference). Adds fix_capitalization.py script.
Expands 'CNN' to 'convolutional neural network (CNN)' and 'RNN' to
'recurrent neural network (RNN)' at first body-prose use in their
respective chapters per MIT Press style. Adds fix_abbreviations.py
check script. i.i.d. expansions already present in source.
Spells out percent in body prose per MIT Press style. Handles both
hard-coded '94%' and inline Python '`{python} x.pct_str`%' patterns.
Skips tables, code fences, LaTeX math, YAML, HTML/CSS blocks, and
fig-cap/fig-alt attributes. Adds fix_percent.py script.
Replaces ' — ' (spaced) with '—' (closed) in body prose per MIT Press
house style. Skips code fences, Python cells, YAML frontmatter, and
inline math. Adds fix_emdash.py script for future use.
- Update example commands to reference MiniMax-M2.7 (latest flagship model)
- MiniMax-M2.7 offers enhanced reasoning and coding capabilities
- Users can also use MiniMax-M2.7-highspeed for low-latency scenarios
Add support for OpenAI-compatible cloud APIs (OpenAI, Groq, MiniMax)
as alternatives to local Ollama for caption generation in manage_captions.py.
New CLI flags:
--provider/-p: select LLM backend (ollama, openai, groq, minimax)
--api-key: API key for cloud providers (or use env vars)
--api-base: custom base URL for any OpenAI-compatible endpoint
The existing Ollama workflow is unchanged (remains the default).
Cloud providers use the already-installed openai SDK.
Remove redundant ml_ prefix from ml_workflow chapter files and update all
Quarto config references. Consolidate custom scripts into native binder
subcommands and archive obsolete tooling.
Unifies Quarto metadata into shared base/format/volume fragments while carrying through chapter path, asset, and tooling updates to keep the repository consistent and easier to maintain.
- Completed full Volume 1 refactor to Safe Class Namespace pattern.
- Fixed render errors and verified all 16 chapters.
- Updated 'binder' CLI with native validation and maintenance namespaces.
- Enhanced VS Code extension with Chapter Navigator and Run History.
- Integrated 'binder validate' into pre-commit workflows.
Add proper equation labels ({#eq-...}) and prose references (@eq-...)
to 138 equations across 15 Volume 1 chapters following the gold-standard
pattern from serving.qmd.
Key changes:
- Label all display math equations with {#eq-kebab-case-name}
- Add @eq-name references in prose before each equation
- Equations include: Iron Law, Amdahl's Law, Roofline Model,
activation functions, backpropagation, attention mechanisms,
queuing theory, quantization, and system throughput formulas
Also includes:
- PDF formatting improvements (newpage directives for Vol 2)
- LaTeX header updates for chapter styling
- Pre-commit config and validation script updates
Fix NameError build failures in ml_systems, data_engineering, and
benchmarking chapters caused by missing imports and variables referenced
before their defining code cells.
- ml_systems: add missing Kparam and Bparam imports from physx.constants
- data_engineering: compute transfer_time_10g_md preview in setup cell,
add md_math import, add deduplication-dividend-calc cell, convert
hardcoded values to physics engine units
- benchmarking: compute BERT roofline preview values in roofline-example-calc
cell before they are referenced in narrative text, convert hardcoded
values to inline Python, condense redundant footnotes
Also includes physics engine integration improvements across all Vol 1
chapters: unit-safe conversions, inline Python for previously hardcoded
values, streamlined footnotes with cross-references, and new content
validation scripts.
All 21 Vol 1 chapters pass PDF build tests.
- Fix \index{} commands breaking rendering when placed before footnote
definitions, div openers (:::), or on same line as headings
- Add check_index_placement.py script with --fix flag to automatically
detect and fix these patterns
- Update training.qmd and data_engineering.qmd with corrected index placement
- Include other pending content and visualization updates
Key changes:
- sci() and sci_latex() now convert Pint quantities to base units
(fixes 10^2 showing instead of 10^14 for TFLOPs values)
- Add md_frac(), md_sci(), md_math() helpers for LaTeX in Markdown()
- Update ml_systems.qmd with proper LaTeX fraction rendering
- Add freeze: false to _quarto.yml to prevent caching issues
- Update CLAUDE.md with QMD inline Python conventions
- Fix LATEX_ADJACENT issues across multiple QMD files (Unicode symbols)
format_tables.py was escaping <, >, & to HTML entities inside Markdown
grid tables, breaking LaTeX math and comparison operators in rendered
output. Removed the escape_html_entities() calls since Quarto grid
tables are Markdown, not HTML.
New validate_tables.py catches rendering issues the structural formatter
misses: bare pipes in LaTeX math, \frac in multiline cells, HTML
entities, and missing table labels.
New check_references.py for cross-reference validation, preview scripts
for diagram and systems-gap visualization, MIT Press release packaging
script, and improvements to forbidden footnotes checker.
Create check_figure_completeness.py pre-commit hook that validates all
figures have captions and alt-text across div, markdown, and code-cell
syntaxes. Add code-cell figure support to extract_figures.py and
figure_table_flow_audit.py. Fix fig-algo-efficiency missing caption in
introduction.qmd and fig-business-cost-curve missing alt-text in ops.qmd.
Vol 1 now passes with 199/199 figures complete.
Scans all .qmd chapters and reports where each figure/table is
defined vs. first referenced, flagging placement gaps (LATE,
EARLY, ORPHAN). Found 55 issues across 653 elements in 35
chapters. Enables targeted editorial fixes for figure/table flow.
- Fix 75 grid tables: ensure first column is always left-aligned
- Update format_tables.py to enforce left-aligned first column rule
- Update convert_pipe_to_grid_tables.py to enforce same rule
- Consolidate redundant callout boxes in ops, model_compression, serving, dl_primer
- Streamline napkin math sections for better flow
- Add support for different JSON structures (list vs dict) in quiz files
- Implement two-pass approach for repair mode to ensure cross-references
are updated before headers are changed
- Fix ID comparison to handle leading # in section_id fields
- Close unclosed callout divs in model_compression, training, workflow
- Move footnote references outside div blocks (ops, communication)
- Move footnote out of table cell in communication.qmd
- Update check_forbidden_footnotes.py to handle :::: nested divs
- Auto-fix table column spacing in 9 tables across 6 files
All pre-commit checks now pass.
Converted 177 simple pipe-style markdown tables to restructuredText
grid-style format across both volumes for consistency with existing
table standards.
Changes:
- Created convert_pipe_to_grid_tables.py script to automate conversion
- Converted tables in 19 chapter files (9 in Vol1, 14 in Vol2)
- Applied format_tables.py to ensure proper formatting:
* Bold headers across all columns
* Bold first column for comparison/category tables
* Intelligent column alignment (left for text, right for numbers)
* Proper spacing and border alignment
Tables now use consistent grid format with +/- borders and alignment
markers (:= for left, =: for right), matching Volume 1 standards.
Adds a pypandoc-based section splitter utility that parses .qmd chapter
files and extracts individual sections for processing. This enables
guaranteed 100% coverage in editorial workflows by processing each
section independently rather than entire chapters at once.
Key features:
- Uses pypandoc JSON AST for robust parsing (correctly ignores headers
inside code blocks, callouts, and TikZ diagrams)
- Falls back to regex-based block tracking if pypandoc unavailable
- Extracts section metadata: title, ID, line numbers, word count
- Supports listing, extraction to files, and JSON manifest output
- Designed for integration with polish workflow agents
Usage:
python3 section_splitter.py -f chapter.qmd --list
python3 section_splitter.py -f chapter.qmd --manifest
python3 section_splitter.py -f chapter.qmd --get-section 3
- Renamed vol2/advanced_intro to vol2/introduction for consistency
- Updated all scripts and configs to use vol1/ instead of core/
- Updated pre-commit config to check all contents/ not just vol1/
- Updated path references in Lua filters, Python scripts, and configs
The script was using 4 parent directories to calculate workspace_root,
but since the script is at book/tools/scripts/content/format_tables.py,
it needs 5 parents to reach the actual repo root.
This was causing "Directory not found: book/book/quarto/contents" errors
in CI when the pre-commit hook passed paths like "book/quarto/contents/".
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Updated Python utility scripts to use correct paths with book/ prefix:
- rename_downloaded_images.py: quarto/contents/labs → book/quarto/contents/labs
- rename_auto_images.py: quarto/contents/labs → book/quarto/contents/labs
- convert_svg_to_png.py: quarto/contents → book/quarto/contents
- check_self_referential_sections.py: quarto/contents → book/quarto/contents
These scripts are run from the repository root, so they need the full
path including the book/ directory.
* Restructure: Move book content to book/ subdirectory
- Move quarto/ → book/quarto/
- Move cli/ → book/cli/
- Move docker/ → book/docker/
- Move socratiQ/ → book/socratiQ/
- Move tools/ → book/tools/
- Move scripts/ → book/scripts/
- Move config/ → book/config/
- Move docs/ → book/docs/
- Move binder → book/binder
Git history fully preserved for all moved files.
Part of repository restructuring to support MLSysBook + TinyTorch.
Pre-commit hooks bypassed for this commit as paths need updating.
* Update pre-commit hooks for book/ subdirectory
- Update all quarto/ paths to book/quarto/
- Update all tools/ paths to book/tools/
- Update config/linting to book/config/linting
- Update project structure checks
Pre-commit hooks will now work with new directory structure.
* Update .gitignore for book/ subdirectory structure
- Update quarto/ paths to book/quarto/
- Update assets/ paths to book/quarto/assets/
- Maintain all existing ignore patterns
* Update GitHub workflows for book/ subdirectory
- Update all quarto/ paths to book/quarto/
- Update cli/ paths to book/cli/
- Update tools/ paths to book/tools/
- Update docker/ paths to book/docker/
- Update config/ paths to book/config/
- Maintain all workflow functionality
* Update CLI config to support book/ subdirectory
- Check for book/quarto/ path first
- Fall back to quarto/ for backward compatibility
- Maintain full CLI functionality
* Create new root and book READMEs for dual structure
- Add comprehensive root README explaining both projects
- Create book-specific README with quick start guide
- Document repository structure and navigation
- Prepare for TinyTorch integration
