Refines the explanation of K,V computation savings in the memoization module,
quantifying redundant computations and highlighting the efficiency gain.
The paper and module now specify that generating 100 tokens requires 5,050
total K,V computations, but only 100 are necessary, resulting in 4,950
redundant calculations.
Add new subsection describing function decomposition pattern used within
modules. Documents how complex operations (attention, convolution, training)
are split into focused helper functions with individual unit tests before
composition into exported functions. Updates pedagogical justification to
cover both inter-module and intra-module progressive disclosure.
Escape unescaped & characters in references.bib (Taylor & Francis,
AI & Machine-Learning) and replace Unicode em-dashes (U+2014) with
LaTeX --- ligatures in paper.tex for T1 font compatibility.
- Name the problem: "algorithm-systems divide"
- Name the approach: "implementation-based systems pedagogy"
- Add concrete systems examples (O(N^2), Adam 3x memory)
- Include MLPerf-style benchmarking in milestones
- Strengthen citable terminology throughout
- Switch from fontspec/TeX Gyre to standard fonts (mathpazo, helvet, courier)
- Replace emoji package with no-op (title is just "TinyTorch")
- Switch from biblatex/biber to natbib/bibtex
- Works with both lualatex (local) and pdflatex (arXiv)
- Acknowledge CS249r students whose feedback shaped the curriculum
- Acknowledge global mlsysbook.ai community
- Expand GenAI statement to cover framework development
- Frame AI assistance as enabling democratization (single author, 20 modules)
- Remove em-dashes throughout for cleaner prose
Reverts arXiv-specific changes to work with compile_paper.sh:
- Restored fontspec with TeX Gyre fonts
- Restored emoji package
- Restored biblatex with biber backend
- Works with lualatex as expected by compile script
- Replace fontspec/custom fonts with standard LaTeX fonts (mathpazo, helvet, courier)
- Remove emoji package, define empty command for \emoji
- Switch from biblatex/biber to natbib/bibtex for better arXiv support
- Change \printbibliography to \bibliography{references}
- Remove "Tensors to Systems" from subtitle for cleaner title
- Fix @article entries that should be @inproceedings (kannan2022astrasim,
micikevicius2018mixed, strubell2019energy, vaswani2017attention)
- Remove duplicate booktitle field from williams2009roofline
- Standardize year fields across entries
- New title: "TinyTorch: Building Machine Learning Systems from First
Principles: Tensors to Systems"
- Add strategic mentions of AI engineering as an emerging discipline
- Update competency matrix caption to reference AI engineer competencies
- Update conclusion to position TinyTorch as training AI engineers
- Add SEI AI Engineering workshop report reference
- Remove fabricated claim about quantum ML and robotics community forks
that don't actually exist
- Remove "quantum ML" from conclusion's list of future fork variants
- Change "validated through pilot implementations" to "designed for
diverse institutional contexts" since validation is planned future work
- Replace MNIST/ImageNet full dataset calculation with the actual Module 1 exercise
- New text: "Students calculate memory footprints, discovering that a single batch of 32 ImageNet images requires 19 MB, while the full dataset exceeds 670 GB."
- Aligns paper claims with the codebase (01_tensor.py Q1)
- Rephrase 'optimization opportunities' sentence to avoid dangling word
- Adjust 'roles industry desperately needs' to 'roles that the industry desperately needs'
- Change 'over dense layers' to 'compared to dense layers' to improve line break
- Reorder 'construct mental models gradually' to 'gradually construct mental models'
- Change 'self-paced professional development' to 'independent professional development'
- Align '09 CNNs' (Spatial) with the top Foundation row (01 Tensor)
- Align '15 Quant' and '16 Compress' with the top Foundation row
- Position '10-13' (Language path) and '17-18' (Accel/Memo) on the bottom row
- Center '14 Profiling' and '19 Benchmark' vertically between the two rows
- Remove 'Vision' and 'Language' labels
- Adjust tier labels to be consistently positioned above the top row
- Define a common LabelLine coordinate relative to the top of Foundation Row 1 (T.north + 1.2cm)
- Ensure all tier labels (FOUNDATION, ARCHITECTURE, OPTIMIZATION) are placed on this line
- Prevents labels from overlapping with nodes or being placed too low
- Maintains horizontal centering for each section
- Place all tier labels (FOUNDATION, ARCHITECTURE, OPTIMIZATION) on the same vertical line
- Center each label horizontally over its respective section using coordinate calculations
- Use Architecture label position as the anchor for height alignment
- Move Vision path definition after Language path to allow relative positioning
- Center '09 CNNs' node horizontally above the Language path (Tok-Trans)
- Update arrow routing to accommodate the new centered position
- Reroute the arrow from 04 Losses to 05 DataLoad to go between the rows
- Prevents the arrow from cutting through 06, 07, 08 nodes
- Improves visual clarity of the two-row layout
- Split Foundation modules (01-08) into two rows of 4 nodes each
- Row 1: 01 Tensor -> 04 Losses
- Row 2: 05 DataLoad -> 08 Training
- Connect end of Row 1 to start of Row 2 with a wrapped arrow
- Center 'FOUNDATION' label over the new two-row layout
- Reposition Profiling node to avoid overlap with Architecture branch
- Center Architecture label and align with Optimization label
- Increase font size of path labels for better readability
- Fix overlap of 'Speed' label with 'Transform' node
- Add single node focus framing throughout paper
- Update ML Systems Competency Matrix caption to clarify single node scope
- Strengthen distributed systems discussion in Curriculum Evolution
- Remove all em-dashes (25 total) replacing with colons, commas, periods
- Switch to lining numbers for technical content
- Add widow/orphan penalties to reduce dangling lines
- Fix single-item bullet list (Intentional Gap section)
- Update author limit to 50 before et al truncation
- Fix tinytorch package references to match actual implementation
Trim detailed phase-by-phase validation plan to a concise summary.
Removes specific dates, sample sizes, and instrument names that would
age poorly. Keeps the Open Science Commitment and general validation
approach. Also removes two orphaned references (paas1992training,
sorva2012visual) that were only cited in the removed text.
The paper now uses biblatex for bibliography management, which requires
biber as the backend processor instead of bibtex. Updated both
compile_paper.sh and Makefile to use biber.
- Switch from natbib to biblatex for better author truncation control
- Fix package structure references (tinytorch.nn.conv → tinytorch.core.spatial)
- Fix import examples to use actual tinytorch API patterns
- Fix class references (Transformer → GPT, Attention → MultiHeadAttention)
- Correct Adam coefficient from 0.001 to 0.01
- Fix 11 bibliography entries with wrong/corrupted data:
- abelson1996sicp, bruner1960process, hotz2023tinygrad
- tanenbaum1987minix, perkins1992transfer, papert1980mindstorms
- vygotsky1978mind, blank2019nbgrader, roberthalf2024talent
- keller2025ai, pytorch04release, tensorflow20
- Fix organization author names using double braces
- Configure maxbibnames=10 for "et al." truncation in bibliography
All 60 references verified via web search for arXiv submission.
Adds 'Building systems creates irreversible understanding' to the
paper's conclusion section, reinforcing the pedagogical thesis with
concrete examples: once you implement autograd, you cannot unsee
the computational graph; once you profile memory, you cannot unknow
the costs.
- Increase main title size (26pt → 28pt) for more impact
- Tighten subtitle line spacing
- Add proper line height to author block
- Create visual hierarchy: name > affiliation > URL
- Make URL more subtle (smaller, lighter gray)
