GPT._create_causal_mask was returning an additive -inf mask
(upper-triangular with -inf, 0 for allowed positions) but
_apply_mask in module 12 expects a binary mask (1=allow, 0=block)
and computes adder = (1 - mask) * MASK_VALUE.
With the old mask, every allowed position (value 0) got masked by
(1-0)*(-1e9) = -1e9, blocking ALL attention -- both past and future
tokens. Only an infinite value in the mask produced the correct
-inf addend for blocked positions.
Fix: use np.tril(np.ones(...)) which is the binary lower-triangular
convention that _apply_mask expects.
gamma and beta were created as Tensor(np.ones/zeros(n)) with no
requires_grad flag, defaulting to False after enable_autograd() patches
Tensor.__init__. The _LayerNormBackward.apply() guards on
beta.requires_grad and gamma.requires_grad, so gradients were silently
never computed for either parameter -- LayerNorm could not learn its
scale and shift during training.
Fix: pass requires_grad=True at construction so the backward pass
computes grad_gamma and grad_beta correctly.
Also remove the manual param.requires_grad = True workaround from
test_layernorm_gradient_flow() which was masking the bug.
Audit (parent commit): every tinytorch/src/NN_xxx/ABOUT.md is a stale
duplicate of tinytorch/quarto/modules/NN_xxx.qmd, with the QMD strictly
newer/richer (each QMD has additional sections like Your TinyTorch,
PyTorch comparison, MLPerf section that ABOUT.md is missing).
Drift confirms ABOUT.md is unmaintained: e.g., 12_attention/ABOUT.md
still says "GPT-3 training (4x inference)" while the QMD was corrected
to 5x. Single-source-of-truth in quarto/modules/ stops the recurrence.
The retired Jupyter Book site (tinytorch/site-legacy/) was the only
historical consumer; it is deleted in the next commit.
Also deleted:
- tools/dev/fix_about_titles.py -- one-shot ABOUT.md title fixer
- tools/dev/fix_mermaid_diagrams.py -- one-shot ABOUT.md mermaid tweaker
Both operated on the now-deleted ABOUT.md files and have no other use.
Updated tinytorch/README.md so the documented repo tree no longer
shows ABOUT.md under each module folder, and repointed a stale
"setup guide" link from site/getting-started.md to quarto/getting-started.qmd.
Replace np.random.randn/rand/seed with np.random.default_rng(7) across
all 93 source modules, tests, and milestones for reproducible, isolated
random state.
Notebooks use short names (tensor.ipynb, not 01_tensor.ipynb) but docs
and Binder postBuild scripts used the prefixed form. This caused broken
Binder links and incorrect paths in troubleshooting guides.
Fixes: harvard-edge/cs249r_book#1176
Replace hardcoded numerical values across all module ABOUT.md files with
Python-computed values using myst_nb glue() references. Each file is now a
MyST Markdown Notebook that executes inline code cells to compute memory
sizes, FLOPs, compression ratios, and other quantitative values.
Key changes:
- Add file_format: mystnb frontmatter and code-cell blocks to all 20 files
- All arithmetic (memory calculations, speedups, ratios) now computed inline
- Fix multiple arithmetic errors discovered during conversion
- Enable execute_notebooks: "cache" in PDF config for glue resolution
- Fix jupyter-book version constraint in Makefile
Add the TinyTorch VS Code extension source package and align module code/docs references so APIs, milestones, and progression notes remain consistent across the curriculum.
Audit and fix consistency issues across all module source files:
- Standardize ML Systems header to "ML Systems Reflection Questions" (01, 13)
- Fix section ordering: test_module before ML Systems in modules 16, 17
- Rename demo_spatial() to demo_convolutions() to match module name (09)
- Rename demo_*_with_profiler() to explore_*_with_profiler() (15, 16, 17)
- Fix test naming to use test_unit_* prefix consistently (03, 05, 11, 12)
- Add missing emojis in test_module/demo patterns (02, 15)
- Standardize tito command format to number-only (01, 03, 06, 07, 18)
- Fix implementation headers: hyphen to colon separator (09, 12)
- Add missing "Where This Code Lives" package section (13)
- Fix export command in module summary (05, 06)
Three categories of changes across 17 modules:
1. Function decomposition (Modules 01,03,05-15,18-19): Break large
monolithic functions into focused _helper + orchestrator pattern.
Each helper teaches one concept with its own unit test.
2. Naming convention fixes (Modules 08,09,11,18,19): Ensure underscore
convention is consistent — standalone _func in export cells renamed
to func (public API), monkey-patched method names match target
visibility, removed unnecessary #| export from internal helpers.
3. Progressive disclosure (Modules 02-05,08,11-15): Remove forward
references to future modules. Replace "you'll learn in Module N"
with concrete descriptions. Trim connection maps to only show
current and prior modules. Keep end-of-module "Next" teasers
as motivational breadcrumbs.
All 17 modified modules pass their test suites.
Replace the slide download card with an inline PDF viewer using PDF.js:
- Change grid layout from 4 cards (2x2) to 3 cards in a row
- Add embedded slide viewer with navigation, zoom, and fullscreen
- Load slides from local _static/slides/ for reliable CORS handling
- Add "· AI-generated" subtitle to match audio card pattern
- Use 🔥 icon consistently across all viewers
Affected: 20 module ABOUT.md files + big-picture.md
Update unit test markers across all 20 modules for consistency:
- Changed header emoji from microscope to test tube
- Maintains visual consistency across module test sections
- Standardize test emoji usage (🔬 for unit tests, 🧪 for module tests)
- Restore missing NBGrader solution blocks in Module 15 (quantization)
- Fix missing #| default_exp directives in modules 05, 12, 13, 15, 17
- Remove duplicate #| default_exp directives
- Ensure all exports are only for core functionality (no tests/demos)
- Apply consistent styling across all 20 modules via module-developer agent
* fix: fix GPT model to use Embedding Layer created in module 11 instead of re-defining token embedding and positional embedding
* fix: fix module import in Transformers module test
Replace text-only card links with styled 54px pill buttons for
Binder (orange) and GitHub (gray) actions. Audio player height
now matches button height for visual consistency across all 20
module pages.
- Module 01: Remove requires_grad, grad, backward() from Tensor class
Students learn pure tensor math first without gradient concepts
- Module 02: Remove requires_grad propagation from Softmax
Activations are now forward-only until autograd is enabled
- Module 03: Remove requires_grad=True from layer weights
Layers store parameters without gradient flags
- Module 05: Update enable_autograd() to ADD gradient infrastructure
Now adds requires_grad, grad, and backward() to Tensor class
Uses helper functions for tensors created before autograd
- Module 09: Remove Conv2dBackward, MaxPool2dBackward classes
Convolutions are now forward-only, no Module 05 import
- Module 11: Remove EmbeddingBackward import and usage
Embeddings are now forward-only
- Modules 12, 13: Remove requires_grad from mask/param tensors
This implements true progressive disclosure: concepts are introduced
only when students are ready to learn them. Gradient tracking is now
completely absent from Modules 01-04 and added in Module 05.
Adds interactive launch cards to all 20 TinyTorch module ABOUT.md files:
- Launch Binder button for browser-based exploration
- View Source link to GitHub implementation
- Audio Overview placeholder for NotebookLM content
Cards wrapped in {only} html directive to exclude from PDF output.
Also removes duplicate README.md files from src modules.
Add machine-readable module.yaml files to each of the 20 modules with
title, subtitle, and description fields. Update tito CLI to read from
these files instead of parsing Python files.
- Create module.yaml in src/NN_*/ directories
- Add YAML parser with validation in tito/core/modules.py
- Update list_modules() to display descriptions from YAML
- Update Binder URLs to use urlpath parameter pointing to generated
notebooks in modules/ directory instead of raw .py files
- Remove Colab links since they cannot run postBuild to generate notebooks
- Fix View Source links to include tinytorch/ prefix
- Fix broken symlink for 09_convolutions_ABOUT.md
- Fix header level warnings (H1 to H3 jumps) in community.md and intro.md
- Remove broken cross-references to deleted files across site pages
- Fix lexing errors by using text blocks for Unicode characters
- Update mermaid diagram in big-picture.md to use light fill colors
for PDF compatibility (mermaid-cli does not respect inline color styles)
Prerequisites now reflect intended learning sequence rather than just
code imports. Understanding why matters as much as what imports.
Pedagogical structure:
- Phase 1 Foundation (01-04): Sequential chain building core concepts
- Phase 2 Training (05-07): Autograd needs loss context, optimizers need gradients
- Phase 3 Architecture: CV track (08-09), NLP track (10-13) branch from foundation
- Phase 4 Optimization (14-19): All require models to optimize (01-13)
Key insight: A student could technically run memoization code with only
Tensor knowledge, but would not understand WHY KV caching matters without
first building attention and transformers. Prerequisites guide learning,
not just code execution.
- Update all repository references to point to harvard-edge/cs249r_book
- Fix Binder URLs to include tinytorch/ path prefix
- Fix Colab URLs to include tinytorch/ path prefix
- Update marimo-badges.js with correct repo and path
- Fix dataset documentation URLs
- Update module ABOUT.md files with correct source links
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Add sphinxsetup configuration with distinct colors for each admonition type
(tip=green, warning=orange, note=blue, caution=red, etc.)
- Convert {admonition} with :class: attribute to native {tip}, {warning},
{seealso} directives for proper Sphinx type detection in LaTeX output
- Remove unsupported emojis from site markdown files for LaTeX compatibility
- Update codespell ignore list for font names (Heros) and PDF options (FitH)
- Update 20 ABOUT.md files and 16 site/*.md files
- Replace site/modules/*_ABOUT.md files with symlinks to src/*/ABOUT.md
- src/ is now the single source of truth for module documentation
- Sync cleaned-up versions (emoji removal) from site/ back to src/
- Remove sync target from Makefile since symlinks handle everything
- Jupyter Book works with symlinks, no build changes needed
- Move 'Getting Started' section earlier (position 6, after Build → Use → Reflect)
- Add 'Common Pitfalls' section to all modules (3-5 pitfalls with code examples)
- Add 'Production Context' section to all modules (framework comparisons, real-world usage)
- Verify professional emoji usage (no emoji in section headers)
- Apply consistent structure across all 20 modules
