- Added typing imports (List, Dict, Tuple, Optional, Set) to export section
- Fixed NameError: name 'List' is not defined
- Fixed milestone copilot references from SimpleTokenizer to CharTokenizer
- Verified transformer learning: 99.1% loss decrease in 500 steps
Training results:
- Initial loss: 3.555
- Final loss: 0.031
- Training time: 52.1s for 500 steps
- Gradient flow: All 21 parameters receiving gradients
- Model: 1-layer GPT with 32d embeddings, 4 heads
- GRADIENT_FLOW_FIX_SUMMARY.md
- TRANSFORMER_VALIDATION_PLAN.md
- ENHANCEMENT_SUMMARY.md
- DEFINITIVE_MODULE_PLAN.md
- VALIDATION_SUITE_PLAN.md
These were temporary files used during development and are no longer needed.
Issue: CharTokenizer was failing with NameError: name 'List' is not defined
Root cause: typing imports were not marked with #| export
Fix:
✅ Added #| export directive to import block in tokenization_dev.py
✅ Re-exported module using 'tito export 10_tokenization'
✅ typing.List, Dict, Tuple, Optional, Set now properly exported
Verification:
- CharTokenizer.build_vocab() works ✅
- encode() and decode() work ✅
- Tested on Shakespeare sample text ✅
This fixes the integration with vaswani_shakespeare.py which now properly
uses CharTokenizer from Module 10 instead of manual tokenization.
- Implemented SoftmaxBackward with proper gradient formula
- Patched Softmax.forward() in enable_autograd()
- Fixed LayerNorm gamma/beta to have requires_grad=True
Progress:
- Softmax now correctly computes gradients
- LayerNorm parameters initialized with requires_grad
- Still debugging: Q/K/V projections, LayerNorms in blocks, MLP first layer
Current: 9/21 parameters receive gradients (was 0/21)
Critical fixes for transformer gradient flow:
EmbeddingBackward:
- Implements scatter-add gradient accumulation for embedding lookups
- Added to Module 05 (autograd_dev.py)
- Module 11 imports and uses it in Embedding.forward()
- Gradients now flow back to embedding weights
ReshapeBackward:
- reshape() was breaking computation graph (no _grad_fn)
- Added backward function that reshapes gradient back to original shape
- Patched Tensor.reshape() in enable_autograd()
- Critical for GPT forward pass (logits.reshape before loss)
Results:
- Before: 0/37 parameters receive gradients, loss stuck
- After: 13/37 parameters receive gradients (35%)
- Single batch overfitting: 4.46 → 0.03 (99.4% improvement!)
- MODEL NOW LEARNS! 🎉
Remaining work: 24 parameters still missing gradients (likely attention)
Tests added:
- tests/milestones/test_05_transformer_architecture.py (Phase 1)
- Multiple debug scripts to isolate issues
TransposeBackward:
- New backward function for transpose operation
- Patch Tensor.transpose() to track gradients
- Critical for attention (Q @ K.T) gradient flow
MatmulBackward batched fix:
- Change np.dot to np.matmul for batched 3D+ tensors
- Use np.swapaxes instead of .T for proper batched transpose
- Fixes gradient shapes in attention mechanisms
Tests added:
- tests/05_autograd/test_batched_matmul_backward.py (3 tests)
- Updated tests/regression/test_gradient_flow_fixes.py (9 tests total)
All gradient flow issues for transformer training are now resolved!
- Change from .data extraction to Tensor arithmetic (x - mean, diff * diff, x / std)
- Preserve computation graph through normalization
- std tensor now preserves requires_grad correctly
LayerNorm is used before and after attention in transformer blocks
Major rewrite for gradient flow:
- scaled_dot_product_attention: Use Tensor ops (matmul, transpose, softmax)
- MultiHeadAttention: Process all heads in parallel with 4D batched tensors
- No explicit batch loops or .data extraction
- Proper mask broadcasting for (batch * heads) dimension
This is the most complex fix - attention is now fully differentiable end-to-end
- Embedding.forward() now preserves requires_grad from weight tensor
- PositionalEncoding.forward() uses Tensor addition (x + pos) instead of .data
- Critical for transformer input embeddings to have gradients
Both changes ensure gradient flows from loss back to embedding weights
- Implement gradient functions for subtraction and division operations
- Patch Tensor.__sub__ and Tensor.__truediv__ in enable_autograd()
- Required for LayerNorm (x - mean) and (normalized / std) operations
These operations are used extensively in normalization layers
- Preserve computation graph by using Tensor arithmetic (x - x_max, exp / sum)
- No more .data extraction that breaks gradient flow
- Numerically stable with max subtraction before exp
Required for transformer attention softmax gradient flow
- Add CLAUDE.md entry point for Claude AI system
- Fix tito test command to set PYTHONPATH for module imports
- Fix embeddings export directive placement for nbdev
- Fix attention module to export imports properly
- Fix transformers embedding index casting to int
- Update transformers module to match tokenization style with improved ASCII diagrams
- Fix attention module to use proper multi-head interface
- Update transformer era milestone for refined module integration
- Fix import paths and ensure forward() method consistency
- All transformer components now work seamlessly together
Following module developer guidelines, added comprehensive visual diagrams:
1. Text-to-Numbers Pipeline (Introduction):
- Added full boxed diagram showing 4-step tokenization process
- Clear visual flow from human text to numerical IDs
- Each step explained inline with the diagram
2. Character Tokenization Process:
- Step-by-step vocabulary building visualization
- Shows corpus → unique chars → vocab with IDs
- Encoding process with ID lookup visualization
- Decoding process with reverse lookup
- All in clear nested boxes
3. BPE Training Algorithm:
- Comprehensive 4-step process with nested boxes
- Pair frequency analysis with bar charts (████)
- Before/After merge visualizations
- Iteration examples showing vocabulary growth
- Final results with key insights
4. Memory Layout for Embedding Tables:
- Visual bars showing relative memory sizes
- Character (204KB) vs BPE-50K (102MB) vs Word-100K (204MB)
- Shows fp32/fp16/int8 precision trade-offs
- Real production model examples (GPT-2/3, BERT, T5, LLaMA)
- Clear table format for comparison
Educational improvements:
- More visual, less text-heavy
- Clearer step-by-step flows
- Better intuition building
- Production context throughout
- Following module developer ASCII diagram patterns
Students now see:
- HOW tokenization works (not just WHAT)
- WHY different strategies exist
- WHAT the memory implications are
- HOW production models make these choices
Changes:
- Removed broken _SimplifiedTensor and internal Module helper classes
- Updated imports to use tinytorch.core instead of dev modules
- Removed Module inheritance from Conv2d, MaxPool2d, AvgPool2d, SimpleCNN
- All spatial classes now standalone like Linear in layers module
This allows spatial module to export cleanly and import correctly:
from tinytorch.core.spatial import Conv2d, MaxPool2d, AvgPool2d
Smoke test: Conv2d(1,3,8,8) → (1,16,6,6) ✓
Added integration tests for DataLoader:
- test_dataloader_integration.py in tests/integration/
- Training workflow integration
- Shuffle consistency across epochs
- Memory efficiency verification
Updated Module 08:
- Added note about optional performance analysis
- Clarified that analysis functions can be run manually
- Clean flow: text → code → tests
Updated datasets/tiny/README.md:
- Minor formatting fixes
Module 08 is now complete and ready to export:
✅ Dataset abstraction
✅ TensorDataset implementation
✅ DataLoader with batching/shuffling
✅ ASCII visualizations for understanding
✅ Unit tests (in module)
✅ Integration tests (in tests/)
✅ Performance analysis tools (optional)
Next: Export with 'bin/tito export 08_dataloader'
Fixed issue where performance analysis functions were called every time
the module was imported, instead of only when needed.
Changes:
- Commented out analyze_dataloader_performance() bare call
- Commented out analyze_memory_usage() bare call
- Removed redundant test_training_integration() comment
These functions are still defined and can be called manually for
performance insights, but won't run on every import.
The test_module() function still calls all necessary tests when
the module is run as __main__.
Result: Module imports cleanly without running expensive performance
benchmarks unless explicitly requested.
Added educational ASCII art showing:
1. **Actual pixel values** - What 8×8 digit images look like as numbers
- Shows digits 5, 3, and 8 with real pixel values (0-16 range)
- Helps students understand images are just 2D arrays
2. **Visual representation** - How humans see the digits
- ASCII art showing recognizable digit shapes
- Connects abstract numbers to concrete patterns
3. **Shape transformations** - How DataLoader batches data
- Individual: (8, 8) → Batched: (32, 8, 8)
- Shows what the model actually receives
4. **Complete example** - Loading and using tiny digits dataset
- Real code showing datasets/tiny/digits_8x8.npz usage
- Demonstrates the full DataLoader workflow
Benefits:
✅ Students visualize what image data IS
✅ Understand DataLoader's batching transformation
✅ See connection between numbers and visual patterns
✅ Ready to work with real datasets in milestones
This makes the abstract concept of 'image tensors' concrete and visual.
Issue: xor_crisis.py was failing with ImportError on matplotlib architecture mismatch
Root cause: losses_dev.py imported matplotlib.pyplot but never used it
Fix:
- ✅ Removed unused imports: matplotlib.pyplot, time
- ✅ Re-exported module 04_losses to update tinytorch package
- ✅ Verified both milestone 02 scripts now run successfully
The matplotlib import was causing failures on M2 Macs where matplotlib
was installed for wrong architecture (x86_64 vs arm64). Since it was
never used, removing it eliminates the dependency entirely.
Tested:
- ✅ milestones/02_xor_crisis_1969/xor_crisis.py (49% accuracy - expected failure)
- ✅ milestones/02_xor_crisis_1969/xor_solved.py (100% accuracy - perfect!)
New Features:
- Add ReLUBackward for proper ReLU gradient computation
- Patch ReLU.forward() in enable_autograd() for gradient tracking
- Create polished XOR milestone scripts matching perceptron style
XOR Milestone Scripts (milestones/02_xor_crisis_1969/):
- xor_crisis.py: Shows single-layer perceptron FAILING (~50% accuracy)
- xor_solved.py: Shows multi-layer network SUCCEEDING (75%+ accuracy)
- Beautiful rich output with tables, panels, historical context
- Pedagogically structured like the perceptron milestone
Results:
✅ Single-layer: Stuck at ~50% (proves the crisis)
✅ Multi-layer: 75% accuracy (proves hidden layers work!)
✅ ReLU gradients flow correctly through network
✅ All 4 core activations now support autograd:
- Sigmoid ✓, ReLU ✓, Tanh ✓ (future), GELU ✓ (future)
Historical Significance:
This recreates the exact problem that killed AI for 17 years
and demonstrates the solution that started the modern era!
Refactored gradient accumulation to use clearer two-step approach:
1. Remove extra leading dimensions (batch dims)
2. Sum over dimensions that were size-1 (broadcast dims)
Benefits:
- Clearer intent: while loop for variable dims, for loop for fixed dims
- Better comments with concrete examples
- Easier for students to understand broadcasting in backprop
- Matches how you'd explain it verbally
Same functionality, cleaner code.
Added:
- SigmoidBackward class to modules/source/05_autograd/autograd_dev.py with #| export
- BCEBackward class to modules/source/05_autograd/autograd_dev.py with #| export
- Both classes exported to tinytorch/core/autograd.py
- Updated Sigmoid activation to track gradients using SigmoidBackward
- Updated BCE loss to track gradients using BCEBackward
ISSUE: Training still not learning - gradients not flowing properly
- Loss stays constant at 0.7911
- Weights don't update
- Sigmoid.forward() code looks correct but a.requires_grad stays False
- Need to investigate why gradient tracking isn't working through activations
Updated docstring examples to use cleaner callable syntax:
- loss_fn(predictions, targets) instead of loss_fn.forward(predictions, targets)
Applied to:
- MSELoss
- CrossEntropyLoss
- BinaryCrossEntropyLoss
Demonstrates proper usage with __call__ methods for cleaner, more Pythonic code.
Updated docstring examples to use cleaner callable syntax:
- sigmoid(x) instead of sigmoid.forward(x)
- relu(x) instead of relu.forward(x)
- tanh(x) instead of tanh.forward(x)
- gelu(x) instead of gelu.forward(x)
- softmax(x) instead of softmax.forward(x)
This demonstrates the proper usage pattern with the __call__ methods
we just added, making examples more Pythonic and PyTorch-compatible.
Enable cleaner API usage by adding __call__ methods to all activation,
layer, and loss classes. This allows students to write:
- relu(x) instead of relu.forward(x)
- layer(x) instead of layer.forward(x)
- loss_fn(pred, target) instead of loss_fn.forward(pred, target)
Changes:
- Module 02 (Activations): Add __call__ to ReLU, Tanh, GELU, Softmax
* Sigmoid already had __call__
- Module 03 (Layers): Add __call__ to Dropout
* Linear already had __call__
- Module 04 (Losses): Add __call__ to MSELoss, CrossEntropyLoss, BinaryCrossEntropyLoss
This matches PyTorch's API convention where model(x) calls model.__call__(x)
which internally calls model.forward(x). Makes code more Pythonic and
intuitive for students familiar with PyTorch.
Expected impact: Test pass rates should improve significantly as tests
expect PyTorch-style callable API.
- Reorganized milestone structure to historical progression (01-06)
- Created single forward_pass.py with student code clearly at top
- Added Rich CLI visualizations: data scatter, network diagram, decision boundary
- Show decision boundary using / or \ based on slope
- No random seed - students see variability in random weights
- Annotated all code with which modules were used (Modules 01-03)
- Added introductory panel explaining what to expect
- Updated DEFINITIVE_MODULE_PLAN.md with corrected milestone structure
PROBLEM:
- nbdev requires #| export directive on EACH cell to export when using # %% markers
- Cell markers inside class definitions split classes across multiple cells
- Only partial classes were being exported to tinytorch package
- Missing matmul, arithmetic operations, and activation classes in exports
SOLUTION:
1. Removed # %% cell markers INSIDE class definitions (kept classes as single units)
2. Added #| export to imports cell at top of each module
3. Added #| export before each exportable class definition in all 20 modules
4. Added __call__ method to Sigmoid for functional usage
5. Fixed numpy import (moved to module level from __init__)
MODULES FIXED:
- 01_tensor: Tensor class with all operations (matmul, arithmetic, shape ops)
- 02_activations: Sigmoid, ReLU, Tanh, GELU, Softmax classes
- 03_layers: Linear, Dropout classes
- 04_losses: MSELoss, CrossEntropyLoss, BinaryCrossEntropyLoss classes
- 05_autograd: Function, AddBackward, MulBackward, MatmulBackward, SumBackward
- 06_optimizers: Optimizer, SGD, Adam, AdamW classes
- 07_training: CosineSchedule, Trainer classes
- 08_dataloader: Dataset, TensorDataset, DataLoader classes
- 09_spatial: Conv2d, MaxPool2d, AvgPool2d, SimpleCNN classes
- 10-20: All exportable classes in remaining modules
TESTING:
- Test functions use 'if __name__ == "__main__"' guards
- Tests run in notebooks but NOT on import
- Rosenblatt Perceptron milestone working perfectly
RESULT:
✅ All 20 modules export correctly
✅ Perceptron (1957) milestone functional
✅ Clean separation: development (modules/source) vs package (tinytorch)
