Package exports:
- Fix tinytorch/__init__.py to export all required components for milestones
- Add Dense as alias for Linear for compatibility
- Add loss functions (MSELoss, CrossEntropyLoss, BinaryCrossEntropyLoss)
- Export spatial operations, data loaders, and transformer components
Test infrastructure:
- Create tests/conftest.py to handle path setup
- Create tests/test_utils.py with shared test utilities
- Rename test_progressive_integration.py files to include module number
- Fix syntax errors in test files (spaces in class names)
- Remove stale test file referencing non-existent modules
Documentation:
- Update README.md with correct milestone file names
- Fix milestone requirements to match actual module dependencies
Export system:
- Run tito export --all to regenerate package from source modules
- Ensure all 20 modules are properly exported
Major directory restructure to support both developer and learner workflows:
Structure Changes:
- NEW: src/ directory for Python source files (version controlled)
- Files renamed: tensor.py → 01_tensor.py (matches directory naming)
- All 20 modules moved from modules/ to src/
- CHANGED: modules/ now holds generated notebooks (gitignored)
- Generated from src/*.py using jupytext
- Learners work in notebooks, developers work in Python source
- UNCHANGED: tinytorch/ package (still auto-generated from notebooks)
Workflow: src/*.py → modules/*.ipynb → tinytorch/*.py
Command Updates:
- Updated export command to read from src/ and generate to modules/
- Export flow: discovers modules in src/, converts to notebooks in modules/, exports to tinytorch/
- All 20 modules tested and working
Configuration:
- Updated .gitignore to ignore modules/ directory
- Updated README.md with new three-layer architecture explanation
- Updated export.py source mappings and paths
Benefits:
- Clean separation: developers edit Python, learners use notebooks
- Better version control: only Python source committed, notebooks generated
- Flexible learning: can work in notebooks OR Python source
- Maintains backward compatibility: tinytorch package unchanged
Tested:
- Single module export: tito export 01_tensor ✅
- All modules export: tito export --all ✅
- Package imports: from tinytorch.core.tensor import Tensor ✅
- 20/20 modules successfully converted and exported
Re-exported all modules after restructuring:
- Updated _modidx.py with new module locations
- Removed outdated autogeneration headers
- Updated all core modules (tensor, autograd, layers, etc.)
- Updated optimization modules (quantization, compression, etc.)
- Updated TITO commands for new structure
Changes include:
- 24 tinytorch/ module files
- 24 tito/ command and core files
- Updated references from modules/source/ to modules/
All modules re-exported via nbdev from their new locations.
- Added typing imports (List, Dict, Tuple, Optional, Set) to export section
- Fixed NameError: name 'List' is not defined
- Fixed milestone copilot references from SimpleTokenizer to CharTokenizer
- Verified transformer learning: 99.1% loss decrease in 500 steps
Training results:
- Initial loss: 3.555
- Final loss: 0.031
- Training time: 52.1s for 500 steps
- Gradient flow: All 21 parameters receiving gradients
- Model: 1-layer GPT with 32d embeddings, 4 heads
Issue: CharTokenizer was failing with NameError: name 'List' is not defined
Root cause: typing imports were not marked with #| export
Fix:
✅ Added #| export directive to import block in tokenization_dev.py
✅ Re-exported module using 'tito export 10_tokenization'
✅ typing.List, Dict, Tuple, Optional, Set now properly exported
Verification:
- CharTokenizer.build_vocab() works ✅
- encode() and decode() work ✅
- Tested on Shakespeare sample text ✅
This fixes the integration with vaswani_shakespeare.py which now properly
uses CharTokenizer from Module 10 instead of manual tokenization.
Critical fixes for transformer gradient flow:
EmbeddingBackward:
- Implements scatter-add gradient accumulation for embedding lookups
- Added to Module 05 (autograd_dev.py)
- Module 11 imports and uses it in Embedding.forward()
- Gradients now flow back to embedding weights
ReshapeBackward:
- reshape() was breaking computation graph (no _grad_fn)
- Added backward function that reshapes gradient back to original shape
- Patched Tensor.reshape() in enable_autograd()
- Critical for GPT forward pass (logits.reshape before loss)
Results:
- Before: 0/37 parameters receive gradients, loss stuck
- After: 13/37 parameters receive gradients (35%)
- Single batch overfitting: 4.46 → 0.03 (99.4% improvement!)
- MODEL NOW LEARNS! 🎉
Remaining work: 24 parameters still missing gradients (likely attention)
Tests added:
- tests/milestones/test_05_transformer_architecture.py (Phase 1)
- Multiple debug scripts to isolate issues
- Embedding.forward() now preserves requires_grad from weight tensor
- PositionalEncoding.forward() uses Tensor addition (x + pos) instead of .data
- Critical for transformer input embeddings to have gradients
Both changes ensure gradient flows from loss back to embedding weights
This commit includes:
- Exported tinytorch package files from nbdev (autograd, losses, optimizers, training, etc.)
- Updated activations.py and layers.py with __call__ methods
- New module exports: attention, spatial, tokenization, transformer, etc.
- Removed old _modidx.py file
- Cleanup of duplicate milestone directories
These are the generated package files that correspond to the source modules
we've been developing. Students will import from these when using TinyTorch.
