3fc83f95d6
- Phase 2 now: Autograd → Spatial → Optimizers → DataLoader → Training - Move Spatial (CNNs) from Phase 3 to Phase 2 Module 7 - Integrate BatchNorm into Spatial module (mirrors PyTorch patterns) - Fix milestone: CNN training achievable at end of Phase 2 (Module 10) - Phase 3 focuses on language: Tokenization → Embeddings → Attention → Transformers - Logical dependency flow: understand conv operations before optimizing them
TinyTorch 🔥
Build ML Systems From First Principles
A Harvard University course that teaches ML systems engineering by building a complete deep learning framework from scratch. From tensors to transformers, understand every line of code powering modern AI.
🎯 What You'll Build
A complete ML framework capable of:
- Training neural networks on CIFAR-10 to 55%+ accuracy (reliably achievable!)
- Building GPT-style language models
- Implementing modern optimizers (Adam, learning rate scheduling)
- Production deployment with monitoring and MLOps
All built from scratch using only NumPy - no PyTorch, no TensorFlow!
🚀 Quick Start
# Clone and setup
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .
# Start learning
cd modules/source/01_setup
jupyter lab setup_dev.py
# Track progress
tito checkpoint status
📚 Three-Part Learning Journey
17 Progressive Modules - Complete Any Part for Industry-Ready Skills!
Part I: Foundations (Modules 1-5)
"I can build neural networks from scratch!"
| Module | Topic | What You Build |
|---|---|---|
| 01 | Setup | Development environment |
| 02 | Tensors | N-dimensional arrays |
| 03 | Activations | ReLU, Sigmoid, Softmax |
| 04 | Layers | Dense layers |
| 05 | Networks | Multi-layer networks |
✅ Capstone: XORNet - Solve non-linear problems
Part II: Computer Vision (Modules 6-11)
"I can build CNNs that classify real images!"
| Module | Topic | What You Build |
|---|---|---|
| 06 | Spatial | Conv2D, Pooling |
| 07 | DataLoader | Efficient data pipelines |
| 08 | Normalization | BatchNorm, LayerNorm |
| 09 | Autograd | Automatic differentiation |
| 10 | Optimizers | SGD, Adam |
| 11 | Training | Complete training loops |
✅ Capstone: CIFAR-10 CNN - 55%+ accuracy on real images
Part III: Language Models (Modules 12-17)
"I can build transformers that generate text!"
| Module | Topic | What You Build |
|---|---|---|
| 12 | Embeddings | Token embeddings, positional encoding |
| 13 | Attention | Multi-head attention |
| 14 | Transformers | Transformer blocks |
| 15 | Generation | Autoregressive decoding |
| 16 | Regularization | Dropout, robustness |
| 17 | Systems | Production deployment |
✅ Capstone: TinyGPT - Generate text with transformers
🎓 Learning Philosophy
Most courses teach you to USE frameworks. TinyTorch teaches you to UNDERSTAND them.
# Traditional Course:
import torch
model.fit(X, y) # Magic happens
# TinyTorch:
# You implement every component
# You measure memory usage
# You optimize performance
# You understand the systems
Why Build Your Own Framework?
✅ Deep Understanding - Know exactly what loss.backward() does
✅ Systems Thinking - Understand memory, compute, and scaling
✅ Debugging Skills - Fix problems at any level of the stack
✅ Production Ready - Learn patterns used in real ML systems
🛠️ Key Features
For Students
- Interactive Demos: Rich CLI visualizations for every concept
- Checkpoint System: Track your learning progress
- Immediate Testing: Validate your implementations instantly
- Real Datasets: Train on CIFAR-10, not toy examples
For Instructors
- NBGrader Integration: Automated grading workflow
- Progress Tracking: Monitor student achievements
- Jupyter Book: Professional course website
- Complete Solutions: Reference implementations included
🔥 Examples You Can Run
As you complete modules, exciting examples unlock to show your framework in action:
After Module 05 → examples/xornet/ 🔥
cd examples/xornet
python train.py
# 🎯 100% accuracy on XOR problem!
After Module 11 → examples/cifar10/ 🎯
cd examples/cifar10
python train.py
# 🏆 55%+ accuracy on real images!
These aren't toy demos - they're real ML applications achieving solid results with YOUR framework built from scratch following KISS principles!
🧪 Testing & Validation
All demos and modules are thoroughly tested:
# Run comprehensive test suite (recommended)
tito test --comprehensive
# Run checkpoint tests
tito checkpoint test 01
# Test specific modules
tito test --module tensor
# Run all module tests
python tests/run_all_modules.py
✅ 16 modules passing all tests with 100% health status
✅ 16 capability checkpoints tracking learning progress
✅ Comprehensive testing framework with module and integration tests
✅ KISS principle design for clear, maintainable code
📖 Documentation
- Course Website - Complete interactive course
- Instructor Guide - Teaching resources
- Student Quickstart - Getting started guide
- CIFAR-10 Training Guide - Detailed training walkthrough
🤝 Contributing
We welcome contributions! See CONTRIBUTING.md for guidelines.
📄 License
MIT License - see LICENSE for details.
🙏 Acknowledgments
Created by Prof. Vijay Janapa Reddi at Harvard University.
Special thanks to students and contributors who helped refine this educational framework.
Start Small. Go Deep. Build ML Systems.