ab23b23daa
Core Fixes: - Fixed Variable/Tensor data access in validation system - Regenerated training module with proper loss functions - Identified original CIFAR-10 script timing issues Working Examples: - XOR network: 100% accuracy (verified working) - CIFAR-10 MLP: 49.2% accuracy in 18 seconds (realistic timing) - Component tests: All core functionality verified Key improvements: - Realistic training parameters (200 batches/epoch vs 500) - Smaller model for faster iteration (512→256→10 vs 1024→512→256→128→10) - Simple augmentation to avoid training bottlenecks - Comprehensive logging to track training progress Performance verified: - XOR: 100% accuracy proving autograd works correctly - CIFAR-10: 49.2% accuracy (much better than 10% random, approaching 50-55% benchmarks) - Training time: 18 seconds (practical for educational use)
TinyTorch 🔥
Build ML Systems From First Principles
A Harvard University course that teaches ML systems engineering by building a complete deep learning framework from scratch. From tensors to transformers, understand every line of code powering modern AI.
🎯 What You'll Build
A complete ML framework capable of:
- Training neural networks on CIFAR-10 to 57%+ accuracy (exceeds course benchmarks!)
- Building GPT-style language models
- Implementing modern optimizers (Adam, learning rate scheduling)
- Production deployment with monitoring and MLOps
All built from scratch using only NumPy - no PyTorch, no TensorFlow!
🚀 Quick Start
# Clone and setup
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .
# Start learning
cd modules/source/01_setup
jupyter lab setup_dev.py
# Track progress
tito checkpoint status
📚 Course Structure
16 Progressive Modules
| Module | Topic | What You Build |
|---|---|---|
| Foundations | ||
| 01 | Setup | Development environment |
| 02 | Tensors | N-dimensional arrays |
| 03 | Activations | ReLU, Sigmoid, Softmax |
| 04 | Layers | Dense layers |
| 05 | Networks | Sequential models |
| Deep Learning | ||
| 06 | Spatial | CNNs for vision |
| 07 | Attention | Transformers |
| 08 | DataLoader | Efficient data pipelines |
| 09 | Autograd | Automatic differentiation |
| 10 | Optimizers | SGD, Adam |
| Production | ||
| 11 | Training | Complete training loops |
| 12 | Compression | Model optimization |
| 13 | Kernels | Performance optimization |
| 14 | Benchmarking | Profiling tools |
| 15 | MLOps | Production deployment |
| Language Models | ||
| 16 | TinyGPT | Complete GPT implementation |
🎓 Learning Philosophy
Most courses teach you to USE frameworks. TinyTorch teaches you to UNDERSTAND them.
# Traditional Course:
import torch
model.fit(X, y) # Magic happens
# TinyTorch:
# You implement every component
# You measure memory usage
# You optimize performance
# You understand the systems
Why Build Your Own Framework?
✅ Deep Understanding - Know exactly what loss.backward() does
✅ Systems Thinking - Understand memory, compute, and scaling
✅ Debugging Skills - Fix problems at any level of the stack
✅ Production Ready - Learn patterns used in real ML systems
🛠️ Key Features
For Students
- Interactive Demos: Rich CLI visualizations for every concept
- Checkpoint System: Track your learning progress
- Immediate Testing: Validate your implementations instantly
- Real Datasets: Train on CIFAR-10, not toy examples
For Instructors
- NBGrader Integration: Automated grading workflow
- Progress Tracking: Monitor student achievements
- Jupyter Book: Professional course website
- Complete Solutions: Reference implementations included
🔥 Examples You Can Run
As you complete modules, exciting examples unlock to show your framework in action:
After Module 05 → examples/xornet/ 🔥
cd examples/xornet
python train.py
# 🎯 100% accuracy on XOR problem!
After Module 11 → examples/cifar10/ 🎯
cd examples/cifar10
python train_cifar10_mlp.py
# 🏆 57.2% accuracy on real images!
These aren't toy demos - they're real ML applications achieving competitive results with YOUR framework built from scratch!
🧪 Testing & Validation
All demos and modules are thoroughly tested:
# Test all demos
python test_all_demos.py
# Validate implementations
python validate_demos.py
# Run checkpoint tests
tito checkpoint test 01
# Run all module tests
python tests/run_all_modules.py
✅ 9 interactive demos covering all major concepts
✅ 16 capability checkpoints tracking learning progress
✅ Comprehensive test suite with module and integration tests
📖 Documentation
- Course Website - Complete interactive course
- Instructor Guide - Teaching resources
- Student Quickstart - Getting started guide
- CIFAR-10 Training Guide - Detailed training walkthrough
🤝 Contributing
We welcome contributions! See CONTRIBUTING.md for guidelines.
📄 License
MIT License - see LICENSE for details.
🙏 Acknowledgments
Created by Prof. Vijay Janapa Reddi at Harvard University.
Special thanks to students and contributors who helped refine this educational framework.
Start Small. Go Deep. Build ML Systems.