# ๐ฅ TinyTorch
### Build Your Own ML Framework From Scratch
[](https://github.com/harvard-edge/cs249r_book/actions/workflows/tinytorch-ci.yml)
[](https://mlsysbook.ai/tinytorch)
[](https://python.org)
[](LICENSE)
[](https://mlsysbook.ai)
**Most ML courses teach you to *use* frameworks. TinyTorch teaches you to *build* them.**
[Get Started](#-quick-start) ยท [20 Modules](#-20-progressive-modules) ยท [Documentation](https://mlsysbook.ai/tinytorch) ยท [Contributing](CONTRIBUTING.md)
---
> ๐งช **Early Access** - TinyTorch is available for early adopters! All 20 modules are implemented with complete solutions and the `tito` CLI for streamlined workflows.
>
> ๐ฏ **Spring 2025**: Full public release with community features and classroom deployment resources.
---
## Why TinyTorch?
**"Most ML education teaches you to _use_ frameworks. TinyTorch teaches you to _build_ them."**
The difference between ML users and ML engineers determines who drives innovation versus who merely consumes it. TinyTorch bridges this gap by teaching you to build every component of modern AI systems from scratch.
A Harvard University course that transforms you from framework user to systems engineer, giving you the deep understanding needed to optimize, debug, and innovate at the foundation of AI.
---
## What You'll Build
A **complete ML framework** capable of:
๐ฏ **North Star Achievement**: Train CNNs for image classification
- Real computer vision on standard benchmark datasets
- Built entirely from scratch using only NumPy
- Competitive performance with modern frameworks
**Additional Capabilities**:
- GPT-style language models with attention mechanisms
- Modern optimizers (Adam, SGD) with learning rate scheduling
- Performance profiling, optimization, and competitive benchmarking
**No dependencies on PyTorch or TensorFlow - everything is YOUR code!**
---
## Quick Start
```bash
# One-line install
curl -sSL tinytorch.ai/install | bash
# Activate and get started
cd tinytorch
source .venv/bin/activate
tito setup
# Start building!
tito module start 01
```
> **Tip**: Run the installer from a project folder (e.g., `~/projects`). It will create a `tinytorch/` directory with everything you need. Update anytime with `tito update`.
---
## 20 Progressive Modules
Build your framework through four progressive parts:
| Part | Modules | What You Build |
|------|---------|----------------|
| **I. Foundations** | 01-07 | Tensors, activations, layers, losses, autograd, optimizers, training |
| **II. Vision** | 08-09 | DataLoaders, Conv2d, CNNs for image classification |
| **III. Language** | 10-13 | Tokenization, embeddings, attention, transformers |
| **IV. Optimization** | 14-20 | Profiling, quantization, compression, acceleration, benchmarking, capstone |
Each module asks: **"Can I build this capability from scratch?"**
๐ **[Full curriculum and module details โ](https://mlsysbook.ai/tinytorch)**
---
## Historical Milestones
As you progress, unlock recreations of landmark ML achievements:
| Year | Milestone | Your Achievement |
|------|-----------|------------------|
| 1957 | Perceptron | Binary classification with gradient descent |
| 1969 | XOR Crisis | Multi-layer networks solve non-linear problems |
| 1986 | Backpropagation | Multi-layer network training |
| 1998 | CNN Revolution | **Image classification with convolutions** |
| 2017 | Transformer Era | Language generation with self-attention |
| 2018+ | MLPerf | Production-ready optimization |
**These aren't toy demos** - they're historically significant ML achievements rebuilt with YOUR framework!
---
## Learning Philosophy
```python
# Traditional Course:
import torch
model.fit(X, y) # Magic happens
# TinyTorch:
# You implement every component
# You measure memory usage
# You optimize performance
# You understand the systems
```
**Why Build Your Own Framework?**
- **Deep Understanding** - Know exactly what `loss.backward()` does
- **Systems Thinking** - Understand memory, compute, and scaling
- **Debugging Skills** - Fix problems at any level of the stack
- **Production Ready** - Learn patterns used in real ML systems
---
## Documentation
| Audience | Resources |
|----------|-----------|
| **Students** | [Course Website](https://mlsysbook.ai/tinytorch) ใป [Quick Start](site/STUDENT_QUICKSTART.md) ใป [FAQ](site/faq.md) |
| **Instructors** | [Instructor Guide](INSTRUCTOR.md) ใป [NBGrader Setup](site/nbgrader/) ใป [TA Guide](TA_GUIDE.md) |
| **Contributors** | [Contributing Guide](CONTRIBUTING.md) ใป [Module Development](site/development/module-rules.md) |
---
## Repository Structure
```
TinyTorch/
โโโ src/ # ๐ป Python source files (developers/contributors edit here)
โ โโโ 01_tensor/ # Module 01: Tensor operations from scratch
โ โ โโโ 01_tensor.py # Python source (version controlled)
โ โ โโโ ABOUT.md # Conceptual overview & learning objectives
โ โโโ 02_activations/ # Module 02: ReLU, Softmax activations
โ โโโ 03_layers/ # Module 03: Linear layers, Module system
โ โโโ 04_losses/ # Module 04: MSE, CrossEntropy losses
โ โโโ 05_autograd/ # Module 05: Automatic differentiation
โ โโโ 06_optimizers/ # Module 06: SGD, Adam optimizers
โ โโโ 07_training/ # Module 07: Complete training loops
โ โโโ 08_dataloader/ # Module 08: Efficient data pipelines
โ โโโ 09_spatial/ # Module 09: Conv2d, MaxPool2d, CNNs
โ โโโ 10_tokenization/ # Module 10: Text processing
โ โโโ 11_embeddings/ # Module 11: Token & positional embeddings
โ โโโ 12_attention/ # Module 12: Multi-head attention
โ โโโ 13_transformers/ # Module 13: Complete transformer blocks
โ โโโ 14_profiling/ # Module 14: Performance analysis
โ โโโ 15_quantization/ # Module 15: Model compression (precision reduction)
โ โโโ 16_compression/ # Module 16: Pruning & distillation
โ โโโ 17_memoization/ # Module 17: KV-cache/memoization
โ โโโ 18_acceleration/ # Module 18: Hardware optimization
โ โโโ 19_benchmarking/ # Module 19: Performance measurement
โ โโโ 20_capstone/ # Module 20: Complete ML systems
โ
โโโ modules/ # ๐ Generated notebooks (learners work here)
โ โโโ 01_tensor/ # Auto-generated from src/
โ โ โโโ 01_tensor.ipynb # Jupyter notebook for learning
โ โ โโโ README.md # Practical implementation guide
โ โ โโโ tensor.py # Your implementation
โ โโโ ... # (20 module directories)
โ
โโโ site/ # ๐ Course website & documentation (Jupyter Book)
โ โโโ intro.md # Landing page
โ โโโ _toc.yml # Site navigation (links to modules)
โ โโโ _config.yml # HTML website configuration
โ โโโ chapters/ # Course content chapters
โ โโโ modules/ # Module documentation
โ
โโโ milestones/ # ๐ Historical ML evolution - prove what you built!
โ โโโ 01_1957_perceptron/ # Rosenblatt's first trainable network
โ โโโ 02_1969_xor/ # Minsky's challenge & multi-layer solution
โ โโโ 03_1986_mlp/ # Backpropagation & MNIST digits
โ โโโ 04_1998_cnn/ # LeCun's CNNs & CIFAR-10
โ โโโ 05_2017_transformer/ # Attention mechanisms & language
โ โโโ 06_2018_mlperf/ # Modern optimization & profiling
โ
โโโ tito/ # ๐๏ธ CLI tool for streamlined workflows
โ โโโ main.py # Entry point
โ โโโ commands/ # 23 command modules
โ โโโ core/ # Core utilities
โ
โโโ tinytorch/ # ๐ฆ Generated package (import from here)
โ โโโ core/ # Core ML components
โ โโโ ... # Your built framework!
โ
โโโ tests/ # โ
Comprehensive test suite (600+ tests)
```
**Key workflow**: `src/*.py` โ `modules/*.ipynb` โ `tinytorch/*.py`
---
## Join the Community
TinyTorch is part of the [ML Systems Book](https://mlsysbook.ai) ecosystem. We're building an open community of learners and educators passionate about ML systems.
**Ways to get involved:**
- โญ Star this repo to show support
- ๐ฌ Join [Discussions](https://github.com/harvard-edge/cs249r_book/discussions) to ask questions
- ๐ Report issues or suggest improvements
- ๐ค Contribute modules, fixes, or documentation
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
---
## Related Projects
"TinyTorch" is a popular name for educational ML frameworks. We acknowledge excellent projects with similar names:
- [tinygrad](https://github.com/tinygrad/tinygrad) - George Hotz's minimalist framework
- [micrograd](https://github.com/karpathy/micrograd) - Andrej Karpathy's tiny autograd
- [MiniTorch](https://minitorch.github.io/) - Cornell's educational framework
**Our TinyTorch** distinguishes itself through its 20-module curriculum, NBGrader integration, ML systems focus, and connection to the [ML Systems Book](https://mlsysbook.ai) ecosystem.
---
## Acknowledgments
Created by [Prof. Vijay Janapa Reddi](https://vijay.seas.harvard.edu) at Harvard University.
Special thanks to students and contributors who helped build this framework.
---
## License
MIT License - see [LICENSE](LICENSE) for details.
---
**[๐ Full Documentation](https://mlsysbook.ai/tinytorch)** ใป **[๐ฌ Discussions](https://github.com/harvard-edge/cs249r_book/discussions)** ใป **[๐ ML Systems Book](https://mlsysbook.ai)**
**Start Small. Go Deep. Build ML Systems.**