# TinyTorch Three-Part Learning Journey šŸš€ ## Overview TinyTorch is structured as a progressive three-part journey, where each part builds toward a concrete achievement. Students can complete any part and have built something meaningful! ## Part I: Foundations (Modules 1-5) **"I can build neural networks from scratch!"** ### Modules 1. **01_setup** - Development environment and tools 2. **02_tensor** - Core data structure and operations 3. **03_activations** - Non-linearity (the key to intelligence!) 4. **04_layers** - Dense layers and matrix operations 5. **05_networks** - Multi-layer neural networks ### Capstone Achievement āœ… **XORNet** - Solve the classic XOR problem that proves you understand non-linear learning ### What You Learn - How tensors store and manipulate data - Why activation functions enable intelligence - How layers compose into networks - Memory layouts and computational complexity - Building blocks of all neural networks --- ## Part II: Computer Vision (Modules 6-11) **"I can build CNNs that classify real images!"** ### Modules 6. **06_spatial** - Convolutions and pooling for image processing 7. **07_dataloader** - Efficient data pipelines and batching 8. **08_normalization** - BatchNorm and LayerNorm for stable training 9. **09_autograd** - Automatic differentiation and computational graphs 10. **10_optimizers** - SGD, Adam, and gradient descent 11. **11_training** - Complete training loops with validation ### Capstone Achievement āœ… **CIFAR-10 CNN** - Classify real 32x32 color images at 55%+ accuracy (5.5x better than random!) ### What You Learn - How convolutions extract spatial features - Data pipeline engineering for real datasets - Why normalization prevents gradient problems - How autograd enables learning - Memory vs speed tradeoffs in optimization - Production training techniques (checkpointing, early stopping) --- ## Part III: Language Models (Modules 12-17) **"I can build transformers that generate text!"** ### Modules 12. **12_embeddings** - Token embeddings and positional encoding 13. **13_attention** - Multi-head attention mechanisms 14. **14_transformers** - Transformer blocks and architectures 15. **15_generation** - Autoregressive decoding and sampling strategies 16. **16_regularization** - Dropout, weight decay, and robustness 17. **17_systems** - Production deployment, optimization, and monitoring ### Capstone Achievement āœ… **TinyGPT** - Generate coherent text character-by-character using transformers ### What You Learn - Why embeddings are often the largest parameters - O(NĀ²) attention complexity and memory bottlenecks - How transformers process sequences in parallel - Temperature, top-k, nucleus sampling tradeoffs - Production ML systems engineering - Deployment, monitoring, and optimization --- ## Progressive Learning Path ```mermaid graph LR A[Start] --> B[Part I: Foundations] B --> C{Can build MLPs} C --> D[Part II: Vision] D --> E{Can build CNNs} E --> F[Part III: Language] F --> G{Can build GPT} C -.->|Exit Point 1| H[Industry Ready:
ML Engineer] E -.->|Exit Point 2| I[Industry Ready:
Computer Vision] G -.->|Exit Point 3| J[Industry Ready:
LLM Engineer] ``` ## Natural Exit Points ### After Part I (Modules 1-5) - **You've built**: Neural networks from scratch - **You understand**: Core ML building blocks - **Industry relevance**: Ready for ML engineering roles - **Concrete proof**: Working XORNet ### After Part II (Modules 6-11) - **You've built**: Complete CNN training pipeline - **You understand**: Real data processing at scale - **Industry relevance**: Ready for computer vision roles - **Concrete proof**: CIFAR-10 at 55% accuracy ### After Part III (Modules 12-17) - **You've built**: Transformer-based language model - **You understand**: Modern LLM architectures - **Industry relevance**: Ready for NLP/LLM engineering - **Concrete proof**: Text-generating TinyGPT --- ## Alignment with MLSysBook.ai This structure perfectly complements the [ML Systems textbook](https://mlsysbook.ai): | Book Section | TinyTorch Part | What You Build | |-------------|----------------|----------------| | Ch 1-4: Foundations | Part I (Modules 1-5) | Neural Networks | | Ch 5-8: Design Principles | Part II (Modules 6-11) | CNNs & Training | | Ch 9-12: Performance | Part III (Modules 12-17) | Transformers | | Ch 13-20: Production | Integrated Throughout | Real Systems | ## Why This Structure Works 1. **Clear Progression**: Each part builds on the previous 2. **Concrete Achievements**: XOR ā†’ CIFAR-10 ā†’ GPT 3. **Industry Aligned**: MLP ā†’ CNN ā†’ Transformer mirrors ML history 4. **Flexible Duration**: Complete 1, 2, or all 3 parts based on course length 5. **Systems Focus**: Every module teaches ML systems engineering, not just algorithms --- ## Module Dependency Graph ``` Part I (Foundations) ā”œā”€ā”€ 01_setup ā”œā”€ā”€ 02_tensor ā† Foundation for everything ā”œā”€ā”€ 03_activations ā† Requires tensor ā”œā”€ā”€ 04_layers ā† Requires tensor, activations ā””ā”€ā”€ 05_networks ā† Requires layers Part II (Computer Vision) ā”œā”€ā”€ 06_spatial ā† Requires tensor, layers ā”œā”€ā”€ 07_dataloader ā† Requires tensor ā”œā”€ā”€ 08_normalization ā† Requires tensor, layers ā”œā”€ā”€ 09_autograd ā† Requires tensor, networks ā”œā”€ā”€ 10_optimizers ā† Requires autograd ā””ā”€ā”€ 11_training ā† Requires all above Part III (Language Models) ā”œā”€ā”€ 12_embeddings ā† Requires tensor, layers ā”œā”€ā”€ 13_attention ā† Requires tensor, layers ā”œā”€ā”€ 14_transformers ā† Requires attention, normalization ā”œā”€ā”€ 15_generation ā† Requires transformers ā”œā”€ā”€ 16_regularization ā† Enhancement for all models ā””ā”€ā”€ 17_systems ā† Production engineering ``` --- ## For Instructors ### Semester Planning Options **Quarter System (10 weeks)** - Weeks 1-4: Part I (Foundations) - Weeks 5-9: Part II (Computer Vision) - Week 10: Final project with XORNet or CIFAR-10 **Semester System (15 weeks)** - Weeks 1-3: Part I (Foundations) - Weeks 4-8: Part II (Computer Vision) - Weeks 9-14: Part III (Language Models) - Week 15: Final project with TinyGPT **Intensive Bootcamp (6 weeks)** - Week 1: Part I (Foundations) - Fast pace - Weeks 2-3: Part II (Computer Vision) - Weeks 4-5: Part III (Language Models) - Week 6: Capstone project ### Assessment Milestones 1. **Part I Assessment**: Working XORNet (25% of grade) 2. **Part II Assessment**: CIFAR-10 >50% accuracy (35% of grade) 3. **Part III Assessment**: Working TinyGPT (40% of grade) Each part has clear, measurable success criteria!