TinyTorch/docs/three-part-structure.md

TinyTorch Three-Part Learning Journey 🚀

Overview

TinyTorch is structured as a progressive three-part journey, where each part builds toward a concrete achievement. Students can complete any part and have built something meaningful!

Part I: Foundations (Modules 1-5)

"I can build neural networks from scratch!"

Modules

1. 01_setup - Development environment and tools
2. 02_tensor - Core data structure and operations
3. 03_activations - Non-linearity (the key to intelligence!)
4. 04_layers - Dense layers and matrix operations
5. 05_networks - Multi-layer neural networks

Capstone Achievement

✅ XORNet - Solve the classic XOR problem that proves you understand non-linear learning

What You Learn

• How tensors store and manipulate data
• Why activation functions enable intelligence
• How layers compose into networks
• Memory layouts and computational complexity
• Building blocks of all neural networks

---

Part II: Computer Vision (Modules 6-11)

"I can build CNNs that classify real images!"

Modules

6. 06_spatial - Convolutions and pooling for image processing
7. 07_dataloader - Efficient data pipelines and batching
8. 08_normalization - BatchNorm and LayerNorm for stable training
9. 09_autograd - Automatic differentiation and computational graphs
10. 10_optimizers - SGD, Adam, and gradient descent
11. 11_training - Complete training loops with validation

Capstone Achievement

✅ CIFAR-10 CNN - Classify real 32x32 color images at 55%+ accuracy (5.5x better than random!)

What You Learn

• How convolutions extract spatial features
• Data pipeline engineering for real datasets
• Why normalization prevents gradient problems
• How autograd enables learning
• Memory vs speed tradeoffs in optimization
• Production training techniques (checkpointing, early stopping)

---

Part III: Language Models (Modules 12-17)

"I can build transformers that generate text!"

Modules

12. 12_embeddings - Token embeddings and positional encoding
13. 13_attention - Multi-head attention mechanisms
14. 14_transformers - Transformer blocks and architectures
15. 15_generation - Autoregressive decoding and sampling strategies
16. 16_regularization - Dropout, weight decay, and robustness
17. 17_systems - Production deployment, optimization, and monitoring

Capstone Achievement

✅ TinyGPT - Generate coherent text character-by-character using transformers

What You Learn

• Why embeddings are often the largest parameters
• O(N²) attention complexity and memory bottlenecks
• How transformers process sequences in parallel
• Temperature, top-k, nucleus sampling tradeoffs
• Production ML systems engineering
• Deployment, monitoring, and optimization

---

Progressive Learning Path

graph LR
    A[Start] --> B[Part I: Foundations]
    B --> C{Can build MLPs}
    C --> D[Part II: Vision]
    D --> E{Can build CNNs}
    E --> F[Part III: Language]
    F --> G{Can build GPT}
    
    C -.->|Exit Point 1| H[Industry Ready:<br/>ML Engineer]
    E -.->|Exit Point 2| I[Industry Ready:<br/>Computer Vision]
    G -.->|Exit Point 3| J[Industry Ready:<br/>LLM Engineer]

Natural Exit Points

After Part I (Modules 1-5)

• You've built: Neural networks from scratch
• You understand: Core ML building blocks
• Industry relevance: Ready for ML engineering roles
• Concrete proof: Working XORNet

After Part II (Modules 6-11)

• You've built: Complete CNN training pipeline
• You understand: Real data processing at scale
• Industry relevance: Ready for computer vision roles
• Concrete proof: CIFAR-10 at 55% accuracy

After Part III (Modules 12-17)

• You've built: Transformer-based language model
• You understand: Modern LLM architectures
• Industry relevance: Ready for NLP/LLM engineering
• Concrete proof: Text-generating TinyGPT

---

Alignment with MLSysBook.ai

This structure perfectly complements the ML Systems textbook:

Book Section	TinyTorch Part	What You Build
Ch 1-4: Foundations	Part I (Modules 1-5)	Neural Networks
Ch 5-8: Design Principles	Part II (Modules 6-11)	CNNs & Training
Ch 9-12: Performance	Part III (Modules 12-17)	Transformers
Ch 13-20: Production	Integrated Throughout	Real Systems

Why This Structure Works

1. Clear Progression: Each part builds on the previous
2. Concrete Achievements: XOR → CIFAR-10 → GPT
3. Industry Aligned: MLP → CNN → Transformer mirrors ML history
4. Flexible Duration: Complete 1, 2, or all 3 parts based on course length
5. Systems Focus: Every module teaches ML systems engineering, not just algorithms

---

Module Dependency Graph

Part I (Foundations)
├── 01_setup
├── 02_tensor ← Foundation for everything
├── 03_activations ← Requires tensor
├── 04_layers ← Requires tensor, activations  
└── 05_networks ← Requires layers

Part II (Computer Vision)
├── 06_spatial ← Requires tensor, layers
├── 07_dataloader ← Requires tensor
├── 08_normalization ← Requires tensor, layers
├── 09_autograd ← Requires tensor, networks
├── 10_optimizers ← Requires autograd
└── 11_training ← Requires all above

Part III (Language Models)
├── 12_embeddings ← Requires tensor, layers
├── 13_attention ← Requires tensor, layers
├── 14_transformers ← Requires attention, normalization
├── 15_generation ← Requires transformers
├── 16_regularization ← Enhancement for all models
└── 17_systems ← Production engineering

---

For Instructors

Semester Planning Options

Quarter System (10 weeks)

• Weeks 1-4: Part I (Foundations)
• Weeks 5-9: Part II (Computer Vision)
• Week 10: Final project with XORNet or CIFAR-10

Semester System (15 weeks)

• Weeks 1-3: Part I (Foundations)
• Weeks 4-8: Part II (Computer Vision)
• Weeks 9-14: Part III (Language Models)
• Week 15: Final project with TinyGPT

Intensive Bootcamp (6 weeks)

• Week 1: Part I (Foundations) - Fast pace
• Weeks 2-3: Part II (Computer Vision)
• Weeks 4-5: Part III (Language Models)
• Week 6: Capstone project

Assessment Milestones

1. Part I Assessment: Working XORNet (25% of grade)
2. Part II Assessment: CIFAR-10 >50% accuracy (35% of grade)
3. Part III Assessment: Working TinyGPT (40% of grade)

Each part has clear, measurable success criteria!