mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-05 01:12:58 -05:00
2f23f757e7
This commit implements the pedagogically optimal "inevitable discovery" module progression based on expert validation and educational design principles. ## Module Reordering Summary **Previous Order (Problems)**: - 05_losses → 06_autograd → 07_dataloader → 08_optimizers → 09_spatial → 10_training - Issues: Autograd before optimizers, DataLoader before training, scattered dependencies **New Order (Beautiful Progression)**: - 05_losses → 06_optimizers → 07_autograd → 08_training → 09_spatial → 10_dataloader - Benefits: Each module creates inevitable need for the next ## Pedagogical Flow Achieved **05_losses** → "Need systematic weight updates" → **06_optimizers** **06_optimizers** → "Need automatic gradients" → **07_autograd** **07_autograd** → "Need systematic training" → **08_training** **08_training** → "MLPs hit limits on images" → **09_spatial** **09_spatial** → "Training is too slow" → **10_dataloader** ## Technical Changes ### Module Directory Renaming - `06_autograd` → `07_autograd` - `07_dataloader` → `10_dataloader` - `08_optimizers` → `06_optimizers` - `10_training` → `08_training` - `09_spatial` → `09_spatial` (no change) ### System Integration Updates - **MODULE_TO_CHECKPOINT mapping**: Updated in tito/commands/export.py - **Test directories**: Renamed module_XX directories to match new numbers - **Documentation**: Updated all references in MD files and agent configurations - **CLI integration**: Updated next-steps suggestions for proper flow ### Agent Configuration Updates - **Quality Assurance**: Updated module audit status with new numbers - **Module Developer**: Updated work tracking with new sequence - **Documentation**: Updated MASTER_PLAN_OF_RECORD.md with beautiful progression ## Educational Benefits 1. **Inevitable Discovery**: Each module naturally leads to the next 2. **Cognitive Load**: Concepts introduced exactly when needed 3. **Motivation**: Students understand WHY each tool is necessary 4. **Synthesis**: Everything flows toward complete ML systems understanding 5. **Professional Alignment**: Matches real ML engineering workflows ## Quality Assurance - ✅ All CLI commands still function - ✅ Checkpoint system mappings updated - ✅ Documentation consistency maintained - ✅ Test directory structure aligned - ✅ Agent configurations synchronized **Impact**: This reordering transforms TinyTorch from a collection of modules into a coherent educational journey where each step naturally motivates the next, creating optimal conditions for deep learning systems understanding.
230 lines
7.4 KiB
Markdown
230 lines
7.4 KiB
Markdown
# TinyTorch Capability Progression System
|
|
|
|
## How TinyTorch Unlocks Your AI Powers
|
|
|
|
TinyTorch follows a unique progression system where each module you complete unlocks new capabilities. As you build the framework, you're simultaneously unlocking the ability to recreate historical AI breakthroughs.
|
|
|
|
## The Learning Flow
|
|
|
|
```
|
|
Write Module → Pass Unit Tests → Run Integration Tests → Unlock Capability → Run Historical Example
|
|
```
|
|
|
|
### For Each Module:
|
|
1. **Build**: Implement the module components
|
|
2. **Test**: Pass all unit tests within the module
|
|
3. **Complete**: Run `tito module complete XX_modulename`
|
|
4. **Integration**: Automatic integration tests verify module works with others
|
|
5. **Unlock**: New capability achieved - run the corresponding historical example!
|
|
|
|
## Capability Unlock Timeline
|
|
|
|
### 🔓 Capability 0: Environment Setup (Module 1)
|
|
**Unlocked**: Development environment configured
|
|
```bash
|
|
tito module complete 01_setup
|
|
✅ Integration tests: Environment validation
|
|
🎯 Achievement: Ready to build AI history!
|
|
```
|
|
|
|
### 🔓 Capability 1: Data Structures (Module 2)
|
|
**Unlocked**: Can create and manipulate tensors
|
|
```bash
|
|
tito module complete 02_tensor
|
|
✅ Integration tests: Tensor operations, shape broadcasting
|
|
🎯 Achievement: Foundation for all neural computation
|
|
```
|
|
|
|
### 🔓 Capability 2: Nonlinearity (Module 3)
|
|
**Unlocked**: Can add intelligence through activation functions
|
|
```bash
|
|
tito module complete 03_activations
|
|
✅ Integration tests: Activation + Tensor compatibility
|
|
🎯 Achievement: Networks can learn non-linear patterns
|
|
```
|
|
|
|
### 🔓 Capability 3: Network Building (Module 4)
|
|
**Unlocked**: Can construct neural network architectures
|
|
```bash
|
|
tito module complete 04_layers
|
|
✅ Integration tests: Layer stacking, parameter management
|
|
🎯 Achievement: Build Rosenblatt's Perceptron (1957)!
|
|
➡️ RUN: python examples/perceptron_1957/rosenblatt_perceptron.py
|
|
```
|
|
|
|
### 🔓 Capability 4: Loss Functions (Module 5)
|
|
**Unlocked**: Can measure network performance
|
|
```bash
|
|
tito module complete 05_losses
|
|
✅ Integration tests: Loss + Tensor + Layer compatibility
|
|
🎯 Achievement: Can evaluate model predictions
|
|
```
|
|
|
|
### 🔓 Capability 5: Optimization (Module 6)
|
|
**Unlocked**: Advanced training algorithms (SGD, Adam)
|
|
```bash
|
|
tito module complete 06_optimizers
|
|
✅ Integration tests: Optimizer algorithms ready
|
|
🎯 Achievement: Systematic weight updates prepared
|
|
```
|
|
|
|
### 🔓 Capability 6: Automatic Differentiation (Module 7)
|
|
**Unlocked**: Networks can learn through backpropagation
|
|
```bash
|
|
tito module complete 07_autograd
|
|
✅ Integration tests: Gradient flow through layers
|
|
🎯 Achievement: Solve the XOR Problem (1969)!
|
|
➡️ RUN: python examples/xor_1969/minsky_xor_problem.py
|
|
```
|
|
|
|
### 🔓 Capability 7: Complete Training (Module 8)
|
|
**Unlocked**: Full training pipelines with validation
|
|
```bash
|
|
tito module complete 08_training
|
|
✅ Integration tests: Complete training loop
|
|
🎯 Achievement: Train networks end-to-end
|
|
➡️ RUN: python examples/xor_1969/minsky_xor_problem.py --train
|
|
```
|
|
|
|
### 🔓 Capability 8: Spatial Processing (Module 9)
|
|
**Unlocked**: Convolutional networks for vision
|
|
```bash
|
|
tito module complete 09_spatial
|
|
✅ Integration tests: Conv2D + Pooling + Tensor shapes
|
|
🎯 Achievement: Build LeNet (1998)!
|
|
➡️ RUN: python examples/lenet_1998/train_mnist.py
|
|
```
|
|
|
|
### 🔓 Capability 9: Data Loading (Module 10)
|
|
**Unlocked**: Can handle real datasets efficiently
|
|
```bash
|
|
tito module complete 10_dataloader
|
|
✅ Integration tests: Batching, shuffling, iteration
|
|
🎯 Achievement: Train AlexNet-scale networks (2012)!
|
|
➡️ RUN: python examples/alexnet_2012/train_cnn.py
|
|
```
|
|
|
|
### 🔓 Capability 10: Text Processing (Module 11)
|
|
**Unlocked**: Tokenization for NLP
|
|
```bash
|
|
tito module complete 11_tokenization
|
|
✅ Integration tests: Tokenizer + Embeddings
|
|
🎯 Achievement: Process text data
|
|
```
|
|
|
|
### 🔓 Capability 11: Embeddings (Module 12)
|
|
**Unlocked**: Dense representations of discrete tokens
|
|
```bash
|
|
tito module complete 12_embeddings
|
|
✅ Integration tests: Embedding + Tensor operations
|
|
🎯 Achievement: Word vectors and position encoding
|
|
```
|
|
|
|
### 🔓 Capability 12: Attention (Module 13)
|
|
**Unlocked**: Self-attention mechanisms
|
|
```bash
|
|
tito module complete 13_attention
|
|
✅ Integration tests: Attention + Layer compatibility
|
|
🎯 Achievement: Core transformer component ready
|
|
```
|
|
|
|
### 🔓 Capability 13: Transformers (Module 14)
|
|
**Unlocked**: Complete transformer architecture
|
|
```bash
|
|
tito module complete 14_transformers
|
|
✅ Integration tests: Full transformer stack
|
|
🎯 Achievement: Build GPT (2018)!
|
|
➡️ RUN: python examples/gpt_2018/simple_tinygpt.py
|
|
```
|
|
|
|
## Integration Test Categories
|
|
|
|
Each module completion triggers these integration tests:
|
|
|
|
### 1. **Import Tests**
|
|
- Module imports without errors
|
|
- All classes instantiate correctly
|
|
- No circular dependencies
|
|
|
|
### 2. **Compatibility Tests**
|
|
- Tensor shapes flow correctly through components
|
|
- Gradients propagate through all operations
|
|
- Memory is managed efficiently
|
|
|
|
### 3. **Integration Tests**
|
|
- Components work together (e.g., Layer + Activation + Loss)
|
|
- Forward and backward passes complete
|
|
- Training loops converge on simple problems
|
|
|
|
### 4. **Performance Tests**
|
|
- Operations complete in reasonable time
|
|
- Memory usage stays within bounds
|
|
- No memory leaks during training
|
|
|
|
## The Milestone System
|
|
|
|
When you complete certain modules, you unlock major milestones:
|
|
|
|
### 🏆 Milestone 1: "I Can Build Networks!" (After Module 4)
|
|
- Capability: Construct any feedforward architecture
|
|
- Historical Achievement: Rosenblatt's Perceptron (1957)
|
|
- What you built: Dense layers, activation functions, forward propagation
|
|
|
|
### 🏆 Milestone 2: "My Networks Can Learn!" (After Module 6)
|
|
- Capability: Train networks with backpropagation
|
|
- Historical Achievement: Solve XOR (1969/1986)
|
|
- What you built: Automatic differentiation, gradient computation
|
|
|
|
### 🏆 Milestone 3: "I Can Process Images!" (After Module 9)
|
|
- Capability: Build convolutional neural networks
|
|
- Historical Achievement: LeNet (1998)
|
|
- What you built: Conv2D, pooling, spatial operations
|
|
|
|
### 🏆 Milestone 4: "Production-Ready Training!" (After Module 10)
|
|
- Capability: Train deep networks on real datasets
|
|
- Historical Achievement: AlexNet (2012)
|
|
- What you built: Complete training pipelines, validation, metrics
|
|
|
|
### 🏆 Milestone 5: "I Built a Transformer!" (After Module 14)
|
|
- Capability: Modern NLP architectures
|
|
- Historical Achievement: GPT (2018)
|
|
- What you built: Attention, embeddings, layer normalization
|
|
|
|
## Seeing Your Progress
|
|
|
|
At any time, check your capabilities:
|
|
|
|
```bash
|
|
# See current capability level
|
|
tito status
|
|
|
|
# Run integration tests for a module
|
|
tito test integration 04_layers
|
|
|
|
# See which examples you can run
|
|
tito examples available
|
|
|
|
# Check milestone progress
|
|
tito milestones
|
|
```
|
|
|
|
## Why This System?
|
|
|
|
1. **Clear Progress**: You always know what you've achieved
|
|
2. **Motivation**: Each module unlocks something concrete
|
|
3. **Historical Context**: You're recreating AI history
|
|
4. **Quality Assurance**: Integration tests catch issues early
|
|
5. **Immediate Gratification**: Run real examples as you progress
|
|
|
|
## The Journey
|
|
|
|
```
|
|
Module 1-3: Foundation (tensors, activations)
|
|
Module 4: 🏆 Build networks → Perceptron works!
|
|
Module 5-6: 🏆 Learning → XOR problem solved!
|
|
Module 7-9: 🏆 Vision → LeNet recognizes digits!
|
|
Module 10: 🏆 Deep learning → AlexNet-scale training!
|
|
Module 11-14:🏆 Transformers → GPT generates text!
|
|
```
|
|
|
|
Each capability you unlock is permanent - once you've built it, it's yours forever! |