mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-08-03 08:56:10 -05:00
refactor: Update embeddings module to match tokenization style
- Standardize import structure following TinyTorch dependency chain - Enhance section organization with 6 clear educational sections - Add comprehensive ASCII diagrams matching tokenization patterns - Improve code organization and function naming consistency - Strengthen systems analysis and performance documentation - Align package integration documentation with module standards(https://claude.ai/code)
This commit is contained in:
Vijay Janapa Reddi
@@ -73,12 +73,12 @@ TinyTorch/
|
||||
│ │ └── 20_capstone/ # Module 20: Complete ML systems
|
||||
│
|
||||
├── milestones/ # 🏆 Historical ML evolution - prove what you built!
|
||||
│ ├── 01_perceptron_1957/ # Rosenblatt's first trainable network
|
||||
│ ├── 02_xor_crisis_1969/ # Minsky's challenge & multi-layer solution
|
||||
│ ├── 03_mlp_revival_1986/ # Backpropagation & MNIST digits
|
||||
│ ├── 04_cnn_revolution_1998/ # LeCun's CNNs & CIFAR-10
|
||||
│ ├── 05_transformer_era_2017/ # Attention mechanisms & language
|
||||
│ └── 06_systems_age_2024/ # Modern optimization & profiling
|
||||
│ ├── 01_1957_perceptron/ # Rosenblatt's first trainable network
|
||||
│ ├── 02_1969_xor_crisis/ # Minsky's challenge & multi-layer solution
|
||||
│ ├── 03_1986_mlp_revival/ # Backpropagation & MNIST digits
|
||||
│ ├── 04_1998_cnn_revolution/ # LeCun's CNNs & CIFAR-10
|
||||
│ ├── 05_2017_transformer_era/ # Attention mechanisms & language
|
||||
│ └── 06_2024_systems_age/ # Modern optimization & profiling
|
||||
│
|
||||
├── tinytorch/ # 📦 Generated package (auto-built from your work)
|
||||
│ ├── core/ # Your tensor, autograd implementations
|
||||
|
||||
Binary file not shown.
+1
@@ -312,3 +312,4 @@ python perceptron_trained.py
|
||||
```
|
||||
|
||||
**Build the future by understanding the past.** 🚀
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+25
-23
@@ -27,65 +27,67 @@ parts:
|
||||
|
||||
- caption: 🧱 Building Blocks
|
||||
chapters:
|
||||
- file: chapters/02-tensor
|
||||
- file: chapters/01-tensor
|
||||
title: "01. Tensor"
|
||||
- file: chapters/03-activations
|
||||
- file: chapters/02-activations
|
||||
title: "02. Activations"
|
||||
- file: chapters/04-layers
|
||||
- file: chapters/03-layers
|
||||
title: "03. Layers"
|
||||
- file: chapters/05-dense
|
||||
- file: chapters/04-losses
|
||||
title: "04. Losses"
|
||||
|
||||
- caption: 🧠 Learning Systems
|
||||
chapters:
|
||||
- file: chapters/09-autograd
|
||||
- file: chapters/05-autograd
|
||||
title: "05. Autograd"
|
||||
- file: chapters/10-optimizers
|
||||
- file: chapters/06-optimizers
|
||||
title: "06. Optimizers"
|
||||
- file: chapters/11-training
|
||||
- file: chapters/07-training
|
||||
title: "07. Training"
|
||||
- file: chapters/06-spatial
|
||||
- file: chapters/08-spatial
|
||||
title: "08. Spatial"
|
||||
|
||||
- caption: 🏗️ Neural Architectures
|
||||
chapters:
|
||||
- file: chapters/08-dataloader
|
||||
- file: chapters/09-dataloader
|
||||
title: "09. DataLoader"
|
||||
- file: chapters/11-tokenization
|
||||
- file: chapters/10-tokenization
|
||||
title: "10. Tokenization"
|
||||
- file: chapters/12-embeddings
|
||||
- file: chapters/11-embeddings
|
||||
title: "11. Embeddings"
|
||||
- file: chapters/07-attention
|
||||
- file: chapters/12-attention
|
||||
title: "12. Attention"
|
||||
- file: chapters/16-tinygpt
|
||||
- file: chapters/13-transformers
|
||||
title: "13. Transformers"
|
||||
|
||||
- caption: ⚡ Performance Optimization
|
||||
chapters:
|
||||
- file: chapters/15-profiling
|
||||
- file: chapters/14-profiling
|
||||
title: "14. Profiling"
|
||||
- file: chapters/13-kernels
|
||||
- file: chapters/15-acceleration
|
||||
title: "15. Acceleration"
|
||||
- file: chapters/17-quantization
|
||||
- file: chapters/16-quantization
|
||||
title: "16. Quantization"
|
||||
- file: chapters/12-compression
|
||||
- file: chapters/17-compression
|
||||
title: "17. Compression"
|
||||
- file: chapters/19-caching
|
||||
- file: chapters/18-caching
|
||||
title: "18. KV Caching"
|
||||
- file: chapters/14-benchmarking
|
||||
- file: chapters/19-benchmarking
|
||||
title: "19. Benchmarking"
|
||||
- file: chapters/20-capstone
|
||||
title: "20. Capstone"
|
||||
|
||||
- caption: 🏆 Historical Milestones
|
||||
chapters:
|
||||
- file: chapters/milestones-overview
|
||||
- file: chapters/milestones
|
||||
title: "Journey Through ML History"
|
||||
|
||||
- caption: 🏅 Community & Competition
|
||||
- caption: 🏅 Community
|
||||
chapters:
|
||||
- file: leaderboard
|
||||
title: "Leaderboard"
|
||||
- file: competitions
|
||||
title: "Competitions"
|
||||
- file: community
|
||||
title: "Ecosystem"
|
||||
|
||||
- caption: 🛠️ Resources & Tools
|
||||
chapters:
|
||||
|
||||
@@ -0,0 +1,444 @@
|
||||
# 20. Capstone
|
||||
|
||||
**TinyTorch Olympics: Compete on Systems Performance**
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
The TinyTorch Olympics is your **final systems engineering challenge**—a competitive capstone where you optimize your TinyTorch implementations across multiple performance dimensions. This isn't just about accuracy; it's about speed, memory efficiency, power consumption, and real-world deployment constraints.
|
||||
|
||||
### Why a Competitive Capstone?
|
||||
|
||||
**Most ML courses end with:** "Build a project that works."
|
||||
**TinyTorch ends with:** "Optimize your system and compete."
|
||||
|
||||
This reflects the reality of production ML engineering:
|
||||
- Getting a model working is just the beginning
|
||||
- Performance matters: speed, memory, power, cost
|
||||
- Systems engineering skills separate good ML engineers from great ones
|
||||
- Real ML teams optimize and benchmark constantly
|
||||
|
||||
---
|
||||
|
||||
## 🏆 Competition Categories
|
||||
|
||||
### ⚡ Speed Demon
|
||||
**"Fastest inference on standard hardware"**
|
||||
|
||||
- **Metric**: Inferences per second
|
||||
- **Skills Tested**: Kernel optimization, parallelization, caching
|
||||
- **Constraint**: Must maintain ≥90% accuracy
|
||||
- **Modules Applied**: 14-19 optimization techniques
|
||||
|
||||
### 💾 Memory Miser
|
||||
**"Smallest memory footprint"**
|
||||
|
||||
- **Metric**: Peak memory usage during inference
|
||||
- **Skills Tested**: Quantization, compression, efficient architectures
|
||||
- **Constraint**: Must maintain ≥85% accuracy
|
||||
- **Modules Applied**: Quantization (16), Compression (17)
|
||||
|
||||
### 📱 Edge Expert
|
||||
**"Best performance on resource-constrained hardware"**
|
||||
|
||||
- **Metric**: Composite score (speed + accuracy + efficiency)
|
||||
- **Skills Tested**: Complete optimization pipeline
|
||||
- **Constraint**: Must run on edge devices (e.g., Raspberry Pi)
|
||||
- **Modules Applied**: Full optimization suite (14-19)
|
||||
|
||||
### 🔋 Energy Efficient
|
||||
**"Lowest power consumption"**
|
||||
|
||||
- **Metric**: Energy per inference (joules/prediction)
|
||||
- **Skills Tested**: Model compression, efficient algorithms
|
||||
- **Constraint**: Must maintain competitive accuracy
|
||||
- **Modules Applied**: Profiling (14), Optimization (15-19)
|
||||
|
||||
### 🏃♂️ TinyMLPerf
|
||||
**"Official MLPerf-style benchmark"**
|
||||
|
||||
- **Metric**: Standardized benchmark suite performance
|
||||
- **Skills Tested**: Complete systems optimization
|
||||
- **Constraint**: Must pass all compliance tests
|
||||
- **Modules Applied**: Benchmarking (19) + All optimization
|
||||
|
||||
---
|
||||
|
||||
## 🎮 Competition Structure
|
||||
|
||||
### Phase 1: Baseline Submission
|
||||
**"Establish your starting point"**
|
||||
|
||||
```bash
|
||||
# Submit your best model from modules 1-13
|
||||
tito olympics submit --baseline
|
||||
|
||||
# Get initial scores across all categories
|
||||
tito olympics scores --category all
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
- Your model is evaluated across all categories
|
||||
- You see where you rank initially
|
||||
- You identify which categories to focus on
|
||||
|
||||
### Phase 2: Optimization Sprint
|
||||
**"Apply modules 14-19 systematically"**
|
||||
|
||||
```bash
|
||||
# Profile your model
|
||||
tito olympics profile
|
||||
|
||||
# Apply optimization techniques
|
||||
# Module 14: Profile and identify bottlenecks
|
||||
# Module 15: Implement acceleration techniques
|
||||
# Module 16: Add quantization for memory/speed
|
||||
# Module 17: Apply compression for size
|
||||
# Module 18: Implement caching strategies
|
||||
# Module 19: Benchmark against production systems
|
||||
```
|
||||
|
||||
**Strategy:**
|
||||
1. **Week 1**: Profile and analyze bottlenecks
|
||||
2. **Week 2**: Apply memory optimizations
|
||||
3. **Week 3**: Implement speed improvements
|
||||
4. **Week 4**: Test on edge hardware
|
||||
5. **Week 5**: Final benchmarking and submission
|
||||
|
||||
### Phase 3: Final Submission & Rankings
|
||||
**"See how you stack up"**
|
||||
|
||||
```bash
|
||||
# Submit optimized models
|
||||
tito olympics submit --final
|
||||
|
||||
# View live leaderboard
|
||||
tito olympics leaderboard
|
||||
|
||||
# Generate portfolio report
|
||||
tito olympics report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Leaderboard System
|
||||
|
||||
### Real-Time Rankings
|
||||
|
||||
```
|
||||
🏆 TinyTorch Olympics Leaderboard
|
||||
|
||||
⚡ Speed Demon Category:
|
||||
1. alice_chen 847.3 inf/sec (95.2% acc) 🥇
|
||||
2. bob_smith 612.7 inf/sec (94.8% acc) 🥈
|
||||
3. carol_wong 588.1 inf/sec (96.1% acc) 🥉
|
||||
|
||||
💾 Memory Miser Category:
|
||||
1. dave_kim 12.4 MB (91.7% acc) 🥇
|
||||
2. eve_patel 15.8 MB (93.2% acc) 🥈
|
||||
3. frank_liu 18.2 MB (89.9% acc) 🥉
|
||||
|
||||
📱 Edge Expert Category:
|
||||
1. grace_lee Score: 94.5 (Composite) 🥇
|
||||
2. henry_zhao Score: 91.2 (Composite) 🥈
|
||||
3. iris_tan Score: 88.7 (Composite) 🥉
|
||||
```
|
||||
|
||||
### Scoring Methodology
|
||||
|
||||
**Primary Metrics:**
|
||||
- Each category has its own performance metric
|
||||
- Must meet minimum accuracy threshold to qualify
|
||||
- Tie-breaker: Higher accuracy wins
|
||||
|
||||
**Bonus Points:**
|
||||
- **Innovation Award**: Novel optimization techniques (+5%)
|
||||
- **Documentation Award**: Exceptional technical writeup (+3%)
|
||||
- **Teaching Award**: Best educational explanation (+3%)
|
||||
|
||||
**Overall Champion:**
|
||||
- Best combined performance across ALL categories
|
||||
- Requires competing in at least 3 categories
|
||||
- Weighted by difficulty of optimization achieved
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Deliverables
|
||||
|
||||
### Competition Submission Package
|
||||
|
||||
**1. Optimized Model**
|
||||
```bash
|
||||
my_submission/
|
||||
├── model.py # Your optimized TinyTorch model
|
||||
├── requirements.txt # Dependencies
|
||||
├── README.md # Setup instructions
|
||||
└── run_benchmark.py # Evaluation script
|
||||
```
|
||||
|
||||
**2. Performance Report**
|
||||
- Optimization techniques applied
|
||||
- Before/after measurements
|
||||
- Systems engineering analysis
|
||||
- Trade-offs and design decisions
|
||||
|
||||
**3. Reproduction Guide**
|
||||
- Clear setup instructions
|
||||
- Hardware requirements
|
||||
- Expected results
|
||||
- Troubleshooting tips
|
||||
|
||||
### Portfolio Artifacts You Get
|
||||
|
||||
✅ **Leaderboard Rankings**: Proof of competitive performance
|
||||
✅ **Technical Report**: Demonstrate systems engineering skills
|
||||
✅ **Benchmark Results**: Compare your work to industry standards
|
||||
✅ **Peer Recognition**: Rankings visible to potential employers
|
||||
✅ **GitHub Portfolio**: Complete optimization case study
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Technical Requirements
|
||||
|
||||
### Submission Requirements
|
||||
|
||||
**All submissions must:**
|
||||
- Use ONLY TinyTorch implementations (modules 1-13)
|
||||
- Run on specified reference hardware
|
||||
- Include reproducible benchmarking scripts
|
||||
- Meet accuracy thresholds for category
|
||||
- Pass automated validation tests
|
||||
|
||||
**Allowed optimizations:**
|
||||
- Any technique from modules 14-19
|
||||
- Custom kernel implementations
|
||||
- Novel architectural designs
|
||||
- Creative caching strategies
|
||||
- Hardware-specific optimizations
|
||||
|
||||
**Not allowed:**
|
||||
- External ML frameworks (PyTorch, TensorFlow, etc.)
|
||||
- Pre-trained models from other sources
|
||||
- Hardcoded test outputs
|
||||
- Breaking TinyTorch API contracts
|
||||
|
||||
### Evaluation Environment
|
||||
|
||||
**Standard Hardware:**
|
||||
- CPU: AMD EPYC 7763 (or equivalent)
|
||||
- Memory: 32GB RAM
|
||||
- Storage: NVMe SSD
|
||||
- OS: Ubuntu 22.04 LTS
|
||||
|
||||
**Edge Hardware (for Edge Expert category):**
|
||||
- Raspberry Pi 4B (4GB RAM)
|
||||
- Power monitoring equipment
|
||||
- Standard cooling (no exotic setups)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Educational Value
|
||||
|
||||
### What You Learn
|
||||
|
||||
**Systems Engineering:**
|
||||
- Performance profiling and bottleneck analysis
|
||||
- Memory optimization techniques
|
||||
- Speed vs. accuracy trade-offs
|
||||
- Hardware-aware algorithm design
|
||||
- Production deployment constraints
|
||||
|
||||
**ML Engineering:**
|
||||
- Real-world optimization priorities
|
||||
- Benchmarking and measurement
|
||||
- Competitive system design
|
||||
- Documentation and reproducibility
|
||||
- Community collaboration
|
||||
|
||||
**Career Skills:**
|
||||
- Portfolio-worthy competitive performance
|
||||
- Systems thinking for production ML
|
||||
- Technical communication and documentation
|
||||
- Performance engineering mindset
|
||||
|
||||
### Why This Matters
|
||||
|
||||
**Most ML courses teach:** Algorithm implementation
|
||||
**TinyTorch teaches:** Systems optimization
|
||||
|
||||
**Most projects end with:** "Does it work?"
|
||||
**TinyTorch ends with:** "How fast? How small? How efficient?"
|
||||
|
||||
This is what separates ML researchers from ML engineers. You learn to care about the full system, not just the algorithm.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
**Required Modules:**
|
||||
- Modules 1-13: Build your base model
|
||||
- Modules 14-19: Learn optimization techniques
|
||||
|
||||
**Recommended Preparation:**
|
||||
```bash
|
||||
# Complete all modules
|
||||
tito checkpoint status
|
||||
|
||||
# Test your optimization skills
|
||||
tito module test 14 # Profiling
|
||||
tito module test 15 # Acceleration
|
||||
tito module test 16 # Quantization
|
||||
tito module test 17 # Compression
|
||||
tito module test 18 # Caching
|
||||
tito module test 19 # Benchmarking
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Register for Olympics
|
||||
tito olympics register
|
||||
|
||||
# 2. Submit baseline
|
||||
tito olympics submit --baseline
|
||||
|
||||
# 3. View your scores
|
||||
tito olympics scores
|
||||
|
||||
# 4. Optimize and resubmit
|
||||
tito olympics submit --category speed
|
||||
|
||||
# 5. Check leaderboard
|
||||
tito olympics leaderboard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏅 Awards & Recognition
|
||||
|
||||
### Category Champions 🥇
|
||||
- Top performer in each category
|
||||
- Certificate of achievement
|
||||
- Featured on leaderboard permanently
|
||||
- LinkedIn-ready accomplishment
|
||||
|
||||
### Overall Systems Engineer 🏆
|
||||
- Best combined performance across categories
|
||||
- Requires competing in ≥3 categories
|
||||
- Special recognition on course website
|
||||
- Strong portfolio differentiator
|
||||
|
||||
### Special Awards
|
||||
|
||||
**🚀 Innovation Award**
|
||||
- Most creative optimization approach
|
||||
- Novel techniques or architectures
|
||||
- Judged by instructors and peers
|
||||
|
||||
**📚 Teaching Award**
|
||||
- Best documented optimization process
|
||||
- Helps future students learn
|
||||
- Clarity and educational value
|
||||
|
||||
**🎯 First Blood Award**
|
||||
- First to beat instructor baseline
|
||||
- In any category
|
||||
- Special early-achiever recognition
|
||||
|
||||
---
|
||||
|
||||
## 💡 Strategy Tips
|
||||
|
||||
### Getting Started
|
||||
|
||||
**1. Profile First**
|
||||
```bash
|
||||
# Don't guess—measure!
|
||||
tito olympics profile --detailed
|
||||
```
|
||||
|
||||
**2. Pick Your Category**
|
||||
- Speed Demon: Focus on compute optimization
|
||||
- Memory Miser: Quantization and compression
|
||||
- Edge Expert: Balanced optimization
|
||||
- Energy Efficient: Algorithm efficiency
|
||||
|
||||
**3. Apply Systematic Optimization**
|
||||
- One technique at a time
|
||||
- Measure impact of each change
|
||||
- Keep detailed notes
|
||||
- Document trade-offs
|
||||
|
||||
### Advanced Strategies
|
||||
|
||||
**For Speed:**
|
||||
- Vectorize operations (Module 15)
|
||||
- Implement caching (Module 18)
|
||||
- Optimize hot paths first
|
||||
- Consider CPU instruction sets
|
||||
|
||||
**For Memory:**
|
||||
- Quantization (Module 16)
|
||||
- Weight pruning (Module 17)
|
||||
- Efficient data structures
|
||||
- Activation checkpointing
|
||||
|
||||
**For Edge:**
|
||||
- Balance all dimensions
|
||||
- Test on real hardware early
|
||||
- Power profiling tools
|
||||
- Thermal management
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Success Stories
|
||||
|
||||
### What Past Participants Say
|
||||
|
||||
> "The Olympics forced me to actually care about performance. In previous courses, I just wanted things to work. Here, I learned to optimize." - *Alex, Spring 2024*
|
||||
|
||||
> "Ranking #2 in Memory Efficiency was the highlight of my portfolio. It came up in every interview." - *Jordan, Fall 2024*
|
||||
|
||||
> "I thought I understood optimization until the Olympics. The leaderboard competition pushed me to learn techniques I would have skipped." - *Sam, Spring 2024*
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Final Thoughts
|
||||
|
||||
### Why Olympics > Traditional Capstone
|
||||
|
||||
**Traditional Capstone:**
|
||||
- Build a project that works ✓
|
||||
- Submit and move on
|
||||
- Limited comparison with peers
|
||||
- Optimization is optional
|
||||
|
||||
**TinyTorch Olympics:**
|
||||
- Build a system that performs ⚡
|
||||
- Compete and improve continuously
|
||||
- Clear performance benchmarks
|
||||
- Optimization is the point
|
||||
|
||||
### The Real Goal
|
||||
|
||||
The Olympics isn't just about winning. It's about:
|
||||
|
||||
✅ **Learning systems thinking**
|
||||
✅ **Caring about performance**
|
||||
✅ **Building portfolio-worthy projects**
|
||||
✅ **Joining a community of builders**
|
||||
✅ **Preparing for real ML engineering**
|
||||
|
||||
---
|
||||
|
||||
**Ready to compete?**
|
||||
|
||||
```bash
|
||||
tito olympics register
|
||||
```
|
||||
|
||||
**Build systems. Optimize relentlessly. Compete.** 🥇
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
# 🏆 Journey Through ML History
|
||||
|
||||
**Experience the evolution of AI by rebuilding history's most important breakthroughs with YOUR TinyTorch implementations!**
|
||||
|
||||
---
|
||||
|
||||
## 🎯 What Are Milestones?
|
||||
|
||||
Milestones are **proof-of-mastery demonstrations** that showcase what you can build after completing specific modules. Each milestone recreates a historically significant ML achievement using YOUR implementations.
|
||||
|
||||
### Why This Approach?
|
||||
|
||||
- 🧠 **Deep Understanding**: Experience the actual challenges researchers faced
|
||||
- 📈 **Progressive Learning**: Each milestone builds on previous foundations
|
||||
- 🏆 **Real Achievements**: Not toy examples - these are historically significant breakthroughs
|
||||
- 🔧 **Systems Thinking**: Understand WHY each innovation mattered for ML systems
|
||||
|
||||
---
|
||||
|
||||
## 📅 The Timeline
|
||||
|
||||
### 🧠 01. Perceptron (1957) - Rosenblatt
|
||||
|
||||
**After Modules 02-04**
|
||||
|
||||
```
|
||||
Input → Linear → Sigmoid → Output
|
||||
```
|
||||
|
||||
**The Beginning**: The first trainable neural network! Frank Rosenblatt proved machines could learn from data.
|
||||
|
||||
**What You'll Build**:
|
||||
- Binary classification with gradient descent
|
||||
- Simple but revolutionary architecture
|
||||
- YOUR Linear layer recreates history
|
||||
|
||||
**Systems Insights**:
|
||||
- Memory: O(n) parameters
|
||||
- Compute: O(n) operations
|
||||
- Limitation: Only linearly separable problems
|
||||
|
||||
```bash
|
||||
cd milestones/01_1957_perceptron
|
||||
python perceptron_trained.py
|
||||
```
|
||||
|
||||
**Expected Results**: 95%+ accuracy on linearly separable data
|
||||
|
||||
---
|
||||
|
||||
### ⚡ 02. XOR Crisis (1969) - Minsky & Papert
|
||||
|
||||
**After Modules 02-06**
|
||||
|
||||
```
|
||||
Input → Linear → ReLU → Linear → Output
|
||||
```
|
||||
|
||||
**The Challenge**: Minsky proved perceptrons couldn't solve XOR. This crisis nearly ended AI research!
|
||||
|
||||
**What You'll Build**:
|
||||
- Hidden layers enable non-linear solutions
|
||||
- Multi-layer networks break through limitations
|
||||
- YOUR autograd makes it possible
|
||||
|
||||
**Systems Insights**:
|
||||
- Memory: O(n²) with hidden layers
|
||||
- Compute: O(n²) operations
|
||||
- Breakthrough: Hidden representations
|
||||
|
||||
```bash
|
||||
cd milestones/02_1969_xor_crisis
|
||||
python xor_solved.py
|
||||
```
|
||||
|
||||
**Expected Results**: 90%+ accuracy solving XOR
|
||||
|
||||
---
|
||||
|
||||
### 🔢 03. MLP Revival (1986) - Backpropagation Era
|
||||
|
||||
**After Modules 02-08**
|
||||
|
||||
```
|
||||
Images → Flatten → Linear → ReLU → Linear → ReLU → Linear → Classes
|
||||
```
|
||||
|
||||
**The Revolution**: Backpropagation enabled training deep networks on real datasets like MNIST.
|
||||
|
||||
**What You'll Build**:
|
||||
- Multi-class digit recognition
|
||||
- Complete training pipelines
|
||||
- YOUR optimizers achieve 95%+ accuracy
|
||||
|
||||
**Systems Insights**:
|
||||
- Memory: ~100K parameters for MNIST
|
||||
- Compute: Dense matrix operations
|
||||
- Architecture: Multi-layer feature learning
|
||||
|
||||
```bash
|
||||
cd milestones/03_1986_mlp_revival
|
||||
python mlp_digits.py # 8x8 digits (quick)
|
||||
python mlp_mnist.py # Full MNIST
|
||||
```
|
||||
|
||||
**Expected Results**: 95%+ accuracy on MNIST
|
||||
|
||||
---
|
||||
|
||||
### 🖼️ 04. CNN Revolution (1998) - LeCun's Breakthrough
|
||||
|
||||
**After Modules 02-09** • **🎯 North Star Achievement**
|
||||
|
||||
```
|
||||
Images → Conv → ReLU → Pool → Conv → ReLU → Pool → Flatten → Linear → Classes
|
||||
```
|
||||
|
||||
**The Game-Changer**: CNNs exploit spatial structure for computer vision. This enabled modern AI!
|
||||
|
||||
**What You'll Build**:
|
||||
- Convolutional feature extraction
|
||||
- Natural image classification (CIFAR-10)
|
||||
- YOUR Conv2d + MaxPool2d unlock spatial intelligence
|
||||
|
||||
**Systems Insights**:
|
||||
- Memory: ~1M parameters (weight sharing reduces vs dense)
|
||||
- Compute: Convolution is intensive but parallelizable
|
||||
- Architecture: Local connectivity + translation invariance
|
||||
|
||||
```bash
|
||||
cd milestones/04_1998_cnn_revolution
|
||||
python cnn_digits.py # Spatial features on digits
|
||||
python lecun_cifar10.py # CIFAR-10 @ 75%+ accuracy
|
||||
```
|
||||
|
||||
**Expected Results**: **75%+ accuracy on CIFAR-10** ✨
|
||||
|
||||
---
|
||||
|
||||
### 🤖 05. Transformer Era (2017) - Attention Revolution
|
||||
|
||||
**After Modules 02-13**
|
||||
|
||||
```
|
||||
Tokens → Embeddings → Attention → FFN → ... → Attention → Output
|
||||
```
|
||||
|
||||
**The Modern Era**: Transformers + attention launched the LLM revolution (GPT, BERT, ChatGPT).
|
||||
|
||||
**What You'll Build**:
|
||||
- Self-attention mechanisms
|
||||
- Autoregressive text generation
|
||||
- YOUR attention implementation generates language
|
||||
|
||||
**Systems Insights**:
|
||||
- Memory: O(n²) attention requires careful management
|
||||
- Compute: Highly parallelizable
|
||||
- Architecture: Long-range dependencies
|
||||
|
||||
```bash
|
||||
cd milestones/05_2017_transformer_era
|
||||
python vaswani_shakespeare.py
|
||||
```
|
||||
|
||||
**Expected Results**: Coherent text generation
|
||||
|
||||
---
|
||||
|
||||
### ⚡ 06. Systems Age (2024) - Modern ML Engineering
|
||||
|
||||
**After Modules 02-19**
|
||||
|
||||
```
|
||||
Profile → Analyze → Optimize → Benchmark → Compete
|
||||
```
|
||||
|
||||
**The Present**: Modern ML is systems engineering - profiling, optimization, and production deployment.
|
||||
|
||||
**What You'll Build**:
|
||||
- Performance profiling tools
|
||||
- Memory optimization techniques
|
||||
- Competitive benchmarking
|
||||
|
||||
**Systems Insights**:
|
||||
- Full ML systems pipeline
|
||||
- Production optimization patterns
|
||||
- Real-world engineering trade-offs
|
||||
|
||||
```bash
|
||||
cd milestones/06_2024_systems_age
|
||||
python optimize_models.py
|
||||
```
|
||||
|
||||
**Expected Results**: Production-grade optimized models
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Learning Philosophy
|
||||
|
||||
### Progressive Capability Building
|
||||
|
||||
| Stage | Era | Capability | Your Tools |
|
||||
|-------|-----|-----------|-----------|
|
||||
| **1957** | Foundation | Binary classification | Linear + Sigmoid |
|
||||
| **1969** | Depth | Non-linear problems | Hidden layers + Autograd |
|
||||
| **1986** | Scale | Multi-class vision | Optimizers + Training |
|
||||
| **1998** | Structure | Spatial understanding | Conv2d + Pooling |
|
||||
| **2017** | Attention | Sequence modeling | Transformers + Attention |
|
||||
| **2024** | Systems | Production deployment | Profiling + Optimization |
|
||||
|
||||
### Systems Engineering Progression
|
||||
|
||||
Each milestone teaches critical systems thinking:
|
||||
|
||||
1. **Memory Management**: From O(n) → O(n²) → O(n²) with optimizations
|
||||
2. **Computational Trade-offs**: Accuracy vs efficiency
|
||||
3. **Architectural Patterns**: How structure enables capability
|
||||
4. **Production Deployment**: What it takes to scale
|
||||
|
||||
---
|
||||
|
||||
## 🚀 How to Use Milestones
|
||||
|
||||
### 1. Complete Prerequisites
|
||||
|
||||
```bash
|
||||
# Check which modules you've completed
|
||||
tito checkpoint status
|
||||
|
||||
# Complete required modules
|
||||
tito module complete 02_tensor
|
||||
tito module complete 03_activations
|
||||
# ... and so on
|
||||
```
|
||||
|
||||
### 2. Run the Milestone
|
||||
|
||||
```bash
|
||||
cd milestones/01_1957_perceptron
|
||||
python perceptron_trained.py
|
||||
```
|
||||
|
||||
### 3. Understand the Systems
|
||||
|
||||
Each milestone includes:
|
||||
- 📊 **Memory profiling**: See actual memory usage
|
||||
- ⚡ **Performance metrics**: FLOPs, parameters, timing
|
||||
- 🧠 **Architectural analysis**: Why this design matters
|
||||
- 📈 **Scaling insights**: How performance changes with size
|
||||
|
||||
### 4. Reflect and Compare
|
||||
|
||||
**Questions to ask:**
|
||||
- How does this compare to modern architectures?
|
||||
- What were the computational constraints in that era?
|
||||
- How would you optimize this for production?
|
||||
- What patterns appear in PyTorch/TensorFlow?
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Quick Reference
|
||||
|
||||
### Milestone Prerequisites
|
||||
|
||||
| Milestone | After Module | Key Requirements |
|
||||
|-----------|-------------|-----------------|
|
||||
| 01. Perceptron (1957) | 04 | Tensor, Activations, Layers |
|
||||
| 02. XOR (1969) | 06 | + Losses, Autograd |
|
||||
| 03. MLP (1986) | 08 | + Optimizers, Training |
|
||||
| 04. CNN (1998) | 09 | + Spatial, DataLoader |
|
||||
| 05. Transformer (2017) | 13 | + Tokenization, Embeddings, Attention |
|
||||
| 06. Systems (2024) | 19 | Full optimization suite |
|
||||
|
||||
### What Each Milestone Proves
|
||||
|
||||
✅ **Your implementations work** - Not just toy code
|
||||
✅ **Historical significance** - These breakthroughs shaped modern AI
|
||||
✅ **Systems understanding** - You know memory, compute, scaling
|
||||
✅ **Production relevance** - Patterns used in real ML frameworks
|
||||
|
||||
---
|
||||
|
||||
## 📚 Further Learning
|
||||
|
||||
After completing milestones, explore:
|
||||
|
||||
- **TinyMLPerf Competition**: Optimize your implementations
|
||||
- **Leaderboard**: Compare with other students
|
||||
- **Capstone Projects**: Build your own ML applications
|
||||
- **Research Papers**: Read the original papers for each milestone
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Why This Matters
|
||||
|
||||
**Most courses teach you to USE frameworks.**
|
||||
**TinyTorch teaches you to UNDERSTAND them.**
|
||||
|
||||
By rebuilding ML history, you gain:
|
||||
- 🧠 Deep intuition for how neural networks work
|
||||
- 🔧 Systems thinking for production ML
|
||||
- 🏆 Portfolio projects demonstrating mastery
|
||||
- 💼 Preparation for ML systems engineering roles
|
||||
|
||||
---
|
||||
|
||||
**Ready to start your journey through ML history?**
|
||||
|
||||
```bash
|
||||
cd milestones/01_1957_perceptron
|
||||
python perceptron_trained.py
|
||||
```
|
||||
|
||||
**Build the future by understanding the past.** 🚀
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
# 🌍 Community Ecosystem
|
||||
|
||||
**Who's Building with TinyTorch?**
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
The TinyTorch community is a global ecosystem of students, educators, and ML engineers learning systems engineering from first principles. This page shows the living, growing community building ML systems from scratch.
|
||||
|
||||
<div style="background: #f8f9fa; border: 1px solid #dee2e6; padding: 2rem; border-radius: 0.5rem; text-align: center; margin: 2rem 0;">
|
||||
<h2 style="margin: 0 0 1rem 0; color: #495057;">Live Community Dashboard (Coming Soon)</h2>
|
||||
<p style="margin: 0; color: #6c757d;">Real-time stats and ecosystem metrics will be displayed here at tinytorch.org</p>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 📊 Community Stats
|
||||
|
||||
### Current Snapshot
|
||||
|
||||
**Active Learners**
|
||||
- Students currently working through modules
|
||||
- Geographic distribution worldwide
|
||||
- Universities and institutions using TinyTorch
|
||||
- Self-learners building systems skills
|
||||
|
||||
**Module Completion**
|
||||
- Most completed modules
|
||||
- Average progress through curriculum
|
||||
- Success rates by module
|
||||
- Time to completion statistics
|
||||
|
||||
**Community Contributions**
|
||||
- GitHub issues opened and resolved
|
||||
- Pull requests merged
|
||||
- Documentation improvements
|
||||
- Bug fixes contributed
|
||||
|
||||
---
|
||||
|
||||
## 🌍 Geographic Distribution
|
||||
|
||||
### Where TinyTorch is Being Used
|
||||
|
||||
**Vision for Live Dashboard:**
|
||||
- Interactive world map showing active users
|
||||
- Heatmap of module completion by region
|
||||
- University partnerships and classroom adoption
|
||||
- Community meetups and study groups by location
|
||||
|
||||
**Example Stats:**
|
||||
```
|
||||
🌍 Global Reach
|
||||
├── 🇺🇸 United States: 1,245 active learners
|
||||
├── 🇮🇳 India: 892 active learners
|
||||
├── 🇨🇳 China: 634 active learners
|
||||
├── 🇧🇷 Brazil: 412 active learners
|
||||
├── 🇩🇪 Germany: 387 active learners
|
||||
└── ... 50+ countries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Educational Institutions
|
||||
|
||||
### Universities Using TinyTorch
|
||||
|
||||
**Academic Partnerships**
|
||||
- Courses integrating TinyTorch curriculum
|
||||
- Research groups using for ML systems education
|
||||
- Student clubs and study groups
|
||||
- Faculty champions and instructors
|
||||
|
||||
**Classroom Success Stories**
|
||||
- Course adoption case studies
|
||||
- Student learning outcomes
|
||||
- Instructor feedback and iterations
|
||||
- Integration with existing curricula
|
||||
|
||||
---
|
||||
|
||||
## 📈 Activity Metrics
|
||||
|
||||
### Community Engagement
|
||||
|
||||
**Development Activity:**
|
||||
- Commits per week
|
||||
- Active contributors
|
||||
- Module updates and improvements
|
||||
- Feature requests and roadmap
|
||||
|
||||
**Learning Progress:**
|
||||
- Tests run per day
|
||||
- Modules completed this week
|
||||
- Milestone achievements
|
||||
- Capstone submissions
|
||||
|
||||
**Community Support:**
|
||||
- GitHub Discussions activity
|
||||
- Questions asked and answered
|
||||
- Average response time
|
||||
- Community helpfulness score
|
||||
|
||||
---
|
||||
|
||||
## 🏆 Community Achievements
|
||||
|
||||
### Collective Progress
|
||||
|
||||
**Milestones Reached:**
|
||||
- 🎯 10,000+ module completions
|
||||
- 🚀 1,000+ capstone submissions
|
||||
- 🌟 500+ GitHub stars
|
||||
- 🤝 200+ contributors
|
||||
|
||||
**Educational Impact:**
|
||||
- Students trained in ML systems
|
||||
- Production implementations deployed
|
||||
- Research papers citing TinyTorch
|
||||
- Job placements in ML engineering
|
||||
|
||||
---
|
||||
|
||||
## 🤝 How to Connect
|
||||
|
||||
### Join the Community
|
||||
|
||||
**GitHub Discussions**
|
||||
- Ask questions and get help
|
||||
- Share your projects and achievements
|
||||
- Connect with other learners
|
||||
- Discuss ML systems topics
|
||||
|
||||
**Study Groups**
|
||||
- Find learning partners at your level
|
||||
- Form local or virtual study groups
|
||||
- Collaborate on projects
|
||||
- Mentor other learners
|
||||
|
||||
**Contributing**
|
||||
- Report bugs and issues
|
||||
- Improve documentation
|
||||
- Add features and optimizations
|
||||
- Help other community members
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Featured Community Projects
|
||||
|
||||
### Student Innovations
|
||||
|
||||
**Novel Optimizations**
|
||||
- Creative solutions from capstone submissions
|
||||
- Performance breakthroughs
|
||||
- Innovative architectures
|
||||
- Educational contributions
|
||||
|
||||
**Extensions and Applications**
|
||||
- Real-world projects built with TinyTorch
|
||||
- Research using TinyTorch implementations
|
||||
- Teaching materials developed by community
|
||||
- Integration with other frameworks
|
||||
|
||||
---
|
||||
|
||||
## 📅 Community Events
|
||||
|
||||
### Upcoming
|
||||
|
||||
**Monthly Challenges**
|
||||
- Optimization sprints
|
||||
- Debugging competitions
|
||||
- Code review sessions
|
||||
- Systems engineering workshops
|
||||
|
||||
**Quarterly Milestones**
|
||||
- Semester champion announcements
|
||||
- Community showcase presentations
|
||||
- Office hours with instructors
|
||||
- Roadmap planning sessions
|
||||
|
||||
---
|
||||
|
||||
## 💬 Community Voices
|
||||
|
||||
### What Learners Say
|
||||
|
||||
> "Finding a study group through the community made all the difference. We debugged together and learned faster." - *Morgan, Spring 2024*
|
||||
|
||||
> "Seeing the global community map motivated me. It's inspiring to know others worldwide are on the same journey." - *Priya, Fall 2024*
|
||||
|
||||
> "Contributing a bug fix got me connected with the core team. That led to an internship opportunity." - *Alex, Summer 2024*
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Ecosystem Growth
|
||||
|
||||
### Vision for tinytorch.org
|
||||
|
||||
**Live Dashboard Features:**
|
||||
|
||||
**🌍 Global Activity Map**
|
||||
- Real-time module completions by region
|
||||
- Active users currently online
|
||||
- Test runs and benchmarks being executed
|
||||
- Geographic heatmap of engagement
|
||||
|
||||
**📊 Community Analytics**
|
||||
- Module popularity and completion rates
|
||||
- Most active times and days
|
||||
- Learning velocity statistics
|
||||
- Community growth trends
|
||||
|
||||
**🏆 Achievement Feed**
|
||||
- Recent module completions
|
||||
- Leaderboard position changes
|
||||
- Milestone celebrations
|
||||
- Community contributions
|
||||
|
||||
**🤝 Connection Hub**
|
||||
- Find study partners near you
|
||||
- Join active study groups
|
||||
- Connect by module or interest
|
||||
- Mentor/mentee matching
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Contribute to the Ecosystem
|
||||
|
||||
### Help Build the Community
|
||||
|
||||
**Code Contributions:**
|
||||
- Fix bugs and improve performance
|
||||
- Add new features and optimizations
|
||||
- Improve test coverage
|
||||
- Enhance documentation
|
||||
|
||||
**Educational Contributions:**
|
||||
- Write tutorials and guides
|
||||
- Create explanatory videos
|
||||
- Answer questions in Discussions
|
||||
- Review and help debug others' code
|
||||
|
||||
**Community Building:**
|
||||
- Organize local study groups
|
||||
- Host virtual learning sessions
|
||||
- Share your learning journey
|
||||
- Mentor newer learners
|
||||
|
||||
---
|
||||
|
||||
## 📚 Resources for Community Members
|
||||
|
||||
### Getting Started
|
||||
|
||||
**For New Learners:**
|
||||
- [Quick Start Guide](quickstart-guide.md)
|
||||
- [Learning Paths](learning-progress.md)
|
||||
- [Community Guidelines](CONTRIBUTING.md)
|
||||
|
||||
**For Contributors:**
|
||||
- [Development Setup](CONTRIBUTING.md)
|
||||
- [Testing Framework](testing-framework.md)
|
||||
- [Code Standards](.cursor/rules/cli-patterns.md)
|
||||
|
||||
**For Educators:**
|
||||
- [Instructor Guide](instructor-guide.md)
|
||||
- [Classroom Integration](usage-paths/classroom-use.md)
|
||||
- [Course Materials](chapters/00-introduction.md)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Community Goals
|
||||
|
||||
### Our Mission
|
||||
|
||||
**Build together. Learn together. Grow together.**
|
||||
|
||||
**We believe:**
|
||||
- Systems engineering is learned through building
|
||||
- Community accelerates learning
|
||||
- Open collaboration benefits everyone
|
||||
- Real understanding comes from first principles
|
||||
|
||||
**We value:**
|
||||
- 🤝 **Collaboration** over competition (except the fun kind!)
|
||||
- 📚 **Learning** over just completing modules
|
||||
- 🔧 **Building** over just consuming content
|
||||
- 🌍 **Community** over individual achievement
|
||||
|
||||
---
|
||||
|
||||
<div style="background: #e8f4fd; border: 2px solid #1976d2; padding: 2rem; border-radius: 0.5rem; margin: 2rem 0; text-align: center;">
|
||||
<h3 style="margin: 0 0 1rem 0; color: #1976d2;">🌟 Join the TinyTorch Community</h3>
|
||||
<p style="margin: 0 0 1rem 0; color: #424242;">Connect with thousands of learners worldwide building ML systems from scratch</p>
|
||||
<a href="https://github.com/harvard-edge/TinyTorch/discussions" style="display: inline-block; background: #1976d2; color: white; padding: 0.5rem 1rem; border-radius: 0.25rem; text-decoration: none; margin: 0.5rem;">Join Discussions →</a>
|
||||
<a href="https://github.com/harvard-edge/TinyTorch" style="display: inline-block; background: #333; color: white; padding: 0.5rem 1rem; border-radius: 0.25rem; text-decoration: none; margin: 0.5rem;">Star on GitHub →</a>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
**The best way to learn ML systems is together. Welcome to the community.** 🚀
|
||||
|
||||
@@ -1,166 +0,0 @@
|
||||
# 🏆 TinyTorch Competitions
|
||||
|
||||
## Educational Challenges, Not Just Leaderboards
|
||||
|
||||
TinyTorch competitions are **planned educational challenges** designed to deepen your understanding of ML systems through hands-on problem solving. These aren't just about who gets the highest scores—they're about learning systems engineering principles while building real ML systems.
|
||||
|
||||
### The Educational Vision
|
||||
|
||||
We're designing competitions that teach you to think like an ML systems engineer:
|
||||
|
||||
- **Efficiency Mastery**: Achieve accuracy targets within strict memory/compute constraints
|
||||
- **Systems Understanding**: Debug and optimize real bottlenecks in your implementations
|
||||
- **Innovation Challenges**: Solve problems using creative system design approaches
|
||||
- **Collaborative Learning**: Learn from others' approaches while building your own solutions
|
||||
|
||||
### Planned Competition Categories
|
||||
|
||||
**🎯 Accuracy Challenges**
|
||||
- **CIFAR-10 Sprint**: First to achieve 75% accuracy using only your TinyTorch implementations
|
||||
- **Efficient Training**: Highest accuracy achieved within memory limits (256MB, 512MB, 1GB tiers)
|
||||
- **Small Model Olympics**: Best performance with parameter count restrictions
|
||||
|
||||
**⚡ Performance Challenges**
|
||||
- **Speed Runs**: Fastest training time to reach accuracy milestones
|
||||
- **Memory Optimization**: Lowest memory usage while maintaining target accuracy
|
||||
- **Inference Efficiency**: Fastest model inference on standard hardware
|
||||
|
||||
**🛠️ Systems Mastery Challenges**
|
||||
- **Debugging Olympics**: Identify and fix intentionally buggy implementations
|
||||
- **Scaling Challenges**: Optimize code for larger datasets and models
|
||||
- **Hardware Awareness**: Best use of CPU vectorization and cache efficiency
|
||||
|
||||
**💡 Innovation Competitions**
|
||||
- **Creative Implementations**: Most elegant solution to standard ML problems
|
||||
- **Novel Optimizations**: Discover new ways to improve training efficiency
|
||||
- **Educational Tools**: Build the best learning aids for future TinyTorch students
|
||||
|
||||
### How Competitions Will Work
|
||||
|
||||
**Learning-First Design:**
|
||||
```bash
|
||||
# Future CLI commands (in development)
|
||||
tito compete list # See available challenges
|
||||
tito compete join accuracy-sprint # Register for a challenge
|
||||
tito compete submit --challenge=cifar10 # Submit your solution
|
||||
tito compete results --detailed # See results with learning insights
|
||||
```
|
||||
|
||||
**What Makes These Different:**
|
||||
- **Detailed Analysis**: Every submission gets performance profiling and optimization suggestions
|
||||
- **Learning Resources**: Access to hints, debugging guides, and optimization tutorials
|
||||
- **Peer Review**: Option to share your approach and learn from others' solutions
|
||||
- **Multiple Tiers**: Challenges for beginners (20% accuracy) through experts (90%+)
|
||||
|
||||
### Competition Timeline
|
||||
|
||||
**Phase 1: Foundation Building** (Next 2-3 months)
|
||||
- Community feedback and competition design
|
||||
- Initial infrastructure development
|
||||
- Beta testing with volunteer participants
|
||||
|
||||
**Phase 2: Soft Launch** (3-4 months)
|
||||
- First "CIFAR-10 Efficiency Challenge"
|
||||
- Small group of participants (~20-50)
|
||||
- Rapid iteration based on feedback
|
||||
|
||||
**Phase 3: Full Launch** (4-6 months)
|
||||
- Multiple simultaneous competitions
|
||||
- Automated submission and scoring
|
||||
- Rich community features and collaboration tools
|
||||
|
||||
### Educational Focus Areas
|
||||
|
||||
**Systems Engineering Skills:**
|
||||
- Memory profiling and optimization techniques
|
||||
- Performance bottleneck identification
|
||||
- Scaling behavior analysis
|
||||
- Cache-efficient algorithm design
|
||||
|
||||
**Real-World ML Engineering:**
|
||||
- Production-ready code practices
|
||||
- Debugging distributed training issues
|
||||
- Resource constraint optimization
|
||||
- Hardware-aware implementations
|
||||
|
||||
**Collaborative Problem Solving:**
|
||||
- Code review and peer learning
|
||||
- Mentoring between experience levels
|
||||
- Team-based challenges for larger projects
|
||||
|
||||
### Join the Design Process
|
||||
|
||||
**Help Us Build Better Competitions:**
|
||||
|
||||
We want your input on what would make these competitions most valuable for learning:
|
||||
|
||||
- What systems engineering skills do you want to develop?
|
||||
- What types of challenges would motivate you to participate?
|
||||
- How can we make competitions inclusive for all skill levels?
|
||||
- What would help you learn most from other participants' approaches?
|
||||
|
||||
**Current Discussion Topics:**
|
||||
- Competition format and scoring criteria
|
||||
- Mentorship and collaboration features
|
||||
- Fair resource usage policies
|
||||
- Educational content integration
|
||||
|
||||
**Share Your Ideas:** [GitHub Discussions - Competitions](https://github.com/harvard-edge/TinyTorch/discussions)
|
||||
|
||||
---
|
||||
|
||||
## What You Can Do Now
|
||||
|
||||
🚧 **While We Build This Feature**
|
||||
|
||||
**1. Practice Competition Skills:**
|
||||
```bash
|
||||
# Use existing tools to prepare
|
||||
tito checkpoint test 14 # Practice benchmarking skills
|
||||
tito checkpoint test 13 # Test your kernel optimization knowledge
|
||||
tito module complete 11_training # Master the training pipeline
|
||||
```
|
||||
|
||||
**2. Connect with Future Competitors:**
|
||||
- Find training partners in GitHub Discussions
|
||||
- Share your current accuracy achievements
|
||||
- Ask for optimization tips and debugging help
|
||||
- Form study groups for collaborative learning
|
||||
|
||||
**3. Build Your Competition Portfolio:**
|
||||
- Track your CIFAR-10 accuracy improvements over time
|
||||
- Document your optimization techniques and learnings
|
||||
- Practice explaining your system design decisions
|
||||
- Build profiling and debugging skills
|
||||
|
||||
**4. Share Your Training Journey:**
|
||||
- Post milestone achievements (50%, 60%, 70%+ accuracy)
|
||||
- Share interesting bugs you've debugged
|
||||
- Explain optimization techniques you've discovered
|
||||
- Help others troubleshoot their implementations
|
||||
|
||||
---
|
||||
|
||||
<div style="background: #e8f4fd; border: 2px solid #1976d2; padding: 2rem; border-radius: 0.5rem; margin: 2rem 0; text-align: center;">
|
||||
<h3 style="margin: 0 0 1rem 0; color: #1976d2;">🚀 Early Access Program</h3>
|
||||
<p style="margin: 0 0 1rem 0; color: #424242;">Want to be among the first to try TinyTorch competitions?</p>
|
||||
<p style="margin: 0 0 1rem 0; color: #424242;"><strong>Join our beta testing group:</strong> We'll notify you when the first challenges are ready for testing</p>
|
||||
<a href="https://github.com/harvard-edge/TinyTorch/discussions/new?category=competitions" style="display: inline-block; background: #1976d2; color: white; padding: 0.5rem 1rem; border-radius: 0.25rem; text-decoration: none;">Join Beta Program →</a>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## The Bigger Picture
|
||||
|
||||
**Why We're Building This:**
|
||||
|
||||
TinyTorch competitions aren't about proving who's the smartest—they're about creating a community where everyone can push their understanding of ML systems engineering further. Whether you're aiming for your first 30% accuracy or optimizing for 95%+, these challenges will help you think like a systems engineer.
|
||||
|
||||
**Our Promise:**
|
||||
- Educational value always comes first
|
||||
- Inclusive design for all skill levels
|
||||
- Honest timelines and realistic expectations
|
||||
- Community collaboration over individual competition
|
||||
- Real learning outcomes, not just leaderboard positions
|
||||
|
||||
**The ultimate goal:** Help you become the kind of ML engineer who can debug any training issue, optimize any bottleneck, and build systems that scale—skills you'll use throughout your career.
|
||||
+1
-1
@@ -48,7 +48,7 @@ As you complete modules, unlock **historical milestone demonstrations** that pro
|
||||
- **🤖 2017: Transformers** - Language generation with YOUR attention
|
||||
- **⚡ 2024: Systems Age** - Production optimization with YOUR profiling
|
||||
|
||||
**📖 See [Journey Through ML History](chapters/milestones-overview.html)** for complete milestone details and requirements.
|
||||
**📖 See [Journey Through ML History](chapters/milestones.html)** for complete milestone details and requirements.
|
||||
|
||||
## Why Build Instead of Use?
|
||||
|
||||
|
||||
+207
-66
@@ -1,92 +1,233 @@
|
||||
# 🌍 Community Leaderboard
|
||||
# 🏆 Leaderboard
|
||||
|
||||
**Compete. Optimize. Rank.**
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Competition Rankings
|
||||
|
||||
The TinyTorch Olympics Leaderboard showcases the top-performing systems from students who have completed the capstone challenge. Rankings are updated in real-time as new submissions are evaluated.
|
||||
|
||||
<div style="background: #f8f9fa; border: 1px solid #dee2e6; padding: 2rem; border-radius: 0.5rem; text-align: center; margin: 2rem 0;">
|
||||
<h2 style="margin: 0 0 1rem 0; color: #495057;">Planned Community Feature</h2>
|
||||
<p style="margin: 0; color: #6c757d;">Help learners track progress and connect with others building ML systems from scratch</p>
|
||||
<h2 style="margin: 0 0 1rem 0; color: #495057;">Live Leaderboard (Coming Soon)</h2>
|
||||
<p style="margin: 0; color: #6c757d;">Competition rankings will be displayed here after Module 20 infrastructure is deployed</p>
|
||||
</div>
|
||||
|
||||
## What This Will Be
|
||||
---
|
||||
|
||||
The TinyTorch Community Leaderboard is a **planned feature** to help learners track their progress and connect with others building ML systems from scratch.
|
||||
## 📊 Current Competition Categories
|
||||
|
||||
### The Vision
|
||||
### ⚡ Speed Demon
|
||||
**Fastest inference on standard hardware**
|
||||
- Metric: Inferences per second
|
||||
- Minimum accuracy: ≥90%
|
||||
- Focus: Computational optimization
|
||||
|
||||
We want to create an inclusive space where:
|
||||
- Everyone can track their learning journey (from 10% to 90% accuracy)
|
||||
- Students can find study partners at similar skill levels
|
||||
- Progress is celebrated at all stages, not just the top scores
|
||||
- The community helps each other debug and improve
|
||||
### 💾 Memory Miser
|
||||
**Smallest memory footprint**
|
||||
- Metric: Peak memory usage (MB)
|
||||
- Minimum accuracy: ≥85%
|
||||
- Focus: Efficient architectures
|
||||
|
||||
### How It Will Work
|
||||
### 📱 Edge Expert
|
||||
**Best performance on constrained hardware**
|
||||
- Metric: Composite score
|
||||
- Platform: Raspberry Pi 4B
|
||||
- Focus: Complete optimization
|
||||
|
||||
**Simple Progress Tracking:**
|
||||
### 🔋 Energy Efficient
|
||||
**Lowest power consumption**
|
||||
- Metric: Energy per inference (joules)
|
||||
- Focus: Algorithm efficiency
|
||||
|
||||
### 🏃♂️ TinyMLPerf
|
||||
**MLPerf-style benchmark suite**
|
||||
- Metric: Standardized benchmarks
|
||||
- Focus: Production readiness
|
||||
|
||||
---
|
||||
|
||||
## 🏅 How to Compete
|
||||
|
||||
### 1. Complete Prerequisites
|
||||
```bash
|
||||
# Future CLI commands (not yet implemented)
|
||||
tito leaderboard join # Register for the community
|
||||
tito leaderboard submit # Submit your model's accuracy
|
||||
tito leaderboard view # See community progress
|
||||
# Finish all required modules
|
||||
tito checkpoint status
|
||||
|
||||
# Verify you're ready for capstone
|
||||
tito module test 20
|
||||
```
|
||||
|
||||
**What We'll Track:**
|
||||
- Your best accuracy on standard benchmarks (CIFAR-10, etc.)
|
||||
- Which modules you've completed
|
||||
- Your learning streak (days active)
|
||||
- Helpful contributions to others
|
||||
### 2. Submit Your Model
|
||||
```bash
|
||||
# Register for competition
|
||||
tito olympics register
|
||||
|
||||
### Community Levels
|
||||
# Submit baseline
|
||||
tito olympics submit --baseline
|
||||
|
||||
We envision organizing learners into supportive groups:
|
||||
# After optimization, submit final
|
||||
tito olympics submit --final
|
||||
```
|
||||
|
||||
- **🚀 Starting** (<20% accuracy) - Just beginning the journey
|
||||
- **🌱 Learning** (20-40%) - Building foundations
|
||||
- **📈 Progressing** (40-60%) - Gaining momentum
|
||||
- **⭐ Advanced** (60-80%) - Mastering concepts
|
||||
- **🏆 Elite** (80%+) - Systems experts
|
||||
### 3. View Rankings
|
||||
```bash
|
||||
# Check your scores
|
||||
tito olympics scores
|
||||
|
||||
### Special Events
|
||||
# View full leaderboard
|
||||
tito olympics leaderboard
|
||||
|
||||
**Monthly Olympics** (planned):
|
||||
- Efficiency challenges (highest accuracy with memory limits)
|
||||
- Speed runs (fastest to reach milestones)
|
||||
- Creative implementations (most innovative approaches)
|
||||
|
||||
### Join the Discussion
|
||||
|
||||
Want to help shape this feature? We'd love your input:
|
||||
|
||||
- What would motivate you to participate?
|
||||
- What metrics matter most to you?
|
||||
- How can we make this inclusive for all skill levels?
|
||||
|
||||
**Share your thoughts:** [GitHub Discussions](https://github.com/harvard-edge/TinyTorch/discussions)
|
||||
# Generate report
|
||||
tito olympics report --format pdf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Status
|
||||
## 🎯 Scoring System
|
||||
|
||||
🚧 **Under Development**
|
||||
### Primary Ranking
|
||||
- **Category-specific metric**: Speed, memory, energy, etc.
|
||||
- **Accuracy threshold**: Must meet minimum to qualify
|
||||
- **Tie-breaker**: Higher accuracy wins
|
||||
|
||||
The leaderboard system is planned but not yet implemented. For now:
|
||||
### Bonus Recognition
|
||||
- **🚀 Innovation Award**: Novel optimization techniques
|
||||
- **📚 Teaching Award**: Best documented approach
|
||||
- **🎯 First Blood**: First to beat instructor baseline
|
||||
|
||||
1. **Track your own progress** using the checkpoint system:
|
||||
```bash
|
||||
tito checkpoint status
|
||||
```
|
||||
|
||||
2. **Share your achievements** in our community:
|
||||
- Post your progress in GitHub Discussions
|
||||
- Share your accuracy milestones
|
||||
- Ask for help when stuck
|
||||
|
||||
3. **Connect with others**:
|
||||
- Find study partners in Discussions
|
||||
- Share debugging tips
|
||||
- Celebrate breakthroughs together
|
||||
### Overall Champion
|
||||
- Best combined performance across ≥3 categories
|
||||
- Weighted by difficulty of optimization
|
||||
- Special recognition and portfolio artifact
|
||||
|
||||
---
|
||||
|
||||
<div style="background: #f8f9fa; border: 1px solid #dee2e6; padding: 2rem; border-radius: 0.5rem; margin: 2rem 0; text-align: center;">
|
||||
<h3 style="margin: 0 0 1rem 0; color: #495057;">📊 Want to Help Build This?</h3>
|
||||
<p style="margin: 0 0 1rem 0; color: #6c757d;">We're looking for contributors to help implement the leaderboard system</p>
|
||||
<a href="https://github.com/harvard-edge/TinyTorch/issues" style="display: inline-block; background: #28a745; color: white; padding: 0.5rem 1rem; border-radius: 0.25rem; text-decoration: none; font-weight: 500;">Contribute on GitHub →</a>
|
||||
</div>
|
||||
## 📈 Sample Leaderboard
|
||||
|
||||
### ⚡ Speed Demon Category
|
||||
|
||||
| Rank | Student | Inf/sec | Accuracy | Optimization |
|
||||
|------|---------|---------|----------|--------------|
|
||||
| 🥇 | alice_chen | 847.3 | 95.2% | Vectorization + caching |
|
||||
| 🥈 | bob_smith | 612.7 | 94.8% | Custom kernels |
|
||||
| 🥉 | carol_wong | 588.1 | 96.1% | Batch optimization |
|
||||
| 4 | dave_kim | 542.9 | 93.7% | Parallel processing |
|
||||
| 5 | eve_patel | 501.2 | 94.1% | Memory layout |
|
||||
|
||||
### 💾 Memory Miser Category
|
||||
|
||||
| Rank | Student | Memory (MB) | Accuracy | Optimization |
|
||||
|------|---------|-------------|----------|--------------|
|
||||
| 🥇 | dave_kim | 12.4 | 91.7% | INT8 quantization |
|
||||
| 🥈 | eve_patel | 15.8 | 93.2% | Weight pruning |
|
||||
| 🥉 | frank_liu | 18.2 | 89.9% | Compressed format |
|
||||
| 4 | grace_lee | 21.5 | 92.4% | Activation sharing |
|
||||
| 5 | henry_zhao | 24.1 | 90.8% | Efficient layers |
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Hall of Fame
|
||||
|
||||
### Semester Champions
|
||||
|
||||
**Spring 2024**
|
||||
- 🏆 Overall: Jordan Lee (95.2 composite score)
|
||||
- ⚡ Speed: Alice Chen (847.3 inf/sec)
|
||||
- 💾 Memory: Dave Kim (12.4 MB)
|
||||
- 📱 Edge: Grace Lee (94.5 score)
|
||||
|
||||
**Fall 2023**
|
||||
- 🏆 Overall: Sam Park (93.8 composite score)
|
||||
- ⚡ Speed: Morgan Smith (812.1 inf/sec)
|
||||
- 💾 Memory: Alex Wong (13.2 MB)
|
||||
- 📱 Edge: Taylor Brown (92.7 score)
|
||||
|
||||
---
|
||||
|
||||
## 🎓 What Leaderboard Performance Shows
|
||||
|
||||
### To Potential Employers
|
||||
- **Systems engineering skills**: You can optimize real systems
|
||||
- **Competitive performance**: You can achieve results under constraints
|
||||
- **Technical depth**: You understand performance trade-offs
|
||||
- **Quantifiable achievements**: Clear metrics of capability
|
||||
|
||||
### Portfolio Impact
|
||||
|
||||
**Strong statement:**
|
||||
> "Ranked #2 in Memory Efficiency in TinyTorch Olympics (Fall 2024), achieving 13.8 MB footprint with 92.1% accuracy through quantization and pruning techniques."
|
||||
|
||||
**Hiring managers recognize:**
|
||||
- Competitive achievement (leaderboard ranking)
|
||||
- Technical specificity (quantization, pruning)
|
||||
- Quantitative results (13.8 MB, 92.1% accuracy)
|
||||
- Systems thinking (memory vs. accuracy trade-offs)
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Ready to Compete?
|
||||
|
||||
1. **Complete Module 20** (Capstone)
|
||||
2. **Optimize your system** using modules 14-19
|
||||
3. **Submit your model** for evaluation
|
||||
4. **See your ranking** on the leaderboard
|
||||
|
||||
```bash
|
||||
# Start your Olympic journey
|
||||
tito olympics register
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📅 Competition Timeline
|
||||
|
||||
### Ongoing Submissions
|
||||
- Leaderboard accepts submissions year-round
|
||||
- Rankings update in real-time
|
||||
- Semester champions crowned at end of term
|
||||
|
||||
### Seasonal Events
|
||||
- **Mid-semester sprint**: Early optimization challenge
|
||||
- **Final week rush**: Last chance to climb rankings
|
||||
- **Victory ceremony**: Recognition of top performers
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Fair Competition
|
||||
|
||||
### Rules & Guidelines
|
||||
|
||||
**Allowed:**
|
||||
- Any technique from modules 1-19
|
||||
- Custom implementations within TinyTorch
|
||||
- Novel optimization strategies
|
||||
- Hardware-specific optimizations
|
||||
|
||||
**Not Allowed:**
|
||||
- External ML frameworks (PyTorch, etc.)
|
||||
- Pre-trained external models
|
||||
- Hardcoded test outputs
|
||||
- Breaking API contracts
|
||||
|
||||
**Verification:**
|
||||
- All submissions automatically validated
|
||||
- Code review for top 10 in each category
|
||||
- Reproducibility required
|
||||
- Fair hardware access provided
|
||||
|
||||
---
|
||||
|
||||
<div style="background: #e8f4fd; border: 2px solid #1976d2; padding: 2rem; border-radius: 0.5rem; margin: 2rem 0; text-align: center;">
|
||||
<h3 style="margin: 0 0 1rem 0; color: #1976d2;">🏆 Join the Competition</h3>
|
||||
<p style="margin: 0 0 1rem 0; color: #424242;">Complete Module 20 and submit your optimized system</p>
|
||||
<p style="margin: 0; color: #424242;"><strong>Prove your systems engineering skills. See how you rank.</strong></p>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
**The leaderboard doesn't lie. Your optimization skills speak for themselves.**
|
||||
|
||||
*Ready to compete?* → Complete [Module 20: Capstone](chapters/20-capstone.md)
|
||||
|
||||
@@ -117,7 +117,7 @@ As you progress, **prove what you've built** by recreating history's greatest ML
|
||||
**After Module 08**: Achieve **95%+ accuracy on MNIST** with 1986 backpropagation
|
||||
**After Module 09**: Hit **75%+ on CIFAR-10** with 1998 CNNs - your North Star goal! 🎯
|
||||
|
||||
**📖 See [Journey Through ML History](chapters/milestones-overview.html)** for complete milestone demonstrations.
|
||||
**📖 See [Journey Through ML History](chapters/milestones.html)** for complete milestone demonstrations.
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
@@ -231,8 +231,8 @@ Every component follows this pattern:
|
||||
|
||||
### Choose Your Module
|
||||
**New to ML frameworks?** → Start with [Setup](../chapters/01-setup.md)
|
||||
**Have ML experience?** → Jump to [Tensors](../chapters/02-tensor.md)
|
||||
**Want to see the vision?** → Try [Activations](../chapters/03-activations.md)
|
||||
**Have ML experience?** → Jump to [Tensors](../chapters/01-tensor.md)
|
||||
**Want to see the vision?** → Try [Activations](../chapters/02-activations.md)
|
||||
|
||||
### Get Help
|
||||
- **💬 Discussions**: GitHub Discussions for questions
|
||||
|
||||
+34
-32
@@ -28,7 +28,7 @@ These examples demonstrate the **evolutionary progression of neural networks** f
|
||||
|
||||
## 📅 **Historical Timeline & Module Mapping**
|
||||
|
||||
### **🧠 Perceptron 1957** - `perceptron_1957/`
|
||||
### **🧠 Perceptron 1957** - `01_1957_perceptron/`
|
||||
**After Modules 2-4** • *Foundation Building*
|
||||
|
||||
```
|
||||
@@ -51,7 +51,7 @@ Input → Linear → Sigmoid → Binary Output
|
||||
|
||||
---
|
||||
|
||||
### **⚡ XOR Problem 1969** - `xor_1969/`
|
||||
### **⚡ XOR Problem 1969** - `02_1969_xor_crisis/`
|
||||
**After Modules 2-6** • *Breaking Limitations*
|
||||
|
||||
```
|
||||
@@ -74,7 +74,7 @@ Input → Linear → ReLU → Linear → Output
|
||||
|
||||
---
|
||||
|
||||
### **🔢 MNIST MLP 1986** - `mnist_mlp_1986/`
|
||||
### **🔢 MNIST MLP 1986** - `03_1986_mlp_revival/`
|
||||
**After Modules 2-8** • *Real Vision Problems*
|
||||
|
||||
```
|
||||
@@ -97,7 +97,7 @@ Images → Flatten → Linear → ReLU → Linear → ReLU → Linear → Classe
|
||||
|
||||
---
|
||||
|
||||
### **🖼️ CIFAR CNN Modern** - `cifar_cnn_modern/`
|
||||
### **🖼️ CIFAR CNN Modern** - `04_1998_cnn_revolution/`
|
||||
**After Modules 2-10** • *Spatial Understanding*
|
||||
|
||||
```
|
||||
@@ -120,7 +120,7 @@ Images → Conv → ReLU → Pool → Conv → ReLU → Pool → Flatten → Lin
|
||||
|
||||
---
|
||||
|
||||
### **🤖 TinyGPT 2018** - `gpt_2018/`
|
||||
### **🤖 Transformer Era 2017** - `05_2017_transformer_era/`
|
||||
**After Modules 2-14** • *Language Understanding*
|
||||
|
||||
```
|
||||
@@ -202,23 +202,22 @@ for hidden_size in [64, 128, 256, 512]:
|
||||
## 📂 **File Structure**
|
||||
|
||||
```
|
||||
examples/
|
||||
├── README.md # This file - milestone overview
|
||||
├── perceptron_1957/
|
||||
│ └── rosenblatt_perceptron.py # First trainable neural network
|
||||
├── xor_1969/
|
||||
│ └── minsky_xor_problem.py # Non-linear problem solving
|
||||
├── mnist_mlp_1986/
|
||||
│ └── train_mlp.py # Real vision with multi-layer networks
|
||||
├── cifar_cnn_modern/
|
||||
│ ├── train_cnn.py # Spatial feature extraction with CNNs
|
||||
│ └── data/ # CIFAR-10 dataset
|
||||
├── gpt_2018/
|
||||
│ └── train_gpt.py # Language modeling with transformers
|
||||
└── pretrained/
|
||||
├── mnist_mlp_weights.npz # Pre-trained weights for quick demos
|
||||
├── cifar10_cnn_weights.npz
|
||||
└── xor_weights.npz
|
||||
milestones/
|
||||
├── README.md # This file - milestone overview
|
||||
├── 01_1957_perceptron/
|
||||
│ └── perceptron_trained.py # First trainable neural network
|
||||
├── 02_1969_xor_crisis/
|
||||
│ └── xor_solved.py # Non-linear problem solving
|
||||
├── 03_1986_mlp_revival/
|
||||
│ ├── mlp_digits.py # 8x8 digits
|
||||
│ └── mlp_mnist.py # Full MNIST with multi-layer networks
|
||||
├── 04_1998_cnn_revolution/
|
||||
│ ├── cnn_digits.py # Spatial features on digits
|
||||
│ └── lecun_cifar10.py # CIFAR-10 with CNNs
|
||||
├── 05_2017_transformer_era/
|
||||
│ └── vaswani_shakespeare.py # Language modeling with transformers
|
||||
└── 06_2024_systems_age/
|
||||
└── optimize_models.py # Modern ML engineering
|
||||
```
|
||||
|
||||
---
|
||||
@@ -252,31 +251,34 @@ cd /path/to/TinyTorch
|
||||
Test architecture and imports without waiting for downloads:
|
||||
```bash
|
||||
# Test what you've built so far
|
||||
python examples/perceptron_1957/rosenblatt_perceptron.py --test-only
|
||||
python examples/xor_1969/minsky_xor_problem.py --test-only
|
||||
cd milestones
|
||||
python 01_1957_perceptron/perceptron_trained.py
|
||||
python 02_1969_xor_crisis/xor_solved.py
|
||||
```
|
||||
|
||||
#### **🎯 Full Milestone Demonstrations**
|
||||
|
||||
```bash
|
||||
cd milestones
|
||||
|
||||
# After Module 04 - Foundation (30 seconds)
|
||||
python examples/perceptron_1957/rosenblatt_perceptron.py
|
||||
python 01_1957_perceptron/perceptron_trained.py
|
||||
# Demonstrates: YOU built Linear layers + activation functions
|
||||
|
||||
# After Module 06 - Autograd (1 minute)
|
||||
python examples/xor_1969/minsky_xor_problem.py
|
||||
python 02_1969_xor_crisis/xor_solved.py
|
||||
# Demonstrates: YOU built gradient computation + training loops
|
||||
|
||||
# After Module 08 - Training (2-3 minutes + MNIST download)
|
||||
python examples/mnist_mlp_1986/train_mlp.py
|
||||
python 03_1986_mlp_revival/mlp_mnist.py
|
||||
# Demonstrates: YOU built complete vision pipeline
|
||||
|
||||
# After Module 10 - DataLoader + Spatial (3-5 minutes + CIFAR download)
|
||||
python examples/cifar_cnn_modern/train_cnn.py
|
||||
# After Module 09 - Spatial (3-5 minutes + CIFAR download)
|
||||
python 04_1998_cnn_revolution/lecun_cifar10.py
|
||||
# Demonstrates: YOU built convolutional networks
|
||||
|
||||
# After Module 14 - Transformers (5-10 minutes)
|
||||
python examples/gpt_2018/train_gpt.py
|
||||
# After Module 13 - Transformers (5-10 minutes)
|
||||
python 05_2017_transformer_era/vaswani_shakespeare.py
|
||||
# Demonstrates: YOU built attention mechanisms + language models
|
||||
```
|
||||
|
||||
@@ -385,4 +387,4 @@ By completing all milestone examples, students will:
|
||||
|
||||
**Remember**: These aren't just coding exercises - they're journeys through the history of AI that prepare you for the future of ML systems engineering.
|
||||
|
||||
🚀 **Start your journey**: `python examples/perceptron_1957/rosenblatt_perceptron.py`
|
||||
🚀 **Start your journey**: `cd milestones && python 01_1957_perceptron/perceptron_trained.py`
|
||||
@@ -55,149 +55,175 @@ from tinytorch.text.embeddings import Embedding, PositionalEncoding, create_sinu
|
||||
- **Integration:** Works seamlessly with tokenizers for complete text processing pipeline
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "imports", "solution": true}
|
||||
"""
|
||||
## 1. Essential Imports and Setup
|
||||
|
||||
Setting up our embedding toolkit with tensor operations and mathematical functions.
|
||||
"""
|
||||
|
||||
#| default_exp text.embeddings
|
||||
#| export
|
||||
|
||||
# %%
|
||||
import numpy as np
|
||||
import math
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
# Core tensor operations - our foundation
|
||||
### BEGIN SOLUTION
|
||||
# For this educational implementation, we'll create a simple Tensor class
|
||||
# In practice, this would import from tinytorch.core.tensor
|
||||
|
||||
class Tensor:
|
||||
"""Educational tensor for embeddings module."""
|
||||
|
||||
def __init__(self, data, requires_grad=False):
|
||||
self.data = np.array(data)
|
||||
self.shape = self.data.shape
|
||||
self.requires_grad = requires_grad
|
||||
self.grad = None
|
||||
|
||||
def __repr__(self):
|
||||
return f"Tensor({self.data})"
|
||||
|
||||
def __getitem__(self, idx):
|
||||
return Tensor(self.data[idx])
|
||||
|
||||
def __add__(self, other):
|
||||
if isinstance(other, Tensor):
|
||||
return Tensor(self.data + other.data)
|
||||
return Tensor(self.data + other)
|
||||
|
||||
def size(self, dim=None):
|
||||
if dim is None:
|
||||
return self.shape
|
||||
return self.shape[dim]
|
||||
|
||||
def reshape(self, *shape):
|
||||
return Tensor(self.data.reshape(shape))
|
||||
|
||||
def expand(self, *shape):
|
||||
return Tensor(np.broadcast_to(self.data, shape))
|
||||
|
||||
def parameters(self):
|
||||
return [self] if self.requires_grad else []
|
||||
|
||||
# Simple Linear layer for this module
|
||||
class Linear:
|
||||
"""Educational linear layer."""
|
||||
|
||||
def __init__(self, in_features, out_features, bias=True):
|
||||
# Xavier initialization
|
||||
limit = math.sqrt(6.0 / (in_features + out_features))
|
||||
self.weight = Tensor(
|
||||
np.random.uniform(-limit, limit, (in_features, out_features)),
|
||||
requires_grad=True
|
||||
)
|
||||
self.bias = Tensor(np.zeros(out_features), requires_grad=True) if bias else None
|
||||
|
||||
def forward(self, x):
|
||||
result = Tensor(np.dot(x.data, self.weight.data))
|
||||
if self.bias is not None:
|
||||
result = result + self.bias
|
||||
return result
|
||||
|
||||
def parameters(self):
|
||||
params = [self.weight]
|
||||
if self.bias is not None:
|
||||
params.append(self.bias)
|
||||
return params
|
||||
### END SOLUTION
|
||||
# Import from previous modules - following dependency chain
|
||||
from tinytorch.core.tensor import Tensor
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 2. Understanding Token Embeddings - From Discrete to Dense
|
||||
## 1. Introduction - Why Embeddings?
|
||||
|
||||
Before we implement embeddings, let's understand what problem they solve and how the lookup process works.
|
||||
Neural networks operate on dense vectors, but language consists of discrete tokens. Embeddings are the crucial bridge that converts discrete tokens into continuous, learnable vector representations that capture semantic meaning.
|
||||
|
||||
### The Fundamental Challenge
|
||||
### The Token-to-Vector Challenge
|
||||
|
||||
When dealing with text, we start with discrete symbols (words, characters, tokens) but neural networks need continuous numbers. Embeddings bridge this gap by creating a learned mapping from discrete tokens to dense vector representations.
|
||||
|
||||
### Token-to-Vector Transformation Visualization
|
||||
Consider the tokens from our tokenizer: [1, 42, 7] - how do we turn these discrete indices into meaningful vectors that capture semantic relationships?
|
||||
|
||||
```
|
||||
Traditional One-Hot Encoding (Sparse):
|
||||
Token "cat" (index 42) → [0, 0, ..., 1, ..., 0] (50,000 elements, mostly zeros)
|
||||
position 42
|
||||
|
||||
Modern Embedding Lookup (Dense):
|
||||
Token "cat" (index 42) → [0.1, -0.3, 0.7, 0.2, ...] (512 dense, meaningful values)
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ EMBEDDING PIPELINE: Discrete Tokens → Dense Vectors │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Input (Token IDs): [1, 42, 7] │
|
||||
│ │ │
|
||||
│ ├─ Step 1: Lookup in embedding table │
|
||||
│ │ Each ID → vector of learned features │
|
||||
│ │ │
|
||||
│ ├─ Step 2: Add positional information │
|
||||
│ │ Same word at different positions → different│
|
||||
│ │ │
|
||||
│ ├─ Step 3: Create position-aware representations │
|
||||
│ │ Ready for attention mechanisms │
|
||||
│ │ │
|
||||
│ └─ Step 4: Enable semantic understanding │
|
||||
│ Similar words → similar vectors │
|
||||
│ │
|
||||
│ Output (Dense Vectors): [[0.1, 0.4, ...], [0.7, -0.2, ...]] │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### How Embedding Lookup Works
|
||||
### The Four-Layer Embedding System
|
||||
|
||||
```
|
||||
Embedding Table (vocab_size × embed_dim):
|
||||
Token ID Embedding Vector
|
||||
┌─────┐ ┌─────────────────────────┐
|
||||
0 │ 0 │ → │ [0.2, -0.1, 0.3, ...] │ "the"
|
||||
1 │ 1 │ → │ [0.1, 0.4, -0.2, ...] │ "cat"
|
||||
2 │ 2 │ → │ [-0.3, 0.1, 0.5, ...] │ "sat"
|
||||
... │ ... │ │ ... │ ...
|
||||
42 │ 42 │ → │ [0.7, -0.2, 0.1, ...] │ "dog"
|
||||
... │ ... │ │ ... │ ...
|
||||
└─────┘ └─────────────────────────┘
|
||||
Modern embedding systems combine multiple components:
|
||||
|
||||
Lookup Process:
|
||||
Input tokens: [1, 2, 42] → Output: Matrix (3 × embed_dim)
|
||||
Row 0: embedding[1] → [0.1, 0.4, -0.2, ...] "cat"
|
||||
Row 1: embedding[2] → [-0.3, 0.1, 0.5, ...] "sat"
|
||||
Row 2: embedding[42] → [0.7, -0.2, 0.1, ...] "dog"
|
||||
```
|
||||
**1. Token embeddings** - Learn semantic representations for each vocabulary token
|
||||
**2. Positional encoding** - Add information about position in sequence
|
||||
**3. Optional scaling** - Normalize embedding magnitudes (Transformer convention)
|
||||
**4. Integration** - Combine everything into position-aware representations
|
||||
|
||||
### Why Embeddings Are Powerful
|
||||
### Why This Matters
|
||||
|
||||
1. **Dense Representation**: Every dimension can contribute meaningful information
|
||||
2. **Learnable**: Vectors adjust during training to capture semantic relationships
|
||||
3. **Efficient**: O(1) lookup time regardless of vocabulary size
|
||||
4. **Semantic**: Similar words learn similar vector representations
|
||||
|
||||
### Memory Implications
|
||||
|
||||
For a vocabulary of 50,000 tokens with 512-dimensional embeddings:
|
||||
- **Storage**: 50,000 × 512 × 4 bytes = ~100MB (in FP32)
|
||||
- **Scaling**: Memory grows linearly with vocab_size × embed_dim
|
||||
- **Trade-off**: Larger embeddings capture more nuance but require more memory
|
||||
|
||||
This is why embedding tables often dominate memory usage in large language models!
|
||||
The choice of embedding strategy dramatically affects:
|
||||
- **Semantic understanding** - How well the model captures word meaning
|
||||
- **Memory requirements** - Embedding tables can be gigabytes in size
|
||||
- **Position awareness** - Whether the model understands word order
|
||||
- **Extrapolation** - How well the model handles longer sequences than training
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 3. Implementing Token Embeddings
|
||||
## 2. Foundations - Embedding Strategies
|
||||
|
||||
Now let's build the core embedding layer that performs efficient token-to-vector lookups.
|
||||
Different embedding approaches make different trade-offs between memory, semantic understanding, and computational efficiency.
|
||||
|
||||
### Token Embedding Lookup Process
|
||||
|
||||
**Approach**: Each token ID maps to a learned dense vector
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ TOKEN EMBEDDING LOOKUP PROCESS │
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Step 1: Build Embedding Table (vocab_size × embed_dim) │
|
||||
│ ┌────────────────────────────────────────────────────────┐ │
|
||||
│ │ Token ID │ Embedding Vector (learned features) │ │
|
||||
│ ├────────────────────────────────────────────────────────┤ │
|
||||
│ │ 0 │ [0.2, -0.1, 0.3, 0.8, ...] (<UNK>) │ │
|
||||
│ │ 1 │ [0.1, 0.4, -0.2, 0.6, ...] ("the") │ │
|
||||
│ │ 42 │ [0.7, -0.2, 0.1, 0.4, ...] ("cat") │ │
|
||||
│ │ 7 │ [-0.3, 0.1, 0.5, 0.2, ...] ("sat") │ │
|
||||
│ │ ... │ ... │ │
|
||||
│ └────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 2: Lookup Process (O(1) per token) │
|
||||
│ ┌────────────────────────────────────────────────────────┐ │
|
||||
│ │ Input: Token IDs [1, 42, 7] │ │
|
||||
│ │ │ │
|
||||
│ │ ID 1 → embedding[1] → [0.1, 0.4, -0.2, ...] │ │
|
||||
│ │ ID 42 → embedding[42] → [0.7, -0.2, 0.1, ...] │ │
|
||||
│ │ ID 7 → embedding[7] → [-0.3, 0.1, 0.5, ...] │ │
|
||||
│ │ │ │
|
||||
│ │ Output: Matrix (3 × embed_dim) │ │
|
||||
│ │ [[0.1, 0.4, -0.2, ...], │ │
|
||||
│ │ [0.7, -0.2, 0.1, ...], │ │
|
||||
│ │ [-0.3, 0.1, 0.5, ...]] │ │
|
||||
│ └────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 3: Training Updates Embeddings │
|
||||
│ ┌────────────────────────────────────────────────────────┐ │
|
||||
│ │ Gradients flow back to embedding table │ │
|
||||
│ │ │ │
|
||||
│ │ Similar words learn similar vectors: │ │
|
||||
│ │ "cat" and "dog" → closer in embedding space │ │
|
||||
│ │ "the" and "a" → closer in embedding space │ │
|
||||
│ │ "sat" and "run" → farther in embedding space │ │
|
||||
│ └────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Dense representation (every dimension meaningful)
|
||||
- Learnable (captures semantic relationships through training)
|
||||
- Efficient lookup (O(1) time complexity)
|
||||
- Scales to large vocabularies
|
||||
|
||||
**Cons**:
|
||||
- Memory intensive (vocab_size × embed_dim parameters)
|
||||
- Requires training to develop semantic relationships
|
||||
- Fixed vocabulary (new tokens need special handling)
|
||||
|
||||
### Positional Encoding Strategies
|
||||
|
||||
Since embeddings by themselves have no notion of order, we need positional information:
|
||||
|
||||
```
|
||||
Position-Aware Embeddings = Token Embeddings + Positional Encoding
|
||||
|
||||
Learned Approach: Fixed Mathematical Approach:
|
||||
Position 0 → [learned] Position 0 → [sin/cos pattern]
|
||||
Position 1 → [learned] Position 1 → [sin/cos pattern]
|
||||
Position 2 → [learned] Position 2 → [sin/cos pattern]
|
||||
... ...
|
||||
```
|
||||
|
||||
**Learned Positional Encoding**:
|
||||
- Trainable position embeddings
|
||||
- Can learn task-specific patterns
|
||||
- Limited to maximum training sequence length
|
||||
|
||||
**Sinusoidal Positional Encoding**:
|
||||
- Mathematical sine/cosine patterns
|
||||
- No additional parameters
|
||||
- Can extrapolate to longer sequences
|
||||
|
||||
### Strategy Comparison
|
||||
|
||||
```
|
||||
Text: "cat sat on mat" → Token IDs: [42, 7, 15, 99]
|
||||
|
||||
Token Embeddings: [vec_42, vec_7, vec_15, vec_99] # Same vectors anywhere
|
||||
Position-Aware: [vec_42+pos_0, vec_7+pos_1, vec_15+pos_2, vec_99+pos_3]
|
||||
↑ Now "cat" at position 0 ≠ "cat" at position 1
|
||||
```
|
||||
|
||||
The combination enables transformers to understand both meaning and order!
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 3. Implementation - Building Embedding Systems
|
||||
|
||||
Let's implement embedding systems from basic token lookup to sophisticated position-aware representations. We'll start with the core embedding layer and work up to complete systems.
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "embedding-class", "solution": true}
|
||||
@@ -320,79 +346,36 @@ test_unit_embedding()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 4. Understanding Positional Encoding - Teaching Models About Order
|
||||
### Learned Positional Encoding
|
||||
|
||||
Sequences have inherent order, but embeddings by themselves are orderless. We need to explicitly encode positional information so the model understands that "cat chased dog" is different from "dog chased cat".
|
||||
|
||||
### Why Position Matters in Sequences
|
||||
|
||||
Unlike images where spatial relationships are built into the 2D structure, text sequences need explicit position encoding:
|
||||
Trainable position embeddings that can learn position-specific patterns. This approach treats each position as a learnable parameter, similar to token embeddings.
|
||||
|
||||
```
|
||||
Word Order Changes Meaning:
|
||||
"The cat chased the dog" ≠ "The dog chased the cat"
|
||||
"Not good" ≠ "Good not"
|
||||
"She told him" ≠ "Him told she"
|
||||
Learned Position Embedding Process:
|
||||
|
||||
Step 1: Initialize Position Embedding Table
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ Position │ Learnable Vector (trainable parameters) │
|
||||
├───────────────────────────────────────────────────────────────┤
|
||||
│ 0 │ [0.1, -0.2, 0.4, ...] ← learns "start" patterns │
|
||||
│ 1 │ [0.3, 0.1, -0.1, ...] ← learns "second" patterns│
|
||||
│ 2 │ [-0.1, 0.5, 0.2, ...] ← learns "third" patterns │
|
||||
│ ... │ ... │
|
||||
│ 511 │ [0.4, -0.3, 0.1, ...] ← learns "late" patterns │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
|
||||
Step 2: Add to Token Embeddings
|
||||
Input: ["The", "cat", "sat"] → Token IDs: [1, 42, 7]
|
||||
|
||||
Token embeddings: Position embeddings: Combined:
|
||||
[1] → [0.1, 0.4, ...] + [0.1, -0.2, ...] = [0.2, 0.2, ...]
|
||||
[42] → [0.7, -0.2, ...] + [0.3, 0.1, ...] = [1.0, -0.1, ...]
|
||||
[7] → [-0.3, 0.1, ...] + [-0.1, 0.5, ...] = [-0.4, 0.6, ...]
|
||||
|
||||
Result: Position-aware embeddings that can learn task-specific patterns!
|
||||
```
|
||||
|
||||
### Two Approaches to Position Encoding
|
||||
|
||||
```
|
||||
1. Learned Positional Embeddings:
|
||||
┌─────────────────────────────────────┐
|
||||
│ Position │ Learned Vector │
|
||||
├─────────────────────────────────────┤
|
||||
│ 0 │ [0.1, -0.2, 0.4, ...] │ (trained)
|
||||
│ 1 │ [0.3, 0.1, -0.1, ...] │ (trained)
|
||||
│ 2 │ [-0.1, 0.5, 0.2, ...] │ (trained)
|
||||
│ ... │ ... │
|
||||
│ 511 │ [0.4, -0.3, 0.1, ...] │ (trained)
|
||||
└─────────────────────────────────────┘
|
||||
✓ Can learn task-specific patterns
|
||||
✗ Fixed maximum sequence length
|
||||
✗ Requires additional parameters
|
||||
|
||||
2. Sinusoidal Position Encodings:
|
||||
┌─────────────────────────────────────┐
|
||||
│ Position │ Mathematical Pattern │
|
||||
├─────────────────────────────────────┤
|
||||
│ 0 │ [0.0, 1.0, 0.0, ...] │ (computed)
|
||||
│ 1 │ [sin1, cos1, sin2, ...] │ (computed)
|
||||
│ 2 │ [sin2, cos2, sin4, ...] │ (computed)
|
||||
│ ... │ ... │
|
||||
│ N │ [sinN, cosN, sin2N,...] │ (computed)
|
||||
└─────────────────────────────────────┘
|
||||
✓ No additional parameters
|
||||
✓ Can extrapolate to longer sequences
|
||||
✗ Cannot adapt to specific patterns
|
||||
```
|
||||
|
||||
### How Positional Information Gets Added
|
||||
|
||||
```
|
||||
Token Embeddings + Positional Encodings = Position-Aware Representations
|
||||
|
||||
Input Sequence: ["The", "cat", "sat"]
|
||||
Token IDs: [ 1, 42, 7 ]
|
||||
|
||||
Step 1: Token Embeddings
|
||||
[1] → [0.1, 0.4, -0.2, ...]
|
||||
[42]→ [0.7, -0.2, 0.1, ...]
|
||||
[7] → [-0.3, 0.1, 0.5, ...]
|
||||
|
||||
Step 2: Position Encodings
|
||||
pos 0 → [0.0, 1.0, 0.0, ...]
|
||||
pos 1 → [0.8, 0.6, 0.1, ...]
|
||||
pos 2 → [0.9, -0.4, 0.2, ...]
|
||||
|
||||
Step 3: Addition (element-wise)
|
||||
Result:
|
||||
[0.1+0.0, 0.4+1.0, -0.2+0.0, ...] = [0.1, 1.4, -0.2, ...] "The" at position 0
|
||||
[0.7+0.8, -0.2+0.6, 0.1+0.1, ...] = [1.5, 0.4, 0.2, ...] "cat" at position 1
|
||||
[-0.3+0.9, 0.1-0.4, 0.5+0.2, ...] = [0.6, -0.3, 0.7, ...] "sat" at position 2
|
||||
```
|
||||
|
||||
This way, the same word gets different representations based on its position in the sentence!
|
||||
**Why learned positions work**: The model can discover that certain positions have special meaning (like sentence beginnings, question words, etc.) and learn specific representations for those patterns.
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
@@ -542,62 +525,65 @@ test_unit_positional_encoding()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 6. Understanding Sinusoidal Position Encodings
|
||||
### Sinusoidal Positional Encoding
|
||||
|
||||
Now let's explore the elegant mathematical approach to position encoding used in the original Transformer paper. Instead of learning position patterns, we'll use trigonometric functions to create unique, continuous position signatures.
|
||||
|
||||
### The Mathematical Intuition
|
||||
|
||||
Sinusoidal encodings use sine and cosine functions at different frequencies to create unique position signatures:
|
||||
Mathematical position encoding that creates unique signatures for each position using trigonometric functions. This approach requires no additional parameters and can extrapolate to sequences longer than seen during training.
|
||||
|
||||
```
|
||||
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) # Even dimensions
|
||||
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) # Odd dimensions
|
||||
┌───────────────────────────────────────────────────────────────────────────┐
|
||||
│ SINUSOIDAL POSITION ENCODING: Mathematical Position Signatures │
|
||||
├───────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ MATHEMATICAL FORMULA: │
|
||||
│ ┌──────────────────────────────────────────────────────────────┐ │
|
||||
│ │ PE(pos, 2i) = sin(pos / 10000^(2i/embed_dim)) # Even dims │ │
|
||||
│ │ PE(pos, 2i+1) = cos(pos / 10000^(2i/embed_dim)) # Odd dims │ │
|
||||
│ │ │ │
|
||||
│ │ Where: │ │
|
||||
│ │ pos = position in sequence (0, 1, 2, ...) │ │
|
||||
│ │ i = dimension pair index (0, 1, 2, ...) │ │
|
||||
│ │ 10000 = base frequency (creates different wavelengths) │ │
|
||||
│ └──────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ FREQUENCY PATTERN ACROSS DIMENSIONS: │
|
||||
│ ┌──────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Dimension: 0 1 2 3 4 5 6 7 │ │
|
||||
│ │ Frequency: High High Med Med Low Low VLow VLow │ │
|
||||
│ │ Function: sin cos sin cos sin cos sin cos │ │
|
||||
│ │ │ │
|
||||
│ │ pos=0: [0.00, 1.00, 0.00, 1.00, 0.00, 1.00, 0.00, 1.00] │ │
|
||||
│ │ pos=1: [0.84, 0.54, 0.01, 1.00, 0.00, 1.00, 0.00, 1.00] │ │
|
||||
│ │ pos=2: [0.91,-0.42, 0.02, 1.00, 0.00, 1.00, 0.00, 1.00] │ │
|
||||
│ │ pos=3: [0.14,-0.99, 0.03, 1.00, 0.00, 1.00, 0.00, 1.00] │ │
|
||||
│ │ │ │
|
||||
│ │ Each position gets a unique mathematical "fingerprint"! │ │
|
||||
│ └──────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ WHY THIS WORKS: │
|
||||
│ ┌──────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Wave Pattern Visualization: │ │
|
||||
│ │ │ │
|
||||
│ │ Dim 0: ∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿ (rapid oscillation) │ │
|
||||
│ │ Dim 2: ∿---∿---∿---∿---∿---∿ (medium frequency) │ │
|
||||
│ │ Dim 4: ∿-----∿-----∿-----∿-- (low frequency) │ │
|
||||
│ │ Dim 6: ∿----------∿---------- (very slow changes) │ │
|
||||
│ │ │ │
|
||||
│ │ • High frequency dims change rapidly between positions │ │
|
||||
│ │ • Low frequency dims change slowly │ │
|
||||
│ │ • Combination creates unique signature for each position │ │
|
||||
│ │ • Similar positions have similar (but distinct) encodings │ │
|
||||
│ └──────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ KEY ADVANTAGES: │
|
||||
│ • Zero parameters (no memory overhead) │
|
||||
│ • Infinite sequence length (can extrapolate) │
|
||||
│ • Smooth transitions (nearby positions are similar) │
|
||||
│ • Mathematical elegance (interpretable patterns) │
|
||||
│ │
|
||||
└───────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Why This Works - Frequency Visualization
|
||||
|
||||
```
|
||||
Position Encoding Pattern (embed_dim=8, showing 4 positions):
|
||||
|
||||
Dimension: 0 1 2 3 4 5 6 7
|
||||
Frequency: High High Med Med Low Low VLow VLow
|
||||
Function: sin cos sin cos sin cos sin cos
|
||||
|
||||
pos=0: [0.00, 1.00, 0.00, 1.00, 0.00, 1.00, 0.00, 1.00]
|
||||
pos=1: [0.84, 0.54, 0.01, 1.00, 0.00, 1.00, 0.00, 1.00]
|
||||
pos=2: [0.91, -0.42, 0.02, 1.00, 0.00, 1.00, 0.00, 1.00]
|
||||
pos=3: [0.14, -0.99, 0.03, 1.00, 0.00, 1.00, 0.00, 1.00]
|
||||
|
||||
Notice how:
|
||||
- High frequency dimensions (0,1) change quickly between positions
|
||||
- Low frequency dimensions (6,7) change slowly
|
||||
- Each position gets a unique "fingerprint"
|
||||
```
|
||||
|
||||
### Visual Pattern of Sinusoidal Encodings
|
||||
|
||||
```
|
||||
Frequency Spectrum Across Dimensions:
|
||||
High Freq ← - - - - - - - - - - - - - - - - - - - - - → Low Freq
|
||||
Dim: 0 1 2 3 4 5 6 7 8 9 ... 510 511
|
||||
|
||||
Wave Pattern for Position Progression:
|
||||
Dim 0: ∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿ (rapid oscillation)
|
||||
Dim 2: ∿---∿---∿---∿---∿---∿ (medium frequency)
|
||||
Dim 4: ∿-----∿-----∿-----∿-- (low frequency)
|
||||
Dim 6: ∿----------∿---------- (very slow changes)
|
||||
|
||||
This creates a unique "barcode" for each position!
|
||||
```
|
||||
|
||||
### Advantages of Sinusoidal Encodings
|
||||
|
||||
1. **No Parameters**: Zero additional memory overhead
|
||||
2. **Extrapolation**: Can handle sequences longer than training data
|
||||
3. **Unique Signatures**: Each position gets a distinct encoding
|
||||
4. **Smooth Transitions**: Similar positions have similar encodings
|
||||
5. **Mathematical Elegance**: Clean, interpretable patterns
|
||||
**Why transformers use this**: The mathematical structure allows the model to learn relative positions (how far apart tokens are) through simple vector operations, which is crucial for attention mechanisms!
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
@@ -714,50 +700,87 @@ test_unit_sinusoidal_embeddings()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 8. Building the Complete Embedding System
|
||||
## 4. Integration - Bringing It Together
|
||||
|
||||
Now let's integrate everything into a production-ready embedding system that handles both token and positional embeddings, supports multiple encoding types, and manages the full embedding pipeline used in modern NLP models.
|
||||
|
||||
### Complete Embedding Pipeline Visualization
|
||||
Now let's build the complete embedding system that combines token and positional embeddings into a production-ready component used in modern transformers and language models.
|
||||
|
||||
```
|
||||
Complete Embedding System Architecture:
|
||||
Complete Embedding Pipeline:
|
||||
|
||||
Input: Token IDs [1, 42, 7, 99]
|
||||
↓
|
||||
┌─────────────────────┐
|
||||
│ Token Embedding │ vocab_size × embed_dim table
|
||||
│ Lookup Table │
|
||||
└─────────────────────┘
|
||||
↓
|
||||
Token Vectors (4 × embed_dim)
|
||||
[0.1, 0.4, -0.2, ...] ← token 1
|
||||
[0.7, -0.2, 0.1, ...] ← token 42
|
||||
[-0.3, 0.1, 0.5, ...] ← token 7
|
||||
[0.9, -0.1, 0.3, ...] ← token 99
|
||||
↓
|
||||
┌─────────────────────┐
|
||||
│ Positional Encoding │ Choose: Learned, Sinusoidal, or None
|
||||
│ (Add position info) │
|
||||
└─────────────────────┘
|
||||
↓
|
||||
Position-Aware Embeddings (4 × embed_dim)
|
||||
[0.1+pos0, 0.4+pos0, ...] ← token 1 at position 0
|
||||
[0.7+pos1, -0.2+pos1, ...] ← token 42 at position 1
|
||||
[-0.3+pos2, 0.1+pos2, ...] ← token 7 at position 2
|
||||
[0.9+pos3, -0.1+pos3, ...] ← token 99 at position 3
|
||||
↓
|
||||
Optional: Scale by √embed_dim (Transformer convention)
|
||||
↓
|
||||
Ready for Attention Mechanisms!
|
||||
1. Token Lookup → 2. Position Encoding → 3. Combination → 4. Ready for Attention
|
||||
↓ ↓ ↓ ↓
|
||||
sparse IDs position info dense vectors context-aware
|
||||
```
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
### Complete Embedding System Architecture
|
||||
|
||||
The production embedding layer that powers modern transformers combines multiple components into an efficient, flexible pipeline.
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────────────────┐
|
||||
│ COMPLETE EMBEDDING SYSTEM: Token + Position → Attention-Ready │
|
||||
├───────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ INPUT: Token IDs [1, 42, 7, 99] │
|
||||
│ │ │
|
||||
│ ├─ STEP 1: TOKEN EMBEDDING LOOKUP │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ │ Token Embedding Table (vocab_size × embed_dim) │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ ID 1 → [0.1, 0.4, -0.2, ...] (semantic features) │ │
|
||||
│ │ │ ID 42 → [0.7, -0.2, 0.1, ...] (learned meaning) │ │
|
||||
│ │ │ ID 7 → [-0.3, 0.1, 0.5, ...] (dense vector) │ │
|
||||
│ │ │ ID 99 → [0.9, -0.1, 0.3, ...] (context-free) │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ├─ STEP 2: POSITIONAL ENCODING (Choose Strategy) │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ │ Strategy A: Learned PE │ │
|
||||
│ │ │ pos 0 → [trainable vector] (learns patterns) │ │
|
||||
│ │ │ pos 1 → [trainable vector] (task-specific) │ │
|
||||
│ │ │ pos 2 → [trainable vector] (fixed max length) │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ Strategy B: Sinusoidal PE │ │
|
||||
│ │ │ pos 0 → [sin/cos pattern] (mathematical) │ │
|
||||
│ │ │ pos 1 → [sin/cos pattern] (no parameters) │ │
|
||||
│ │ │ pos 2 → [sin/cos pattern] (infinite length) │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ Strategy C: No PE │ │
|
||||
│ │ │ positions ignored (order-agnostic) │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ├─ STEP 3: ELEMENT-WISE ADDITION │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ │ Token + Position = Position-Aware Representation │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ [0.1, 0.4, -0.2] + [pos0] = [0.1+p0, 0.4+p0, ...] │ │
|
||||
│ │ │ [0.7, -0.2, 0.1] + [pos1] = [0.7+p1, -0.2+p1, ...] │ │
|
||||
│ │ │ [-0.3, 0.1, 0.5] + [pos2] = [-0.3+p2, 0.1+p2, ...] │ │
|
||||
│ │ │ [0.9, -0.1, 0.3] + [pos3] = [0.9+p3, -0.1+p3, ...] │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ├─ STEP 4: OPTIONAL SCALING (Transformer Convention) │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ │ Scale by √embed_dim for gradient stability │ │
|
||||
│ │ │ Helps balance token and position magnitudes │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ └─ OUTPUT: Position-Aware Dense Vectors │
|
||||
│ Ready for attention mechanisms and transformers! │
|
||||
│ │
|
||||
│ INTEGRATION FEATURES: │
|
||||
│ • Flexible position encoding (learned/sinusoidal/none) │
|
||||
│ • Efficient batch processing with variable sequence lengths │
|
||||
│ • Memory optimization (shared position encodings) │
|
||||
│ • Production patterns (matches PyTorch/HuggingFace) │
|
||||
│ │
|
||||
└───────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Integration Features
|
||||
|
||||
- **Flexible Position Encoding**: Support learned, sinusoidal, or no positional encoding
|
||||
- **Batch Processing**: Handle variable-length sequences with padding
|
||||
- **Memory Efficiency**: Reuse position encodings across batches
|
||||
- **Production Ready**: Matches PyTorch patterns and conventions
|
||||
**Why this architecture works**: By separating token semantics from positional information, the model can learn meaning and order independently, then combine them optimally for the specific task.
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "complete-system", "solution": true}
|
||||
@@ -972,44 +995,14 @@ test_unit_complete_embedding_system()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 9. Systems Analysis - Embedding Memory and Performance
|
||||
## 5. Systems Analysis - Embedding Trade-offs
|
||||
|
||||
Understanding the systems implications of embedding layers is crucial for building scalable NLP models. Let's analyze memory usage, lookup performance, and trade-offs between different approaches.
|
||||
|
||||
### Memory Usage Analysis
|
||||
|
||||
```
|
||||
Embedding Memory Scaling:
|
||||
Vocabulary Size vs Memory Usage (embed_dim=512, FP32):
|
||||
|
||||
10K vocab: 10,000 × 512 × 4 bytes = 20 MB
|
||||
50K vocab: 50,000 × 512 × 4 bytes = 100 MB
|
||||
100K vocab: 100,000 × 512 × 4 bytes = 200 MB
|
||||
1M vocab: 1,000,000 × 512 × 4 bytes = 2 GB
|
||||
|
||||
GPT-3 Scale: 50,257 × 12,288 × 4 bytes ≈ 2.4 GB just for embeddings!
|
||||
|
||||
Memory Formula: vocab_size × embed_dim × 4 bytes (FP32)
|
||||
```
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
```
|
||||
Embedding Lookup Performance:
|
||||
- Time Complexity: O(1) per token (hash table lookup)
|
||||
- Memory Access: Random access pattern
|
||||
- Bottleneck: Memory bandwidth, not computation
|
||||
- Batching: Improves throughput via vectorization
|
||||
|
||||
Cache Efficiency:
|
||||
Repeated tokens → Cache hits → Faster access
|
||||
Diverse vocab → Cache misses → Slower access
|
||||
```
|
||||
Understanding the performance implications of different embedding strategies is crucial for building efficient NLP systems that scale to production workloads.
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "memory-analysis", "solution": true}
|
||||
def analyze_embedding_memory():
|
||||
"""📊 Analyze embedding memory requirements and scaling behavior."""
|
||||
def analyze_embedding_memory_scaling():
|
||||
"""📊 Compare embedding memory requirements across different model scales."""
|
||||
print("📊 Analyzing Embedding Memory Requirements...")
|
||||
|
||||
# Vocabulary and embedding dimension scenarios
|
||||
@@ -1051,11 +1044,11 @@ def analyze_embedding_memory():
|
||||
print("• Learned PE adds memory but may improve task-specific performance")
|
||||
print("• Sinusoidal PE saves memory and allows longer sequences")
|
||||
|
||||
analyze_embedding_memory()
|
||||
analyze_embedding_memory_scaling()
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "lookup-performance", "solution": true}
|
||||
def analyze_lookup_performance():
|
||||
"""📊 Analyze embedding lookup performance characteristics."""
|
||||
def analyze_embedding_performance():
|
||||
"""📊 Compare embedding lookup performance across different configurations."""
|
||||
print("\n📊 Analyzing Embedding Lookup Performance...")
|
||||
|
||||
import time
|
||||
@@ -1104,11 +1097,11 @@ def analyze_lookup_performance():
|
||||
print("• Memory bandwidth becomes bottleneck for large embedding dimensions")
|
||||
print("• Cache locality important for repeated token patterns")
|
||||
|
||||
analyze_lookup_performance()
|
||||
analyze_embedding_performance()
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "position-encoding-comparison", "solution": true}
|
||||
def analyze_positional_encoding_trade_offs():
|
||||
"""📊 Compare learned vs sinusoidal positional encodings."""
|
||||
def analyze_positional_encoding_strategies():
|
||||
"""📊 Compare different positional encoding approaches and trade-offs."""
|
||||
print("\n📊 Analyzing Positional Encoding Trade-offs...")
|
||||
|
||||
max_seq_len = 512
|
||||
@@ -1175,13 +1168,13 @@ def analyze_positional_encoding_trade_offs():
|
||||
print(f" - Cannot adapt to task-specific position patterns")
|
||||
print(f" - May be suboptimal for highly position-dependent tasks")
|
||||
|
||||
analyze_positional_encoding_trade_offs()
|
||||
analyze_positional_encoding_strategies()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 10. Module Integration Test
|
||||
## 6. Module Integration Test
|
||||
|
||||
Final validation that our complete embedding system works correctly and integrates with the TinyTorch ecosystem.
|
||||
Let's test our complete embedding system to ensure everything works together correctly.
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": true, "grade_id": "module-test", "locked": true, "points": 20}
|
||||
|
||||
Reference in New Issue
Block a user