mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-03-12 03:03:37 -05:00
Add comprehensive capstone design documentation
- AI Olympics: Competitive leaderboard system for systems engineering - Edge AI Deployment: Hardware deployment focused capstone - Complete evaluation of 7 different capstone approaches - Detailed implementation timeline and technical requirements AI Olympics emerges as best option for student motivation, systems integration, and community building.
This commit is contained in:
Vijay Janapa Reddi
185
capstone-ideas/README.md
Normal file
185
capstone-ideas/README.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# 🎓 TinyTorch Capstone Project Ideas
|
||||
|
||||
## **Background: The Capstone Design Problem**
|
||||
|
||||
**Original Issue**: Module 20 was "TinyGPT Capstone" but students can already build TinyGPT after Module 13 (Transformers). This made:
|
||||
- Modules 14-19 (optimization) feel like "optional extras"
|
||||
- Module 20 anticlimactic ("TinyGPT again?")
|
||||
- No integration of crucial systems engineering skills
|
||||
|
||||
**Solution Requirements**:
|
||||
- Must integrate ALL modules 1-19 (especially optimization modules 14-19)
|
||||
- Must be genuinely exciting and different
|
||||
- Must demonstrate complete ML systems engineering mastery
|
||||
- Must create portfolio-worthy deliverables
|
||||
|
||||
---
|
||||
|
||||
## **🏆 RECOMMENDED: AI Olympics Competition**
|
||||
|
||||
**📁 See: [ai-olympics.md](ai-olympics.md)**
|
||||
|
||||
**Core Concept**: Competitive leaderboard where students optimize TinyTorch models across systems engineering dimensions.
|
||||
|
||||
**Why This is Best**:
|
||||
- ✅ **Natural motivation**: Students want to rank high on leaderboards
|
||||
- ✅ **Systems focus**: Compete on speed, memory, efficiency - not just accuracy
|
||||
- ✅ **Community building**: Creates ongoing engagement and peer interaction
|
||||
- ✅ **Portfolio impact**: "I ranked #3 in TinyTorch AI Olympics" is compelling
|
||||
- ✅ **Forces optimization**: ALL modules 14-19 become essential for competitive performance
|
||||
|
||||
**Competition Categories**:
|
||||
- 🏃♂️ **Speed Demon**: Fastest inference
|
||||
- 💾 **Memory Miser**: Smallest memory footprint
|
||||
- 📱 **Edge Expert**: Best Raspberry Pi performance
|
||||
- 🔋 **Energy Efficient**: Lowest power consumption
|
||||
- 🏆 **TinyMLPerf**: Overall benchmark champion
|
||||
|
||||
---
|
||||
|
||||
## **🛠️ Alternative Ideas Considered**
|
||||
|
||||
### **1. Edge AI Deployment System**
|
||||
**Concept**: Deploy optimized neural networks to actual edge hardware (Raspberry Pi)
|
||||
|
||||
**Pros**:
|
||||
- Integrates all optimization modules (essential for edge constraints)
|
||||
- Creates tangible deliverable ("I run neural networks on a $35 computer")
|
||||
- Teaches real-world deployment challenges
|
||||
|
||||
**Cons**:
|
||||
- Individual project (no community/competition aspect)
|
||||
- Hardware dependencies (students need Pi)
|
||||
- Less motivating than competition
|
||||
|
||||
### **2. Multi-Modal AI Assistant**
|
||||
**Concept**: Combine vision (CNNs) + language (transformers) + optimization for real-time performance
|
||||
|
||||
**Pros**:
|
||||
- Showcases multiple architectures working together
|
||||
- Demonstrates practical AI applications
|
||||
- Requires optimization for real-time performance
|
||||
|
||||
**Cons**:
|
||||
- Complex scope potentially overwhelming
|
||||
- Optimization feels secondary to "getting it working"
|
||||
- Limited portfolio differentiation
|
||||
|
||||
### **3. ML Performance Laboratory**
|
||||
**Concept**: Comprehensive benchmarking suite comparing different ML frameworks
|
||||
|
||||
**Pros**:
|
||||
- Heavy focus on profiling and benchmarking skills
|
||||
- Creates useful tool for community
|
||||
- Deep systems engineering focus
|
||||
|
||||
**Cons**:
|
||||
- More about measurement than optimization
|
||||
- Limited creative expression for students
|
||||
- May feel academic rather than practical
|
||||
|
||||
### **4. Neural Architecture Search**
|
||||
**Concept**: Automated model design and optimization system
|
||||
|
||||
**Pros**:
|
||||
- Cutting-edge research area
|
||||
- Requires sophisticated optimization
|
||||
- Highly technical achievement
|
||||
|
||||
**Cons**:
|
||||
- Very advanced, may be beyond course scope
|
||||
- Optimization becomes means rather than end
|
||||
- Difficult to assess fairly
|
||||
|
||||
### **5. Distributed Training System**
|
||||
**Concept**: Multi-GPU/multi-node training infrastructure
|
||||
|
||||
**Pros**:
|
||||
- Advanced systems engineering skills
|
||||
- High industry relevance
|
||||
- Impressive technical achievement
|
||||
|
||||
**Cons**:
|
||||
- Requires expensive hardware
|
||||
- Complex debugging and setup
|
||||
- May overshadow core ML concepts
|
||||
|
||||
### **6. ML Model Marketplace**
|
||||
**Concept**: Complete system for sharing/deploying/optimizing models (like Hugging Face)
|
||||
|
||||
**Pros**:
|
||||
- Full-stack systems engineering
|
||||
- Practical deployment focus
|
||||
- Creates useful community resource
|
||||
|
||||
**Cons**:
|
||||
- Web development skills needed
|
||||
- Broad scope potentially unfocused
|
||||
- Less emphasis on optimization techniques
|
||||
|
||||
---
|
||||
|
||||
## **📊 Evaluation Criteria**
|
||||
|
||||
| Criteria | AI Olympics | Edge Deployment | Multi-Modal | ML Lab | NAS | Distributed | Marketplace |
|
||||
|----------|-------------|-----------------|-------------|--------|-----|-------------|-------------|
|
||||
| **Integrates All Modules** | ✅✅✅ | ✅✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| **Student Motivation** | ✅✅✅ | ✅ | ✅ | ⚠️ | ⚠️ | ⚠️ | ✅ |
|
||||
| **Portfolio Impact** | ✅✅✅ | ✅✅ | ✅ | ✅ | ✅✅ | ✅✅ | ✅ |
|
||||
| **Systems Engineering Focus** | ✅✅✅ | ✅✅ | ✅ | ✅✅✅ | ✅ | ✅✅✅ | ✅ |
|
||||
| **Implementation Feasibility** | ✅✅ | ✅✅✅ | ✅ | ✅✅ | ⚠️ | ⚠️ | ✅ |
|
||||
| **Community Building** | ✅✅✅ | ⚠️ | ⚠️ | ✅ | ⚠️ | ⚠️ | ✅✅ |
|
||||
| **Scalability** | ✅✅✅ | ✅ | ✅ | ✅ | ⚠️ | ⚠️ | ✅ |
|
||||
|
||||
**Legend**: ✅✅✅ Excellent, ✅✅ Good, ✅ Adequate, ⚠️ Challenging
|
||||
|
||||
---
|
||||
|
||||
## **🎯 Final Recommendation**
|
||||
|
||||
**AI Olympics** emerges as the clear winner because it:
|
||||
|
||||
1. **Maximizes student motivation** through competitive leaderboards
|
||||
2. **Forces integration** of ALL optimization modules (14-19)
|
||||
3. **Creates lasting community** beyond individual course completion
|
||||
4. **Produces compelling portfolio artifacts** (leaderboard rankings)
|
||||
5. **Scales naturally** as more students participate
|
||||
6. **Emphasizes systems engineering** over algorithmic implementation
|
||||
|
||||
### **Implementation Priority**
|
||||
1. **Phase 1**: Design and build leaderboard infrastructure
|
||||
2. **Phase 2**: Create standard benchmark evaluation suite
|
||||
3. **Phase 3**: Deploy beta version with small student cohort
|
||||
4. **Phase 4**: Full launch with all TinyTorch students
|
||||
|
||||
### **Success Metrics**
|
||||
- **Participation Rate**: % of students who submit to multiple categories
|
||||
- **Optimization Depth**: Average number of techniques applied per submission
|
||||
- **Community Engagement**: Forum activity, peer collaboration, ongoing submissions
|
||||
- **Portfolio Impact**: Industry feedback on graduate capabilities
|
||||
|
||||
---
|
||||
|
||||
## **📝 Notes for Implementation**
|
||||
|
||||
### **Technical Requirements**
|
||||
- Automated submission and evaluation pipeline
|
||||
- Standard benchmark datasets and environments
|
||||
- Real-time leaderboard with rich visualizations
|
||||
- Robust measurement and scoring systems
|
||||
|
||||
### **Educational Integration**
|
||||
- Clear rubrics linking competition performance to course grades
|
||||
- Structured optimization process through modules 14-19
|
||||
- Portfolio development guidance and templates
|
||||
- Peer review and collaboration opportunities
|
||||
|
||||
### **Community Features**
|
||||
- Student profiles and achievement tracking
|
||||
- Optimization technique sharing and discussion
|
||||
- Mentorship connections between high performers and struggling students
|
||||
- Industry guest judging and feedback
|
||||
|
||||
---
|
||||
|
||||
**🚀 The AI Olympics transforms TinyTorch from "just another ML course" into a competitive systems engineering community that motivates deep learning, creates lasting engagement, and produces industry-ready graduates.**
|
||||
227
capstone-ideas/ai-olympics.md
Normal file
227
capstone-ideas/ai-olympics.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# 🏅 AI Olympics: TinyTorch Systems Competition Capstone
|
||||
|
||||
## **Core Concept: Compete on Systems Performance, Not Just Accuracy**
|
||||
|
||||
Instead of individual projects, Module 20 becomes a **competitive leaderboard** where students optimize their TinyTorch models across multiple **systems engineering dimensions**.
|
||||
|
||||
### **🎯 Why AI Olympics is Perfect for TinyTorch**
|
||||
|
||||
- **Systems Focus**: Compete on memory, speed, efficiency - not just accuracy
|
||||
- **Real ML Engineering**: Production systems care about performance, not just "does it work"
|
||||
- **Leaderboard Motivation**: Students naturally want to rank high and beat peers
|
||||
- **Portfolio Value**: "I ranked #3 in TinyTorch AI Olympics" is impressive
|
||||
- **Community Building**: Creates ongoing engagement beyond the course
|
||||
|
||||
---
|
||||
|
||||
## **🏆 Competition Categories**
|
||||
|
||||
### **Category 1: Speed Demon** ⚡
|
||||
*"Fastest inference on standard hardware"*
|
||||
- **Metric**: Inferences per second on reference hardware
|
||||
- **Required Skills**: Modules 14-19 optimization techniques
|
||||
- **Constraint**: Must maintain >90% accuracy on test dataset
|
||||
|
||||
### **Category 2: Memory Miser** 💾
|
||||
*"Smallest memory footprint"*
|
||||
- **Metric**: Peak memory usage during inference
|
||||
- **Required Skills**: Quantization, compression, efficient architectures
|
||||
- **Constraint**: Must maintain >85% accuracy on test dataset
|
||||
|
||||
### **Category 3: Edge Expert** 📱
|
||||
*"Best performance on Raspberry Pi"*
|
||||
- **Metric**: Composite score (speed + accuracy + power efficiency)
|
||||
- **Required Skills**: ALL optimization modules for edge constraints
|
||||
- **Constraint**: Must actually run on Pi hardware
|
||||
|
||||
### **Category 4: Energy Efficient** 🔋
|
||||
*"Lowest power consumption"*
|
||||
- **Metric**: Energy per inference (joules/prediction)
|
||||
- **Required Skills**: Model compression, efficient algorithms
|
||||
- **Constraint**: Must maintain competitive accuracy
|
||||
|
||||
### **Category 5: TinyMLPerf** 🏃♂️
|
||||
*"Official MLPerf-style benchmark"*
|
||||
- **Metric**: Standardized benchmark suite performance
|
||||
- **Required Skills**: Complete systems optimization pipeline
|
||||
- **Constraint**: Must pass all benchmark compliance tests
|
||||
|
||||
---
|
||||
|
||||
## **🎮 Competition Structure**
|
||||
|
||||
### **Phase 1: Baseline Submission (Week 1)**
|
||||
- Submit working model from modules 1-13 (CNN, transformer, or multi-modal)
|
||||
- Get baseline scores across all categories
|
||||
- See where you rank on initial leaderboard
|
||||
|
||||
### **Phase 2: Optimization Sprint (Weeks 2-4)**
|
||||
- Apply techniques from modules 14-19 systematically
|
||||
- **Module 14**: Profile and identify bottlenecks
|
||||
- **Module 15**: Implement acceleration techniques
|
||||
- **Module 16**: Add quantization for memory/speed
|
||||
- **Module 17**: Apply compression for size reduction
|
||||
- **Module 18**: Implement caching for inference speed
|
||||
- **Module 19**: Benchmark against production systems
|
||||
|
||||
### **Phase 3: Final Submission & Olympics (Week 5)**
|
||||
- Submit optimized models to all relevant categories
|
||||
- **Live leaderboard updates** as submissions come in
|
||||
- **Victory ceremony** with category winners
|
||||
- **Portfolio artifacts**: Leaderboard rankings + optimization reports
|
||||
|
||||
---
|
||||
|
||||
## **📊 Leaderboard & Scoring System**
|
||||
|
||||
### **Public Leaderboard Features**
|
||||
```
|
||||
🏆 TinyTorch AI Olympics Leaderboard
|
||||
|
||||
Speed Demon Category:
|
||||
1. alice_chen 847.3 inf/sec (95.2% acc) 🥇
|
||||
2. bob_smith 612.7 inf/sec (94.8% acc) 🥈
|
||||
3. carol_wong 588.1 inf/sec (96.1% acc) 🥉
|
||||
|
||||
Memory Miser Category:
|
||||
1. dave_kim 12.4 MB (91.7% acc) 🥇
|
||||
2. eve_patel 15.8 MB (93.2% acc) 🥈
|
||||
3. frank_liu 18.2 MB (89.9% acc) 🥉
|
||||
```
|
||||
|
||||
### **Scoring Methodology**
|
||||
- **Primary Metric**: Category-specific performance (speed, memory, etc.)
|
||||
- **Accuracy Threshold**: Must meet minimum accuracy to qualify
|
||||
- **Tie-Breaker**: Higher accuracy wins ties in primary metric
|
||||
- **Bonus Points**: Novel optimization techniques, exceptional documentation
|
||||
|
||||
### **Awards & Recognition**
|
||||
- **🥇 Category Champions**: Top performer in each category
|
||||
- **🏆 Overall Systems Engineer**: Best combined performance across categories
|
||||
- **🚀 Innovation Award**: Most creative optimization approach
|
||||
- **📚 Teaching Award**: Best documented optimization process
|
||||
|
||||
---
|
||||
|
||||
## **🎯 Required Deliverables**
|
||||
|
||||
### **Competition Submission Package**
|
||||
1. **Optimized Model**: Runnable TinyTorch implementation
|
||||
2. **Performance Report**: Detailed analysis of optimization techniques applied
|
||||
3. **Reproduction Guide**: Clear instructions for others to run your solution
|
||||
4. **Systems Engineering Documentation**: What you learned about ML systems
|
||||
|
||||
### **Portfolio Artifacts Students Get**
|
||||
- **Leaderboard ranking** across multiple categories
|
||||
- **Technical optimization report** demonstrating systems engineering skills
|
||||
- **Benchmark results** comparing their work to industry standards
|
||||
- **Peer recognition** from competitive performance
|
||||
|
||||
---
|
||||
|
||||
## **🔧 Technical Infrastructure Needed**
|
||||
|
||||
### **Leaderboard System**
|
||||
- Automated submission processing
|
||||
- Standard evaluation environment
|
||||
- Real-time ranking updates
|
||||
- Historical performance tracking
|
||||
|
||||
### **Benchmark Suite**
|
||||
- Reference datasets for each category
|
||||
- Standard hardware for testing
|
||||
- Automated compliance checking
|
||||
- Performance measurement tools
|
||||
|
||||
### **Submission Portal**
|
||||
- Code upload and validation
|
||||
- Automatic testing pipeline
|
||||
- Results processing and ranking
|
||||
- Student dashboard with progress
|
||||
|
||||
---
|
||||
|
||||
## **📈 Why This Beats Individual Projects**
|
||||
|
||||
### **Individual Project Problems:**
|
||||
- ❌ No motivation to optimize beyond "it works"
|
||||
- ❌ Hard to compare student achievements
|
||||
- ❌ No ongoing engagement after submission
|
||||
- ❌ Limited portfolio impact
|
||||
|
||||
### **AI Olympics Advantages:**
|
||||
- ✅ **Natural optimization motivation**: Students want to rank higher
|
||||
- ✅ **Clear performance comparison**: Leaderboard shows relative achievement
|
||||
- ✅ **Ongoing engagement**: Leaderboard creates lasting community
|
||||
- ✅ **Strong portfolio impact**: "I ranked #2 in Memory Efficiency" is compelling
|
||||
|
||||
### **Systems Engineering Focus:**
|
||||
- Forces students to care about **ALL** optimization dimensions
|
||||
- Makes modules 14-19 essential for competitive performance
|
||||
- Teaches that "getting it working" is only the beginning
|
||||
- Demonstrates real-world ML engineering priorities
|
||||
|
||||
---
|
||||
|
||||
## **🚀 Implementation Timeline**
|
||||
|
||||
### **Phase 1: Core Infrastructure (4 weeks)**
|
||||
- Build leaderboard system
|
||||
- Create benchmark evaluation suite
|
||||
- Set up automated testing pipeline
|
||||
- Design submission portal
|
||||
|
||||
### **Phase 2: Beta Testing (2 weeks)**
|
||||
- Test with small group of students
|
||||
- Refine scoring methodology
|
||||
- Fix technical issues
|
||||
- Gather feedback and iterate
|
||||
|
||||
### **Phase 3: Full Launch (Ongoing)**
|
||||
- Deploy for all TinyTorch students
|
||||
- Monitor and maintain leaderboard
|
||||
- Regular benchmark updates
|
||||
- Community management and awards
|
||||
|
||||
---
|
||||
|
||||
## **🎓 Educational Impact**
|
||||
|
||||
### **Learning Outcomes**
|
||||
Students learn that ML engineering is about:
|
||||
- **Systems performance**, not just algorithmic correctness
|
||||
- **Trade-offs** between speed, memory, accuracy, and power
|
||||
- **Optimization techniques** for real-world constraints
|
||||
- **Benchmarking and measurement** for objective evaluation
|
||||
- **Competition and collaboration** in technical communities
|
||||
|
||||
### **Career Preparation**
|
||||
Students graduate with:
|
||||
- **Demonstrable systems optimization skills**
|
||||
- **Portfolio evidence of competitive performance**
|
||||
- **Experience with ML engineering trade-offs**
|
||||
- **Understanding of production ML constraints**
|
||||
- **Community connections** with other systems engineers
|
||||
|
||||
---
|
||||
|
||||
## **💡 Future Extensions**
|
||||
|
||||
### **Multi-Semester Competitions**
|
||||
- New benchmark challenges each semester
|
||||
- Evolving leaderboards with increasing difficulty
|
||||
- Alumni participation and mentorship
|
||||
|
||||
### **Industry Integration**
|
||||
- Company-sponsored benchmark challenges
|
||||
- Internship opportunities for top performers
|
||||
- Guest judging from ML systems engineers
|
||||
|
||||
### **Research Integration**
|
||||
- Novel optimization techniques become research contributions
|
||||
- Student innovations feed back into TinyTorch framework
|
||||
- Academic publications from exceptional submissions
|
||||
|
||||
---
|
||||
|
||||
**🎯 CONCLUSION: AI Olympics transforms Module 20 from "individual project" to "competitive systems engineering challenge" that motivates optimization, builds community, and produces compelling portfolio artifacts.**
|
||||
Reference in New Issue
Block a user