Add MLPerf methodology to Module 19 and rebrand Module 20 as TinyMLPerf

Module 19 Updates:
- Added Section 4.4: MLPerf Principles & Methodology
- Explains MLPerf framework (industry-standard benchmarking)
- Teaches Closed vs Open Division concepts
- Covers reproducibility and standardization requirements
- References TinyMLPerf for embedded systems
- Prepares students for professional ML benchmarking

Module 20 Updates:
- Rebranded as TinyMLPerf Competition (from generic competition)
- Emphasizes MLPerf Closed Division rules throughout
- Section 1: TinyMLPerf rules and what is/isnt allowed
- Section 2: Official baseline following MLPerf standards
- Section 3: Complete workflow following MLPerf methodology
- Section 4: Submission template with MLPerf compliance

Pedagogical Improvement:
- Grounds capstone in real-world MLPerf methodology
- Students learn industry-standard benchmarking practices
- Competition has professional credibility
- Clear rules ensure fair comparison
- Reproducibility and documentation emphasized
This commit is contained in:
Vijay Janapa Reddi
2025-11-06 23:34:00 -05:00
parent 803ac39b07
commit 5b93f4e711
3 changed files with 421 additions and 675 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -1936,6 +1936,91 @@ def test_unit_optimization_comparison():
test_unit_optimization_comparison()
# %% [markdown]
"""
## 4.4 MLPerf Principles - Industry-Standard Benchmarking
Before we dive into optimization strategies, let's learn from **MLPerf** - the industry-standard ML benchmarking framework. Understanding MLPerf principles will ground your capstone competition in professional ML systems evaluation.
### What is MLPerf?
MLPerf is the industry-standard benchmark suite for measuring ML system performance. Think of it as the "Olympics" of ML systems, but with rigorous scientific methodology:
- **Created by:** MLCommons (Google, NVIDIA, Intel, universities)
- **Used by:** All major ML hardware/software companies
- **Purpose:** Fair, reproducible comparison of ML systems
- **Impact:** Drives billions in hardware/software decisions
### Core MLPerf Principles
**1. Reproducibility**
- Exact hardware specifications reported
- Software versions documented
- Random seeds controlled
- Multiple runs required for statistical validity
**2. Standardization**
- Fixed model architectures (everyone runs the same models)
- Fixed datasets (same training/test data)
- Fixed quality targets (must achieve X% accuracy)
- Fair comparison (apples-to-apples)
**3. Divisions for Different Goals**
MLPerf has TWO main divisions:
**🔒 Closed Division** (Strict Rules):
- Use provided model architectures exactly
- Use provided datasets exactly
- Can optimize: training algorithms, hardware, software stack
- **Goal:** Fair comparison of SYSTEMS (not algorithms)
- Example: "Which GPU trains ResNet-50 fastest?"
**🔓 Open Division** (Flexible Rules):
- Modify model architectures
- Use different datasets
- Novel algorithms allowed
- **Goal:** Show innovation and new approaches
- Example: "New pruning technique achieves 10x speedup!"
**Why Two Divisions?**
- Closed: Answers "What's the best hardware/software for X?"
- Open: Answers "What's the best algorithm/innovation for Y?"
### MLPerf Inference Benchmarks
MLPerf Inference (what we care about) measures:
- **Latency:** Single-stream inference time
- **Throughput:** Offline batch processing speed
- **Accuracy:** Must meet quality targets
- **Power:** Energy efficiency (advanced)
Common scenarios:
- **Server:** Datacenter deployment (high throughput)
- **Edge:** On-device inference (low latency, low power)
- **Mobile:** Smartphone deployment (tiny models)
### TinyMLPerf - MLPerf for Tiny Systems
TinyMLPerf is MLPerf for embedded/edge devices:
- Models <1MB
- Latency <100ms
- Power <10mW
- Real deployment constraints
**This is what inspires your capstone!**
### Key Takeaways for Your Competition
1. **Reproducibility Matters:** Document everything
2. **Fair Comparison:** Same baseline for everyone
3. **Multiple Metrics:** Not just accuracy - latency, memory, energy
4. **Real Constraints:** Optimize for actual deployment scenarios
5. **Closed vs Open:** Understand the rules of your competition
**In Module 20**, you'll participate in **TinyMLPerf-style competition** following these principles!
"""
# %% [markdown]
"""
## 4.5 Combination Strategies - Preparing for TorchPerf Olympics

12
tinytorch/_modidx.py generated
View File

@@ -65,6 +65,18 @@ d = { 'settings': { 'branch': 'main',
'tinytorch/benchmarking/benchmark.py'),
'tinytorch.benchmarking.benchmark.test_unit_tinymlperf': ( '19_benchmarking/benchmarking_dev.html#test_unit_tinymlperf',
'tinytorch/benchmarking/benchmark.py')},
'tinytorch.competition.submit': { 'tinytorch.competition.submit.generate_baseline': ( '20_competition/competition_dev.html#generate_baseline',
'tinytorch/competition/submit.py'),
'tinytorch.competition.submit.generate_submission': ( '20_competition/competition_dev.html#generate_submission',
'tinytorch/competition/submit.py'),
'tinytorch.competition.submit.load_baseline_model': ( '20_competition/competition_dev.html#load_baseline_model',
'tinytorch/competition/submit.py'),
'tinytorch.competition.submit.optimize_for_competition': ( '20_competition/competition_dev.html#optimize_for_competition',
'tinytorch/competition/submit.py'),
'tinytorch.competition.submit.validate_installation': ( '20_competition/competition_dev.html#validate_installation',
'tinytorch/competition/submit.py'),
'tinytorch.competition.submit.worked_example_optimization': ( '20_competition/competition_dev.html#worked_example_optimization',
'tinytorch/competition/submit.py')},
'tinytorch.core.activations': { 'tinytorch.core.activations.GELU': ( '02_activations/activations_dev.html#gelu',
'tinytorch/core/activations.py'),
'tinytorch.core.activations.GELU.__call__': ( '02_activations/activations_dev.html#gelu.__call__',