mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-04-27 21:47:51 -05:00
Add MLPerf methodology to Module 19 and rebrand Module 20 as TinyMLPerf
Module 19 Updates: - Added Section 4.4: MLPerf Principles & Methodology - Explains MLPerf framework (industry-standard benchmarking) - Teaches Closed vs Open Division concepts - Covers reproducibility and standardization requirements - References TinyMLPerf for embedded systems - Prepares students for professional ML benchmarking Module 20 Updates: - Rebranded as TinyMLPerf Competition (from generic competition) - Emphasizes MLPerf Closed Division rules throughout - Section 1: TinyMLPerf rules and what is/isnt allowed - Section 2: Official baseline following MLPerf standards - Section 3: Complete workflow following MLPerf methodology - Section 4: Submission template with MLPerf compliance Pedagogical Improvement: - Grounds capstone in real-world MLPerf methodology - Students learn industry-standard benchmarking practices - Competition has professional credibility - Clear rules ensure fair comparison - Reproducibility and documentation emphasized
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -1936,6 +1936,91 @@ def test_unit_optimization_comparison():
|
||||
|
||||
test_unit_optimization_comparison()
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 4.4 MLPerf Principles - Industry-Standard Benchmarking
|
||||
|
||||
Before we dive into optimization strategies, let's learn from **MLPerf** - the industry-standard ML benchmarking framework. Understanding MLPerf principles will ground your capstone competition in professional ML systems evaluation.
|
||||
|
||||
### What is MLPerf?
|
||||
|
||||
MLPerf is the industry-standard benchmark suite for measuring ML system performance. Think of it as the "Olympics" of ML systems, but with rigorous scientific methodology:
|
||||
|
||||
- **Created by:** MLCommons (Google, NVIDIA, Intel, universities)
|
||||
- **Used by:** All major ML hardware/software companies
|
||||
- **Purpose:** Fair, reproducible comparison of ML systems
|
||||
- **Impact:** Drives billions in hardware/software decisions
|
||||
|
||||
### Core MLPerf Principles
|
||||
|
||||
**1. Reproducibility**
|
||||
- Exact hardware specifications reported
|
||||
- Software versions documented
|
||||
- Random seeds controlled
|
||||
- Multiple runs required for statistical validity
|
||||
|
||||
**2. Standardization**
|
||||
- Fixed model architectures (everyone runs the same models)
|
||||
- Fixed datasets (same training/test data)
|
||||
- Fixed quality targets (must achieve X% accuracy)
|
||||
- Fair comparison (apples-to-apples)
|
||||
|
||||
**3. Divisions for Different Goals**
|
||||
|
||||
MLPerf has TWO main divisions:
|
||||
|
||||
**🔒 Closed Division** (Strict Rules):
|
||||
- Use provided model architectures exactly
|
||||
- Use provided datasets exactly
|
||||
- Can optimize: training algorithms, hardware, software stack
|
||||
- **Goal:** Fair comparison of SYSTEMS (not algorithms)
|
||||
- Example: "Which GPU trains ResNet-50 fastest?"
|
||||
|
||||
**🔓 Open Division** (Flexible Rules):
|
||||
- Modify model architectures
|
||||
- Use different datasets
|
||||
- Novel algorithms allowed
|
||||
- **Goal:** Show innovation and new approaches
|
||||
- Example: "New pruning technique achieves 10x speedup!"
|
||||
|
||||
**Why Two Divisions?**
|
||||
- Closed: Answers "What's the best hardware/software for X?"
|
||||
- Open: Answers "What's the best algorithm/innovation for Y?"
|
||||
|
||||
### MLPerf Inference Benchmarks
|
||||
|
||||
MLPerf Inference (what we care about) measures:
|
||||
- **Latency:** Single-stream inference time
|
||||
- **Throughput:** Offline batch processing speed
|
||||
- **Accuracy:** Must meet quality targets
|
||||
- **Power:** Energy efficiency (advanced)
|
||||
|
||||
Common scenarios:
|
||||
- **Server:** Datacenter deployment (high throughput)
|
||||
- **Edge:** On-device inference (low latency, low power)
|
||||
- **Mobile:** Smartphone deployment (tiny models)
|
||||
|
||||
### TinyMLPerf - MLPerf for Tiny Systems
|
||||
|
||||
TinyMLPerf is MLPerf for embedded/edge devices:
|
||||
- Models <1MB
|
||||
- Latency <100ms
|
||||
- Power <10mW
|
||||
- Real deployment constraints
|
||||
|
||||
**This is what inspires your capstone!**
|
||||
|
||||
### Key Takeaways for Your Competition
|
||||
|
||||
1. **Reproducibility Matters:** Document everything
|
||||
2. **Fair Comparison:** Same baseline for everyone
|
||||
3. **Multiple Metrics:** Not just accuracy - latency, memory, energy
|
||||
4. **Real Constraints:** Optimize for actual deployment scenarios
|
||||
5. **Closed vs Open:** Understand the rules of your competition
|
||||
|
||||
**In Module 20**, you'll participate in **TinyMLPerf-style competition** following these principles!
|
||||
"""
|
||||
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 4.5 Combination Strategies - Preparing for TorchPerf Olympics
|
||||
|
||||
12
tinytorch/_modidx.py
generated
12
tinytorch/_modidx.py
generated
@@ -65,6 +65,18 @@ d = { 'settings': { 'branch': 'main',
|
||||
'tinytorch/benchmarking/benchmark.py'),
|
||||
'tinytorch.benchmarking.benchmark.test_unit_tinymlperf': ( '19_benchmarking/benchmarking_dev.html#test_unit_tinymlperf',
|
||||
'tinytorch/benchmarking/benchmark.py')},
|
||||
'tinytorch.competition.submit': { 'tinytorch.competition.submit.generate_baseline': ( '20_competition/competition_dev.html#generate_baseline',
|
||||
'tinytorch/competition/submit.py'),
|
||||
'tinytorch.competition.submit.generate_submission': ( '20_competition/competition_dev.html#generate_submission',
|
||||
'tinytorch/competition/submit.py'),
|
||||
'tinytorch.competition.submit.load_baseline_model': ( '20_competition/competition_dev.html#load_baseline_model',
|
||||
'tinytorch/competition/submit.py'),
|
||||
'tinytorch.competition.submit.optimize_for_competition': ( '20_competition/competition_dev.html#optimize_for_competition',
|
||||
'tinytorch/competition/submit.py'),
|
||||
'tinytorch.competition.submit.validate_installation': ( '20_competition/competition_dev.html#validate_installation',
|
||||
'tinytorch/competition/submit.py'),
|
||||
'tinytorch.competition.submit.worked_example_optimization': ( '20_competition/competition_dev.html#worked_example_optimization',
|
||||
'tinytorch/competition/submit.py')},
|
||||
'tinytorch.core.activations': { 'tinytorch.core.activations.GELU': ( '02_activations/activations_dev.html#gelu',
|
||||
'tinytorch/core/activations.py'),
|
||||
'tinytorch.core.activations.GELU.__call__': ( '02_activations/activations_dev.html#gelu.__call__',
|
||||
|
||||
Reference in New Issue
Block a user