mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-08-01 18:59:41 -05:00
refactor(modules): Reorganize optimization tier structure (14-19)
Module renaming and reordering: - 15_profiling → 14_profiling (now first in optimization tier) - 14_kvcaching → 15_memoization (renamed to emphasize pattern) - 17_quantization → 16_quantization - 18_compression → 17_compression - 16_acceleration → 18_acceleration (moved after compression) - 19_benchmarking (unchanged) All module metadata updated (numbers, prerequisites, connection maps)
This commit is contained in:
@@ -0,0 +1,258 @@
|
||||
# Module 15 Profiling - Restructuring Summary
|
||||
|
||||
## Goal
|
||||
Completely restructure Module 15 (Profiling) to match Module 12's clean, consolidated class structure.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. **Consolidated Profiler Class** (Lines 235-737)
|
||||
|
||||
**Before**: Split structure with methods defined outside class + monkey-patching
|
||||
**After**: Single complete class with all methods inside
|
||||
|
||||
#### All Methods Now Inside Profiler Class:
|
||||
- `__init__()` - Initialize profiler state
|
||||
- `count_parameters()` - Count model parameters
|
||||
- `count_flops()` - Count FLOPs for operations
|
||||
- `measure_memory()` - Track memory usage
|
||||
- `measure_latency()` - Measure inference latency
|
||||
- `profile_layer()` - Comprehensive layer profiling
|
||||
- `profile_forward_pass()` - Complete forward pass analysis
|
||||
- `profile_backward_pass()` - Training pass analysis
|
||||
|
||||
**Educational Pattern Preserved**:
|
||||
- TODO/APPROACH/HINTS/EXAMPLE in every method
|
||||
- BEGIN SOLUTION / END SOLUTION blocks
|
||||
- Clear scaffolding for students
|
||||
|
||||
### 2. **Removed Monkey-Patching**
|
||||
|
||||
**Deleted these anti-patterns**:
|
||||
- Line 293: `Profiler.count_parameters = count_parameters`
|
||||
- Line 456: `Profiler.count_flops = count_flops`
|
||||
- Line 612: `Profiler.measure_memory = measure_memory`
|
||||
- Line 772: `Profiler.profile_layer = profile_layer`
|
||||
|
||||
**Benefit**: Clean, predictable class structure. IDE-friendly autocomplete.
|
||||
|
||||
### 3. **Removed Duplicate ProfilerComplete Class**
|
||||
|
||||
**Deleted**: Lines 1523-1627 (ProfilerComplete duplicate)
|
||||
|
||||
**Reason**: Redundant. Single Profiler class serves both learning and export purposes.
|
||||
|
||||
### 4. **Added Module 14 Connection** (Lines 106-131)
|
||||
|
||||
New section connecting profiling to KV caching optimization:
|
||||
|
||||
```
|
||||
### 🔗 From Optimization to Discovery: Connecting Module 14
|
||||
|
||||
**In Module 14**, you implemented KV caching and saw 10-15x speedup.
|
||||
**In Module 15**, you'll learn HOW to discover such optimization opportunities.
|
||||
|
||||
**The Real ML Engineering Workflow**:
|
||||
Step 1: Measure (This Module!) → Step 2: Analyze → Step 3: Optimize (Module 14) → Step 4: Validate
|
||||
```
|
||||
|
||||
**Educational Value**: Shows students the complete optimization discovery cycle.
|
||||
|
||||
### 5. **Fixed Diagnostic Issues**
|
||||
|
||||
All unused variable warnings resolved by prefixing with `_`:
|
||||
- Line 342: `_dummy_input` (unused but kept for interface consistency)
|
||||
- Line 359: `_batch_size` (unused in simplified FLOP calculation)
|
||||
- Line 424: `_baseline_memory` (unused but kept for completeness)
|
||||
- Line 449: `_current_memory` (unused, peak_memory used instead)
|
||||
|
||||
**Result**: Zero warnings or errors from pylint.
|
||||
|
||||
### 6. **Preserved Educational Content**
|
||||
|
||||
**All retained**:
|
||||
- NBGrader metadata in every cell
|
||||
- TODO/APPROACH/HINTS/EXAMPLE scaffolding
|
||||
- BEGIN SOLUTION / END SOLUTION markers
|
||||
- Markdown explanations between functions
|
||||
- ASCII diagrams for visualization
|
||||
- Unit tests (`test_unit_X()`)
|
||||
- Integration test (`test_module()`)
|
||||
- Systems analysis functions
|
||||
- Module summary and reflection questions
|
||||
|
||||
## File Structure After Restructuring
|
||||
|
||||
```
|
||||
Module 15: Profiling (1609 lines)
|
||||
├── Part 1: Introduction (Lines 1-131)
|
||||
│ └── Added Module 14 connection
|
||||
├── Part 2: Foundations (Lines 133-209)
|
||||
├── Part 3: Implementation (Lines 211-737)
|
||||
│ └── CONSOLIDATED Profiler Class
|
||||
│ ├── __init__()
|
||||
│ ├── count_parameters()
|
||||
│ ├── count_flops()
|
||||
│ ├── measure_memory()
|
||||
│ ├── measure_latency()
|
||||
│ ├── profile_layer()
|
||||
│ ├── profile_forward_pass()
|
||||
│ └── profile_backward_pass()
|
||||
├── Part 4: Unit Tests (Lines 739-1170)
|
||||
│ ├── test_unit_parameter_counting()
|
||||
│ ├── test_unit_flop_counting()
|
||||
│ ├── test_unit_memory_measurement()
|
||||
│ ├── test_unit_latency_measurement()
|
||||
│ └── test_unit_advanced_profiling()
|
||||
├── Part 5: Systems Analysis (Lines 1172-1436)
|
||||
│ ├── analyze_model_scaling()
|
||||
│ ├── analyze_batch_size_effects()
|
||||
│ ├── benchmark_operation_efficiency()
|
||||
│ └── analyze_profiling_overhead()
|
||||
├── Part 6: Module Integration (Lines 1438-1547)
|
||||
│ ├── test_module()
|
||||
│ └── if __name__ == "__main__"
|
||||
└── Part 7: Summary (Lines 1549-1609)
|
||||
├── Reflection questions
|
||||
└── Module summary
|
||||
```
|
||||
|
||||
## Comparison with Module 12
|
||||
|
||||
### Module 12 Pattern (Reference):
|
||||
```python
|
||||
class MultiHeadAttention:
|
||||
def __init__(self, embed_dim, num_heads):
|
||||
"""TODO/APPROACH/HINTS/EXAMPLE"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
|
||||
def forward(self, x, mask=None):
|
||||
"""TODO/APPROACH/HINTS/EXAMPLE"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
|
||||
def parameters(self):
|
||||
"""TODO/APPROACH"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
```
|
||||
|
||||
### Module 15 Pattern (After Restructuring):
|
||||
```python
|
||||
class Profiler:
|
||||
def __init__(self):
|
||||
"""TODO/APPROACH/HINTS/EXAMPLE"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
|
||||
def count_parameters(self, model):
|
||||
"""TODO/APPROACH/HINTS/EXAMPLE"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
|
||||
def count_flops(self, model, input_shape):
|
||||
"""TODO/APPROACH/HINTS/EXAMPLE"""
|
||||
### BEGIN SOLUTION
|
||||
# Implementation
|
||||
### END SOLUTION
|
||||
|
||||
# ... 5 more methods following same pattern
|
||||
```
|
||||
|
||||
**Perfect alignment!** Both modules now follow the same clean structure.
|
||||
|
||||
## Validation Results
|
||||
|
||||
### Test Results
|
||||
```bash
|
||||
$ python modules/source/15_profiling/profiling_dev.py
|
||||
|
||||
✅ Unit Test: Parameter Counting... PASSED
|
||||
✅ Unit Test: FLOP Counting... PASSED
|
||||
✅ Unit Test: Memory Measurement... PASSED
|
||||
✅ Unit Test: Latency Measurement... PASSED
|
||||
✅ Unit Test: Advanced Profiling... PASSED
|
||||
📊 Systems Analysis... COMPLETED
|
||||
🧪 Module Integration Test... ALL TESTS PASSED
|
||||
```
|
||||
|
||||
### Code Quality
|
||||
```bash
|
||||
$ pylint modules/source/15_profiling/profiling_dev.py
|
||||
|
||||
No warnings or errors!
|
||||
```
|
||||
|
||||
### Functionality Verification
|
||||
- ✅ `profiler = Profiler()` works
|
||||
- ✅ All methods callable: `profiler.count_parameters(model)`
|
||||
- ✅ No import errors
|
||||
- ✅ All tests pass
|
||||
- ✅ Systems analysis runs successfully
|
||||
|
||||
## Benefits of Restructuring
|
||||
|
||||
### For Students:
|
||||
1. **Clear Mental Model**: Single class, all methods inside
|
||||
2. **IDE-Friendly**: Autocomplete works immediately
|
||||
3. **Debugger-Friendly**: Consistent class structure
|
||||
4. **No Confusion**: No mysterious monkey-patching
|
||||
|
||||
### For Instructors:
|
||||
1. **Maintainable**: Easy to update methods
|
||||
2. **Consistent**: Matches Module 12 pattern
|
||||
3. **Professional**: Production-like code structure
|
||||
4. **Testable**: Clear unit testing flow
|
||||
|
||||
### For Production:
|
||||
1. **Clean Architecture**: No runtime class modification
|
||||
2. **Type-Safe**: Static analysis tools work correctly
|
||||
3. **Documentation**: Methods show up in help/docstrings
|
||||
4. **Predictable**: No surprising behavior
|
||||
|
||||
## Line Count Reduction
|
||||
|
||||
**Before**: 1659 lines (with duplicate ProfilerComplete class and monkey-patching)
|
||||
**After**: 1609 lines (consolidated, no duplication)
|
||||
**Reduction**: 50 lines (~3% cleaner)
|
||||
|
||||
## Key Achievements
|
||||
|
||||
1. ✅ **Single Profiler Class** - All methods inside, no external definitions
|
||||
2. ✅ **No Monkey-Patching** - Clean, predictable class structure
|
||||
3. ✅ **No Duplicates** - Removed redundant ProfilerComplete class
|
||||
4. ✅ **Module 14 Connection** - Shows optimization discovery workflow
|
||||
5. ✅ **Zero Diagnostics** - No warnings or errors
|
||||
6. ✅ **All Tests Pass** - Full functionality preserved
|
||||
7. ✅ **Educational Content Preserved** - All scaffolding and explanations intact
|
||||
8. ✅ **Module 12 Compliant** - Perfect structural alignment
|
||||
|
||||
## Migration Path for Other Modules
|
||||
|
||||
This restructuring provides a template for cleaning up other modules:
|
||||
|
||||
1. **Identify** methods defined outside classes
|
||||
2. **Move** them inside the class definition
|
||||
3. **Remove** monkey-patching assignments
|
||||
4. **Delete** duplicate/redundant classes
|
||||
5. **Add** module connections for context
|
||||
6. **Fix** diagnostic issues (unused variables)
|
||||
7. **Validate** all tests still pass
|
||||
8. **Verify** functionality unchanged
|
||||
|
||||
## Conclusion
|
||||
|
||||
Module 15 has been successfully restructured to match Module 12's gold standard:
|
||||
- **Single consolidated Profiler class** with all methods inside
|
||||
- **No monkey-patching** or runtime class modification
|
||||
- **No duplicates** or redundant code
|
||||
- **Clear module connections** showing optimization workflow
|
||||
- **Zero diagnostic issues** with clean, professional code
|
||||
- **All educational content preserved** for student learning
|
||||
|
||||
The module is now production-ready, maintainable, and provides an excellent example of clean class-based design for educational purposes.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -168,7 +168,7 @@ import platform
|
||||
from contextlib import contextmanager
|
||||
import warnings
|
||||
|
||||
# Import Profiler from Module 15 for measurement reuse
|
||||
# Import Profiler from Module 14 for measurement reuse
|
||||
from tinytorch.profiling.profiler import Profiler
|
||||
|
||||
# %%
|
||||
@@ -196,7 +196,7 @@ We'll build a comprehensive benchmarking system that handles statistical analysi
|
||||
|
||||
The architecture follows a hierarchical design:
|
||||
```
|
||||
Profiler (Module 15) ← Base measurement tools
|
||||
Profiler (Module 14) ← Base measurement tools
|
||||
↓
|
||||
BenchmarkResult ← Statistical container for measurements
|
||||
↓
|
||||
@@ -207,7 +207,7 @@ BenchmarkSuite ← Multi-metric comprehensive evaluation
|
||||
TinyMLPerf ← Standardized industry-style benchmarks
|
||||
```
|
||||
|
||||
**Key Architectural Decision**: The `Benchmark` class reuses `Profiler` from Module 15 for individual model measurements, then adds statistical comparison across multiple models. This demonstrates proper systems architecture - build once, reuse everywhere!
|
||||
**Key Architectural Decision**: The `Benchmark` class reuses `Profiler` from Module 14 for individual model measurements, then adds statistical comparison across multiple models. This demonstrates proper systems architecture - build once, reuse everywhere!
|
||||
|
||||
Each level adds capability while maintaining statistical rigor at the foundation.
|
||||
"""
|
||||
@@ -552,7 +552,7 @@ class Benchmark:
|
||||
self.measurement_runs = measurement_runs
|
||||
self.results = {}
|
||||
|
||||
# Use Profiler from Module 15 for measurements
|
||||
# Use Profiler from Module 14 for measurements
|
||||
self.profiler = Profiler()
|
||||
|
||||
# System information for metadata
|
||||
|
||||
Reference in New Issue
Block a user