From be3f3503a1fdfa866dd4cd6ff25fa93d195aaa09 Mon Sep 17 00:00:00 2001 From: Vijay Janapa Reddi Date: Wed, 16 Jul 2025 01:44:49 -0400 Subject: [PATCH] Standardize all 14 module READMEs with consistent structure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit โœจ Complete standardization of all TinyTorch module READMEs: ๐Ÿ“Š **Module Info**: Consistent difficulty, time, prerequisites, next steps ๐ŸŽฏ **Learning Objectives**: Clear, measurable, action-oriented outcomes ๐Ÿง  **Pedagogical Framework**: Build โ†’ Use โ†’ [Context-specific verb] ๐Ÿ“š **What You'll Build**: Concrete code examples and implementations ๐Ÿš€ **Getting Started**: Prerequisites check + development workflow ๐Ÿงช **Testing**: Comprehensive test coverage + inline feedback ๐ŸŽฏ **Key Concepts**: Real-world applications + technical foundations ๐ŸŽ‰ **Ready to Build**: Motivational + grid cards for all modules โœ… All 14 modules now follow identical structure: - 01_setup: Foundation workflow mastery - 02_tensor: Core data structures - 03_activations: Neural network fundamentals - 04_layers: Building blocks - 05_networks: Architecture design - 06_cnn: Computer vision foundations - 07_dataloader: Data pipeline engineering - 08_autograd: Automatic differentiation - 09_optimizers: Learning algorithms - 10_training: End-to-end orchestration - 11_compression: Model optimization - 12_kernels: Performance optimization - 13_benchmarking: Systematic evaluation - 14_mlops: Production deployment (capstone) ๐ŸŽ“ **Student Experience**: Predictable navigation, clear expectations, motivational flow ๐Ÿ‘จโ€๐Ÿซ **Instructor Experience**: Professional consistency, easy maintenance, coherent course This establishes the single source of truth that will automatically convert to clean website chapters via book/convert_readmes.py --- modules/source/01_setup/README.md | 241 ++++----- modules/source/03_activations/README.md | 330 +++++------- modules/source/04_layers/README.md | 297 ++++++----- modules/source/05_networks/README.md | 377 ++++++-------- modules/source/06_cnn/README.md | 237 +++++++-- modules/source/07_dataloader/README.md | 455 +++++++---------- modules/source/08_autograd/README.md | 393 ++++++-------- modules/source/09_optimizers/README.md | 323 ++++++------ modules/source/10_training/README.md | 413 +++++++-------- modules/source/11_compression/README.md | 413 +++++++++------ modules/source/12_kernels/README.md | 342 +++++++++---- modules/source/13_benchmarking/README.md | 353 ++++++++----- modules/source/14_mlops/README.md | 625 +++++++++++++---------- 13 files changed, 2593 insertions(+), 2206 deletions(-) diff --git a/modules/source/01_setup/README.md b/modules/source/01_setup/README.md index 05f3986f..777e2e1a 100644 --- a/modules/source/01_setup/README.md +++ b/modules/source/01_setup/README.md @@ -6,154 +6,157 @@ - **Prerequisites**: Basic Python knowledge - **Next Steps**: Tensor module -Welcome to TinyTorch! This is your first module in the Machine Learning Systems course. +Welcome to TinyTorch! This foundational module introduces the complete development workflow that powers every subsequent module. You'll master the nbdev notebook-to-Python workflow, implement your first TinyTorch components, and establish the coding patterns used throughout the entire course. ## ๐ŸŽฏ Learning Objectives -- Understand the nbdev notebook-to-Python workflow -- Write your first TinyTorch code with `#| export` directives -- Implement system information collection and developer profiles -- Run tests and use the CLI tools -- Get comfortable with the development rhythm +By the end of this module, you will be able to: -## ๐Ÿง  Overview +- **Master the nbdev workflow**: Write code with `#| export` directives and understand the notebook-to-package pipeline +- **Implement system utilities**: Build functions for system information collection and developer profiles +- **Use TinyTorch CLI tools**: Run tests, sync modules, and check development progress +- **Write production-ready code**: Follow professional patterns for error handling, testing, and documentation +- **Establish development rhythm**: Understand the build โ†’ test โ†’ iterate cycle that drives all TinyTorch modules -The setup module teaches you the complete TinyTorch development workflow while introducing fundamental programming concepts. You'll learn to write code with NBDev directives, implement classes and functions, and understand the module-to-package export system. +## ๐Ÿง  Build โ†’ Use โ†’ Master + +This module follows TinyTorch's **Build โ†’ Use โ†’ Master** framework: + +1. **Build**: Implement core utilities (hello functions, system info, developer profiles) +2. **Use**: Apply these components in real development workflows and testing scenarios +3. **Master**: Understand how this foundation supports all advanced TinyTorch modules and establish professional development habits ## ๐Ÿ“š What You'll Build -### 1. Basic Functions -- `hello_tinytorch()` - Display ASCII art and welcome message -- `add_numbers()` - Basic arithmetic (foundation of ML operations) - -### 2. System Information Class -- `SystemInfo` - Collect and display Python version, platform, and machine info -- Compatibility checking for minimum requirements - -### 3. Developer Profile Class -- `DeveloperProfile` - Personalized developer information and signatures -- ASCII art customization and file loading -- Professional code attribution system - -## Usage - -### Python Script +### Core Functions ```python -from setup_dev import hello_tinytorch, add_numbers, SystemInfo, DeveloperProfile +# Welcome and basic operations +hello_tinytorch() # ASCII art and course introduction +add_numbers(2, 3) # Foundation arithmetic operations -# Display welcome message -hello_tinytorch() - -# Basic arithmetic -result = add_numbers(2, 3) - -# System information +# System information utilities info = SystemInfo() -print(f"System: {info}") +print(f"Platform: {info.platform}") print(f"Compatible: {info.is_compatible()}") -# Developer profile -profile = DeveloperProfile() +# Developer profile management +profile = DeveloperProfile(name="Your Name", affiliation="University") print(profile.get_full_profile()) ``` -### Jupyter Notebook -Open `setup_dev.ipynb` and work through the educational content step by step. +### System Information Class +- **Platform detection**: Python version, operating system, machine architecture +- **Compatibility checking**: Verify minimum requirements for TinyTorch development +- **Environment validation**: Ensure proper setup for course progression -## Testing +### Developer Profile Class +- **Personalized signatures**: Professional code attribution and contact information +- **ASCII art integration**: Custom flame art loading with graceful fallbacks +- **Educational customization**: Personalize your TinyTorch learning experience -Run the comprehensive test suite using pytest: - -```bash -# Using the TinyTorch CLI (recommended) -tito test --module setup - -# Or directly with pytest -python -m pytest tests/test_setup.py -v -``` - -### Test Coverage - -The test suite includes **20 comprehensive tests** covering: -- โœ… **Function execution** - All functions run without errors -- โœ… **Output validation** - Correct content and formatting -- โœ… **Arithmetic operations** - Basic, negative, and floating-point math -- โœ… **System information** - Platform detection and compatibility -- โœ… **Developer profiles** - Default and custom configurations -- โœ… **ASCII art handling** - File loading and fallback behavior -- โœ… **Error recovery** - Graceful handling of missing files -- โœ… **Integration testing** - All components work together - -## Getting Started +## ๐Ÿš€ Getting Started ### Prerequisites +Ensure you have completed the TinyTorch installation and environment setup: -1. **Activate the virtual environment**: - ```bash - source bin/activate-tinytorch.sh - ``` +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -2. **Test the setup module**: - ```bash - tito test --module setup - ``` - -## Development Workflow - -This module teaches the core TinyTorch development cycle: - -1. **Write code** in the notebook using `#| export` directives -2. **Export code** with `tito sync --module setup` -3. **Run tests** with `tito test --module setup` -4. **Check progress** with `tito info` - -## Key Concepts - -- **NBDev workflow** - Write in notebooks, export to Python packages -- **Export directives** - Use `#| export` to mark code for export -- **Module โ†’ Package mapping** - This module exports to `tinytorch/core/utils.py` -- **Teaching vs. Building** - Learn by modules, build by function -- **Student implementation** - TODO sections with instructor solutions hidden - -## Personalization Features - -### ASCII Art Customization -The ASCII art is loaded from `tinytorch_flame.txt`. You can customize it by: - -1. **Edit the file directly** - Modify `tinytorch_flame.txt` with your own ASCII art -2. **Custom parameter** - Pass your own ASCII art to `DeveloperProfile` -3. **Create your own design** - Your initials, logo, or motivational art - -### Developer Profile Customization -```python -my_profile = DeveloperProfile( - name="Your Name", - affiliation="Your University", - email="your.email@example.com", - github_username="yourgithub", - ascii_art="Your custom ASCII art here!" -) +# Verify installation +tito doctor ``` -## What You'll Learn +### Development Workflow +1. **Open the development notebook**: `modules/source/01_setup/setup_dev.py` +2. **Follow the guided implementation**: Complete TODO sections with provided scaffolding +3. **Export your code**: `tito export --module setup` +4. **Test your implementation**: `tito test --module setup` +5. **Verify integration**: `tito nbdev build` to ensure package compatibility -This comprehensive module introduces: -- **NBDev educational patterns** - `#| export` directives and NBGrader solution markers -- **File I/O operations** - Loading ASCII art with error handling -- **Object-oriented programming** - Classes, methods, and properties -- **System programming** - Platform detection and compatibility -- **Testing with pytest** - Professional test structure and assertions -- **Code organization** - Module structure and package exports -- **The TinyTorch development workflow** - Complete cycle from code to tests +## ๐Ÿงช Testing Your Implementation -## Next Steps +### Comprehensive Test Suite +Run the full test suite to verify your implementation: -Once you've completed this module and all tests pass, you're ready to move on to the **tensor module** where you'll build the core data structures that power TinyTorch neural networks! +```bash +# TinyTorch CLI (recommended) +tito test --module setup -The skills you learn here - the development workflow, testing patterns, and code organization - will be used throughout every module in TinyTorch. +# Direct pytest execution +python -m pytest tests/ -k setup -v +``` + +### Test Coverage (20 Tests) +- โœ… **Function execution**: All functions run without errors +- โœ… **Output validation**: Correct content and formatting +- โœ… **Arithmetic operations**: Basic, negative, and floating-point math +- โœ… **System information**: Platform detection and compatibility +- โœ… **Developer profiles**: Default and custom configurations +- โœ… **ASCII art handling**: File loading and fallback behavior +- โœ… **Error recovery**: Graceful handling of missing files +- โœ… **Integration testing**: All components work together + +### Inline Testing +The module includes educational inline tests that run during development: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: SystemInfo functionality... +โœ… System detection works +โœ… Compatibility checking works +๐Ÿ“ˆ Progress: SystemInfo โœ“ +``` + +## ๐ŸŽฏ Key Concepts + +### Real-World Applications +- **Development Environment Management**: Like PyTorch's system compatibility checking +- **Professional Code Attribution**: Similar to open-source project contributor systems +- **Educational Scaffolding**: Mirrors industry onboarding and training workflows +- **System Validation**: Foundation for deployment compatibility (used in modules 12-14) + +### Core Programming Patterns +- **NBDev Integration**: Write once in notebooks, deploy everywhere as Python packages +- **Export Directives**: Strategic use of `#| export` for clean package structure +- **Error Handling**: Graceful fallbacks for missing resources and system incompatibilities +- **Object-Oriented Design**: Classes with clear responsibilities and professional interfaces +- **Testing Philosophy**: Comprehensive coverage with both unit and integration approaches + +### TinyTorch Foundation +This module establishes patterns used throughout the course: +- **Module โ†’ Package Mapping**: `setup_dev.py` โ†’ `tinytorch.core.setup` +- **Development Workflow**: Edit โ†’ Export โ†’ Test โ†’ Iterate cycle +- **Educational Structure**: Guided implementation with instructor solutions +- **Professional Standards**: Production-ready code with full test coverage ## ๐ŸŽ‰ Ready to Build? -Welcome to TinyTorch! This is your foundation module where you'll master the development workflow that powers every subsequent module. You're about to build your first components and establish the coding patterns that will carry you through the entire course. +You're about to establish the foundation that will power your entire TinyTorch journey! This module teaches the development workflow mastery that professional ML engineers use daily. -Take your time, test thoroughly, and enjoy building something that really works! ๐Ÿ”ฅ \ No newline at end of file +Every advanced concept you'll learn - from tensors to optimizers to MLOps - builds on the solid patterns you're about to implement here. Take your time, test thoroughly, and enjoy building something that really works! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/01_setup/setup_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/01_setup/setup_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/01_setup/setup_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/03_activations/README.md b/modules/source/03_activations/README.md index 24e17762..e321868e 100644 --- a/modules/source/03_activations/README.md +++ b/modules/source/03_activations/README.md @@ -6,156 +6,122 @@ - **Prerequisites**: Tensor module - **Next Steps**: Layers module -Welcome to the **Activations** module! This is where you'll implement the mathematical functions that give neural networks their power to learn complex patterns. +Welcome to the **Activations** module! This is where you'll implement the mathematical functions that give neural networks their power to learn complex patterns. Without activation functions, neural networks would just be linear transformationsโ€”with them, you unlock the ability to learn any function. ## ๐ŸŽฏ Learning Objectives -By the end of this module, you will: -1. **Understand** why activation functions are essential for neural networks -2. **Implement** the three most important activation functions: ReLU, Sigmoid, and Tanh -3. **Test** your functions with various inputs to understand their behavior -4. **Grasp** the mathematical properties that make each function useful +By the end of this module, you will be able to: -## ๐Ÿง  Why This Module Matters +- **Understand the critical role** of activation functions in enabling neural networks to learn non-linear patterns +- **Implement three core activation functions**: ReLU, Sigmoid, and Tanh with proper numerical stability +- **Apply mathematical reasoning** to understand function properties, ranges, and appropriate use cases +- **Debug and test** activation implementations using both automated tests and visual analysis +- **Connect theory to practice** by understanding when and why to use each activation function -**Without activation functions, neural networks are just linear transformations!** +## ๐Ÿง  Build โ†’ Use โ†’ Analyze -``` -Linear โ†’ Linear โ†’ Linear = Still just Linear -Linear โ†’ Activation โ†’ Linear = Can learn complex patterns! -``` +This module follows TinyTorch's **Build โ†’ Use โ†’ Analyze** framework: -This module teaches you the mathematical foundations that make deep learning possible. +1. **Build**: Implement ReLU, Sigmoid, and Tanh activation functions with numerical stability +2. **Use**: Apply these functions in testing scenarios and visualize their mathematical behavior +3. **Analyze**: Compare function properties, performance characteristics, and appropriate use cases through quantitative analysis ## ๐Ÿ“š What You'll Build -### 1. **ReLU** (Rectified Linear Unit) +### Core Activation Functions +```python +# ReLU: Simple but powerful +relu = ReLU() +output = relu(Tensor([-2, -1, 0, 1, 2])) # [0, 0, 0, 1, 2] + +# Sigmoid: Probabilistic outputs +sigmoid = Sigmoid() +output = sigmoid(Tensor([0, 1, -1])) # [0.5, 0.73, 0.27] + +# Tanh: Zero-centered activation +tanh = Tanh() +output = tanh(Tensor([0, 1, -1])) # [0, 0.76, -0.76] +``` + +### ReLU (Rectified Linear Unit) - **Formula**: `f(x) = max(0, x)` -- **Properties**: Simple, sparse, unbounded -- **Use case**: Hidden layers (most common) +- **Properties**: Simple, sparse, unbounded, most commonly used +- **Implementation**: Element-wise maximum with zero +- **Use Cases**: Hidden layers in most modern architectures -### 2. **Sigmoid** +### Sigmoid Activation - **Formula**: `f(x) = 1 / (1 + e^(-x))` -- **Properties**: Bounded to (0,1), smooth, probabilistic -- **Use case**: Binary classification, gates +- **Properties**: Bounded to (0,1), smooth, probabilistic interpretation +- **Implementation**: Numerically stable version preventing overflow +- **Use Cases**: Binary classification, attention mechanisms, gates -### 3. **Tanh** (Hyperbolic Tangent) +### Tanh (Hyperbolic Tangent) - **Formula**: `f(x) = tanh(x)` -- **Properties**: Bounded to (-1,1), zero-centered, smooth -- **Use case**: Hidden layers, RNNs +- **Properties**: Bounded to (-1,1), zero-centered, symmetric +- **Implementation**: Direct NumPy implementation with shape preservation +- **Use Cases**: Hidden layers, RNNs, when zero-centered outputs are beneficial ## ๐Ÿš€ Getting Started ### Prerequisites +Ensure you have completed the tensor module and understand basic tensor operations: -1. **Activate the virtual environment**: - ```bash - source bin/activate-tinytorch.sh - ``` +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -2. **Start development environment**: - ```bash - tito jupyter - ``` +# Verify tensor module is working +tito test --module tensor +``` ### Development Workflow +1. **Open the development file**: `modules/source/03_activations/activations_dev.py` +2. **Implement functions progressively**: Start with ReLU, then Sigmoid (numerical stability), then Tanh +3. **Test each implementation**: Use inline tests for immediate feedback +4. **Visualize function behavior**: Leverage plotting sections for mathematical understanding +5. **Export and verify**: `tito export --module activations && tito test --module activations` -1. **Open the development file**: - ```bash - # Then open assignments/source/02_activations/activations_dev.py - ``` +## ๐Ÿงช Testing Your Implementation -2. **Implement the functions**: - - Start with ReLU (simplest) - - Move to Sigmoid (numerical stability challenge) - - Finish with Tanh (symmetry properties) +### Comprehensive Test Suite +Run the full test suite to verify mathematical correctness: -3. **Visualize your functions**: - - Each function has plotting sections - - See how your implementation transforms inputs - - Compare all functions side-by-side +```bash +# TinyTorch CLI (recommended) +tito test --module activations -4. **Test as you go**: - ```bash - tito test --module activations - ``` - -5. **Export to package**: - ```bash - tito sync - ``` - -### ๐Ÿ“Š Visual Learning Features - -This module includes comprehensive plotting sections to help you understand: - -- **Individual Function Plots**: See each activation function's curve -- **Implementation Comparison**: Your implementation vs ideal side-by-side -- **Mathematical Explanations**: Visual breakdown of function properties -- **Error Analysis**: Quantitative feedback on implementation accuracy -- **Comprehensive Comparison**: All functions analyzed together - -**Enhanced Features**: -- **4-Panel Plots**: Implementation vs ideal, mathematical definition, properties, error analysis -- **Real-time Feedback**: Immediate accuracy scores with color-coded status -- **Mathematical Insights**: Detailed explanations of function properties -- **Numerical Stability Testing**: Verification with extreme values -- **Property Verification**: Symmetry, monotonicity, and zero-centering tests - -**Why enhanced plots matter**: -- **Visual Debugging**: See exactly where your implementation differs -- **Quantitative Feedback**: Get precise error measurements -- **Mathematical Understanding**: Connect formulas to visual behavior -- **Implementation Confidence**: Know immediately if your code is correct -- **Learning Reinforcement**: Multiple visual perspectives of the same concept - -### Implementation Tips - -#### ReLU Implementation -```python -def forward(self, x: Tensor) -> Tensor: - return Tensor(np.maximum(0, x.data)) +# Direct pytest execution +python -m pytest tests/ -k activations -v ``` -#### Sigmoid Implementation (Numerical Stability) +### Test Coverage Areas +- โœ… **Mathematical Correctness**: Verify function outputs match expected mathematical formulas +- โœ… **Numerical Stability**: Test with extreme values and edge cases +- โœ… **Shape Preservation**: Ensure input and output tensors have identical shapes +- โœ… **Range Validation**: Confirm outputs fall within expected ranges +- โœ… **Integration Testing**: Verify compatibility with tensor operations + +### Inline Testing & Visualization +The module includes comprehensive educational feedback: ```python -def forward(self, x: Tensor) -> Tensor: - # For x >= 0: sigmoid(x) = 1 / (1 + exp(-x)) - # For x < 0: sigmoid(x) = exp(x) / (1 + exp(x)) - x_data = x.data - result = np.zeros_like(x_data) - - positive_mask = x_data >= 0 - result[positive_mask] = 1.0 / (1.0 + np.exp(-x_data[positive_mask])) - result[~positive_mask] = np.exp(x_data[~positive_mask]) / (1.0 + np.exp(x_data[~positive_mask])) - - return Tensor(result) +# Example inline test output +๐Ÿ”ฌ Unit Test: ReLU activation... +โœ… ReLU handles negative inputs correctly +โœ… ReLU preserves positive inputs +โœ… ReLU output range is [0, โˆž) +๐Ÿ“ˆ Progress: ReLU โœ“ + +# Visual feedback with plotting +๐Ÿ“Š Plotting ReLU behavior across range [-5, 5]... +๐Ÿ“ˆ Function visualization shows expected behavior ``` -#### Tanh Implementation +### Manual Testing Examples ```python -def forward(self, x: Tensor) -> Tensor: - return Tensor(np.tanh(x.data)) -``` - -### Testing Your Implementation - -1. **Run the tests**: - ```bash - tito test --module activations - ``` - -2. **Export to package**: - ```bash - tito sync - ``` - -### Manual Testing -```python -# Test all activations from tinytorch.core.tensor import Tensor -from modules.activations.activations_dev import ReLU, Sigmoid, Tanh +from activations_dev import ReLU, Sigmoid, Tanh +# Test with various inputs x = Tensor([[-2.0, -1.0, 0.0, 1.0, 2.0]]) relu = ReLU() @@ -163,96 +129,64 @@ sigmoid = Sigmoid() tanh = Tanh() print("Input:", x.data) -print("ReLU:", relu(x).data) -print("Sigmoid:", sigmoid(x).data) -print("Tanh:", tanh(x).data) +print("ReLU:", relu(x).data) # [0, 0, 0, 1, 2] +print("Sigmoid:", sigmoid(x).data) # [0.12, 0.27, 0.5, 0.73, 0.88] +print("Tanh:", tanh(x).data) # [-0.96, -0.76, 0, 0.76, 0.96] ``` -## ๐Ÿ“Š Understanding Function Properties +## ๐ŸŽฏ Key Concepts -### Range Comparison -| Function | Input Range | Output Range | Zero Point | -|----------|-------------|--------------|------------| -| ReLU | (-โˆž, โˆž) | [0, โˆž) | f(0) = 0 | -| Sigmoid | (-โˆž, โˆž) | (0, 1) | f(0) = 0.5 | -| Tanh | (-โˆž, โˆž) | (-1, 1) | f(0) = 0 | +### Real-World Applications +- **Computer Vision**: ReLU activations enable CNNs to learn hierarchical features (like those in ResNet, VGG) +- **Natural Language Processing**: Sigmoid/Tanh functions power LSTM and GRU gates for memory control +- **Recommendation Systems**: Sigmoid activations provide probability outputs for binary predictions +- **Generative Models**: Different activations shape the output distributions in GANs and VAEs -### Key Properties -- **ReLU**: Sparse (zeros out negatives), unbounded, simple -- **Sigmoid**: Probabilistic (0-1 range), smooth, saturating -- **Tanh**: Zero-centered, symmetric, stronger gradients than sigmoid +### Mathematical Properties Comparison +| Function | Input Range | Output Range | Zero Point | Key Property | +|----------|-------------|--------------|------------|--------------| +| ReLU | (-โˆž, โˆž) | [0, โˆž) | f(0) = 0 | Sparse, unbounded | +| Sigmoid | (-โˆž, โˆž) | (0, 1) | f(0) = 0.5 | Probabilistic | +| Tanh | (-โˆž, โˆž) | (-1, 1) | f(0) = 0 | Zero-centered | -## ๐Ÿ”ง Integration with TinyTorch +### Numerical Stability Considerations +- **ReLU**: No stability issues (simple max operation) +- **Sigmoid**: Requires careful implementation to prevent `exp()` overflow +- **Tanh**: Generally stable, but NumPy implementation handles edge cases -After implementation, your activations will be available as: - -```python -from tinytorch.core.activations import ReLU, Sigmoid, Tanh - -# Use in neural networks -relu = ReLU() -output = relu(input_tensor) -``` - -## ๐ŸŽฏ Common Issues & Solutions - -### Issue 1: Sigmoid Overflow -**Problem**: `exp()` overflow with large inputs -**Solution**: Use numerically stable implementation (see code above) - -### Issue 2: Wrong Output Range -**Problem**: Sigmoid/Tanh outputs outside expected range -**Solution**: Check your mathematical implementation - -### Issue 3: Shape Mismatch -**Problem**: Output shape differs from input shape -**Solution**: Ensure element-wise operations preserve shape - -### Issue 4: Import Errors -**Problem**: Cannot import after implementation -**Solution**: Run `tito sync` to export to package - -## ๐Ÿ“ˆ Performance Considerations - -- **ReLU**: Fastest (simple max operation) -- **Sigmoid**: Moderate (exponential computation) -- **Tanh**: Moderate (hyperbolic function) - -All implementations use NumPy for vectorized operations. - -## ๐Ÿš€ What's Next - -After mastering activations, you'll use them in: -1. **Layers Module**: Building neural network layers -2. **Loss Functions**: Computing training objectives -3. **Advanced Architectures**: CNNs, RNNs, and more - -These functions are the mathematical foundation for everything that follows! - -## ๐Ÿ“š Further Reading - -**Mathematical Background**: -- [Activation Functions in Neural Networks](https://en.wikipedia.org/wiki/Activation_function) -- [Deep Learning Book - Chapter 6](http://www.deeplearningbook.org/) - -**Advanced Topics**: -- ReLU variants (Leaky ReLU, ELU, Swish) -- Activation function choice and impact -- Gradient flow and vanishing gradients - -## ๐ŸŽ‰ Success Criteria - -You've mastered this module when: -- [ ] All tests pass (`tito test --module activations`) -- [ ] You understand why each function is useful -- [ ] You can explain the mathematical properties -- [ ] You can use activations in neural networks -- [ ] You appreciate the importance of nonlinearity - -**Great work! You've built the mathematical foundation of neural networks!** ๐ŸŽ‰ +### Performance and Gradient Properties +- **ReLU**: Fastest computation, sparse gradients, can cause "dying ReLU" problem +- **Sigmoid**: Moderate computation, smooth gradients, susceptible to vanishing gradients +- **Tanh**: Moderate computation, stronger gradients than sigmoid, zero-centered helps optimization ## ๐ŸŽ‰ Ready to Build? -The activations module is where neural networks come alive! You're about to implement the mathematical functions that give networks their power to learn complex patterns and make intelligent decisions. +The activations module is where neural networks truly come alive! You're about to implement the mathematical functions that transform simple linear operations into powerful pattern recognition systems. -Take your time, test thoroughly, and enjoy building something that really works! ๐Ÿ”ฅ +Every major breakthrough in deep learningโ€”from image recognition to language modelsโ€”relies on the functions you're about to build. Take your time, understand the mathematics, and enjoy creating the foundation of intelligent systems! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/03_activations/activations_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/03_activations/activations_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/03_activations/activations_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` diff --git a/modules/source/04_layers/README.md b/modules/source/04_layers/README.md index d0b6802b..b0a5a487 100644 --- a/modules/source/04_layers/README.md +++ b/modules/source/04_layers/README.md @@ -6,207 +6,198 @@ - **Prerequisites**: Tensor, Activations modules - **Next Steps**: Networks module -**Build the fundamental transformations that compose into neural networks** +Build the fundamental transformations that compose into neural networks. This module teaches you that layers are simply functions that transform tensors, and neural networks are just sophisticated function composition using these building blocks. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand layers as functions that transform tensors: `y = f(x)` -- Implement Dense layers with linear transformations: `y = Wx + b` -- Add activation functions for nonlinearity (ReLU, Sigmoid, Tanh) -- See how neural networks are just function composition -- Build intuition for neural network architecture before diving into training +By the end of this module, you will be able to: -## ๐Ÿงฑ Build โ†’ Use โ†’ Understand +- **Understand layers as mathematical functions**: Recognize that layers transform tensors through well-defined mathematical operations +- **Implement Dense layers**: Build linear transformations using matrix multiplication and bias addition (`y = Wx + b`) +- **Integrate activation functions**: Combine linear layers with nonlinear activations to enable complex pattern learning +- **Compose simple building blocks**: Chain layers together to create complete neural network architectures +- **Debug layer implementations**: Use shape analysis and mathematical properties to verify correct implementation -This module follows the TinyTorch pedagogical framework: +## ๐Ÿง  Build โ†’ Use โ†’ Reflect -1. **Build**: Dense layers and activation functions from scratch -2. **Use**: Transform tensors and see immediate results -3. **Understand**: How neural networks transform information +This module follows TinyTorch's **Build โ†’ Use โ†’ Reflect** framework: + +1. **Build**: Implement Dense layers and activation functions from mathematical foundations +2. **Use**: Transform tensors through layer operations and see immediate results in various scenarios +3. **Reflect**: Understand how simple layers compose into complex neural networks and why architecture matters ## ๐Ÿ“š What You'll Build -### **Dense Layer** +### Core Layer Implementation ```python +# Dense layer: fundamental building block layer = Dense(input_size=3, output_size=2) x = Tensor([[1.0, 2.0, 3.0]]) -y = layer(x) # Shape: (1, 2) -``` +y = layer(x) # Shape transformation: (1, 3) โ†’ (1, 2) -### **Activation Functions** -```python +# With activation functions relu = ReLU() -sigmoid = Sigmoid() -tanh = Tanh() +activated = relu(y) # Apply nonlinearity -x = Tensor([[-1.0, 0.0, 1.0]]) -y_relu = relu(x) # [0.0, 0.0, 1.0] -y_sigmoid = sigmoid(x) # [0.27, 0.5, 0.73] -y_tanh = tanh(x) # [-0.76, 0.0, 0.76] +# Chaining operations +layer1 = Dense(784, 128) # Image โ†’ hidden +layer2 = Dense(128, 10) # Hidden โ†’ classes +activation = ReLU() + +# Forward pass composition +x = Tensor([[1.0, 2.0, 3.0, ...]]) # Input data +h1 = activation(layer1(x)) # First transformation +output = layer2(h1) # Final prediction ``` -### **Neural Networks** -```python -# 3 โ†’ 4 โ†’ 2 network -layer1 = Dense(input_size=3, output_size=4) -activation1 = ReLU() -layer2 = Dense(input_size=4, output_size=2) -activation2 = Sigmoid() +### Dense Layer Implementation +- **Mathematical foundation**: Linear transformation `y = Wx + b` +- **Weight initialization**: Xavier/Glorot uniform initialization for stable gradients +- **Bias handling**: Optional bias terms for translation invariance +- **Shape management**: Automatic handling of batch dimensions and matrix operations -# Forward pass -x = Tensor([[1.0, 2.0, 3.0]]) -h1 = layer1(x) -h1_activated = activation1(h1) -h2 = layer2(h1_activated) -output = activation2(h2) -``` +### Activation Layer Integration +- **ReLU integration**: Most common activation for hidden layers +- **Sigmoid integration**: Probability outputs for binary classification +- **Tanh integration**: Zero-centered outputs for better optimization +- **Composition patterns**: Standard ways to combine layers and activations ## ๐Ÿš€ Getting Started ### Prerequisites -- Complete Module 1: Tensor โœ… -- Understand basic linear algebra (matrix multiplication) -- Familiar with Python classes and methods +Ensure you have completed the foundational modules: -### Quick Start ```bash -# Navigate to the layers module -cd modules/layers +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -# Work in the development notebook -jupyter notebook layers_dev.ipynb - -# Or work in the Python file -code layers_dev.py +# Verify prerequisite modules +tito test --module tensor +tito test --module activations ``` -## ๐Ÿ“– Module Structure - -``` -modules/layers/ -โ”œโ”€โ”€ layers_dev.py # Main development file (work here!) -โ”œโ”€โ”€ layers_dev.ipynb # Jupyter notebook version -โ”œโ”€โ”€ tests/ -โ”‚ โ””โ”€โ”€ test_layers.py # Comprehensive tests -โ”œโ”€โ”€ README.md # This file -โ””โ”€โ”€ solutions/ # Reference implementations (if stuck) -``` - -## ๐ŸŽ“ Learning Path - -### Step 1: Dense Layer (Linear Transformation) -- Understand `y = Wx + b` -- Implement weight initialization -- Handle matrix multiplication and bias addition -- Test with single examples and batches - -### Step 2: Activation Functions -- Implement ReLU: `max(0, x)` -- Implement Sigmoid: `1 / (1 + e^(-x))` -- Implement Tanh: `tanh(x)` -- Understand why nonlinearity is crucial - -### Step 3: Layer Composition -- Chain layers together -- Build complete neural networks -- See how simple layers create complex functions - -### Step 4: Real-World Application -- Build an image classification network -- Understand how architecture affects capability +### Development Workflow +1. **Open the development file**: `modules/source/04_layers/layers_dev.py` +2. **Implement Dense layer class**: Start with `__init__` and `forward` methods +3. **Test layer functionality**: Use inline tests for immediate feedback +4. **Add activation integration**: Combine layers with activation functions +5. **Build complete networks**: Chain multiple layers together +6. **Export and verify**: `tito export --module layers && tito test --module layers` ## ๐Ÿงช Testing Your Implementation -### Module-Level Tests -```bash -# Run comprehensive tests -python -m pytest tests/test_layers.py -v +### Comprehensive Test Suite +Run the full test suite to verify mathematical correctness: -# Quick test -python -c "from layers_dev import Dense, ReLU; print('โœ… Layers working!')" +```bash +# TinyTorch CLI (recommended) +tito test --module layers + +# Direct pytest execution +python -m pytest tests/ -k layers -v ``` -### Package-Level Tests -```bash -# Export to package -python ../../bin/tito.py sync +### Test Coverage Areas +- โœ… **Layer Functionality**: Verify Dense layers perform correct linear transformations +- โœ… **Weight Initialization**: Ensure proper weight initialization for training stability +- โœ… **Shape Preservation**: Confirm layers handle batch dimensions correctly +- โœ… **Activation Integration**: Test seamless combination with activation functions +- โœ… **Network Composition**: Verify layers can be chained into complete networks -# Test integration -python ../../bin/tito.py test --module layers +### Inline Testing & Development +The module includes educational feedback during development: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: Dense layer functionality... +โœ… Dense layer computes y = Wx + b correctly +โœ… Weight initialization within expected range +โœ… Output shape matches expected dimensions +๐Ÿ“ˆ Progress: Dense Layer โœ“ + +# Integration testing +๐Ÿ”ฌ Unit Test: Layer composition... +โœ… Multiple layers chain correctly +โœ… Activations integrate seamlessly +๐Ÿ“ˆ Progress: Layer Composition โœ“ +``` + +### Manual Testing Examples +```python +from tinytorch.core.tensor import Tensor +from layers_dev import Dense +from activations_dev import ReLU + +# Test basic layer functionality +layer = Dense(input_size=3, output_size=2) +x = Tensor([[1.0, 2.0, 3.0]]) +y = layer(x) +print(f"Input shape: {x.shape}, Output shape: {y.shape}") + +# Test layer composition +layer1 = Dense(3, 4) +layer2 = Dense(4, 2) +relu = ReLU() + +# Forward pass +h1 = relu(layer1(x)) +output = layer2(h1) +print(f"Final output: {output.data}") ``` ## ๐ŸŽฏ Key Concepts -### **Layers as Functions** -- Input: Tensor with some shape -- Transformation: Mathematical operation -- Output: Tensor with possibly different shape +### Real-World Applications +- **Computer Vision**: Dense layers process flattened image features in CNNs (like VGG, ResNet final layers) +- **Natural Language Processing**: Dense layers transform word embeddings in transformers and RNNs +- **Recommendation Systems**: Dense layers combine user and item features for preference prediction +- **Scientific Computing**: Dense layers approximate complex functions in physics simulations and engineering -### **Linear vs Nonlinear** -- Dense layers: Linear transformations -- Activation functions: Nonlinear transformations -- Composition: Linear + Nonlinear = Complex functions +### Mathematical Foundations +- **Linear Transformation**: `y = Wx + b` where W is the weight matrix and b is the bias vector +- **Matrix Multiplication**: Efficient batch processing through vectorized operations +- **Weight Initialization**: Xavier/Glorot initialization prevents vanishing/exploding gradients +- **Function Composition**: Networks as nested function calls: `f3(f2(f1(x)))` -### **Neural Networks = Function Composition** -``` -Input โ†’ Dense โ†’ ReLU โ†’ Dense โ†’ Sigmoid โ†’ Output -``` +### Neural Network Building Blocks +- **Modularity**: Layers as reusable components that can be combined in different ways +- **Standardized Interface**: All layers follow the same input/output pattern for easy composition +- **Shape Consistency**: Automatic handling of batch dimensions and shape transformations +- **Nonlinearity**: Activation functions between layers enable learning of complex patterns -### **Why This Matters** -- **Modularity**: Build complex networks from simple parts -- **Reusability**: Same layers work for different problems -- **Understanding**: Know how each part contributes to the whole +### Implementation Patterns +- **Class-based Design**: Layers as objects with state (weights) and behavior (forward pass) +- **Initialization Strategy**: Proper weight initialization for stable training dynamics +- **Error Handling**: Graceful handling of shape mismatches and invalid inputs +- **Testing Philosophy**: Comprehensive testing of mathematical properties and edge cases -## ๐Ÿ” Common Issues +## ๐ŸŽ‰ Ready to Build? -### **Import Errors** -```python -# Make sure you're in the right directory -import sys -sys.path.append('../../') -from modules.tensor.tensor_dev import Tensor -``` +You're about to build the fundamental building blocks that power every neural network! Dense layers might seem simple, but they're the workhorses of deep learningโ€”from the final layers of image classifiers to the core components of language models. -### **Shape Mismatches** -```python -# Check input/output sizes match -layer1 = Dense(input_size=3, output_size=4) -layer2 = Dense(input_size=4, output_size=2) # 4 matches output of layer1 -``` +Understanding how these simple linear transformations compose into complex intelligence is one of the most beautiful insights in machine learning. Take your time, understand the mathematics, and enjoy building the foundation of artificial intelligence! -### **Gradient Issues (Later)** -```python -# Use proper weight initialization -limit = math.sqrt(6.0 / (input_size + output_size)) -weights = np.random.uniform(-limit, limit, (input_size, output_size)) -``` +```{grid} 3 +:gutter: 3 +:margin: 2 -## ๐ŸŽ‰ Success Criteria +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/04_layers/layers_dev.py +:class-title: text-center +:class-body: text-center -You've successfully completed this module when: -- โœ… All tests pass (`pytest tests/test_layers.py`) -- โœ… You can build a 2-layer neural network -- โœ… You understand how layers transform tensors -- โœ… You see the connection between layers and neural networks -- โœ… Package export works (`tito test --module layers`) +Interactive development environment -## ๐Ÿš€ What's Next +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/04_layers/layers_dev.ipynb +:class-title: text-center +:class-body: text-center -After completing this module, you're ready for: -- **Module 3: Networks** - Compose layers into common architectures -- **Module 4: Training** - Learn how networks improve through experience -- **Module 5: Applications** - Use networks for real problems +Google Colab notebook -## ๐Ÿค Getting Help +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/04_layers/layers_dev.py +:class-title: text-center +:class-body: text-center -- Check the tests for examples of expected behavior -- Look at the solutions/ directory if you're stuck -- Review the pedagogical principles in `docs/pedagogy/` -- Remember: Build โ†’ Use โ†’ Understand! - ---- - -**Great job building the foundation of neural networks!** ๐ŸŽ‰ - -*This module implements the core insight: neural networks are just function composition of simple building blocks.* \ No newline at end of file +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/05_networks/README.md b/modules/source/05_networks/README.md index 1aa7ff76..3913025c 100644 --- a/modules/source/05_networks/README.md +++ b/modules/source/05_networks/README.md @@ -4,273 +4,228 @@ - **Difficulty**: โญโญโญ Advanced - **Time Estimate**: 5-7 hours - **Prerequisites**: Tensor, Activations, Layers modules -- **Next Steps**: Training, CNN modules +- **Next Steps**: CNN, Training modules -**Compose layers into complete neural network architectures with powerful visualizations** +Compose layers into complete neural network architectures with powerful visualizations. This module teaches you that neural networks are function composition at scaleโ€”taking simple building blocks and combining them into systems capable of learning complex patterns and making intelligent decisions. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand networks as function composition: `f(x) = layer_n(...layer_2(layer_1(x)))` -- Build common architectures (MLP, CNN) from layers -- Visualize network structure and data flow -- See how architecture affects capability -- Master forward pass inference (no training yet!) +By the end of this module, you will be able to: -> **Note:** -> **MLP (Multi-Layer Perceptron) is not a fundamental building block, but a use case of composing Dense layers and activations in sequence.** -> In TinyTorch, you will learn to build MLPs by composing primitives, not as a separate module. This approach helps you see that all architectures (MLP, CNN, etc.) are just patterns of composition, not new primitives. +- **Master function composition**: Understand how networks are built as `f(x) = layer_n(...layer_2(layer_1(x)))` +- **Design neural architectures**: Build MLPs, classifiers, and regressors from compositional principles +- **Visualize network behavior**: Use advanced plotting to understand data flow and architectural decisions +- **Analyze architectural trade-offs**: Compare depth vs width, activation choices, and design patterns +- **Apply networks to real tasks**: Create appropriate architectures for classification and regression problems -## ๐Ÿง  Build โ†’ Use โ†’ Understand +## ๐Ÿง  Build โ†’ Use โ†’ Optimize -This module follows the TinyTorch pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: -1. **Build**: Compose layers into complete networks -2. **Use**: Create different architectures and run inference -3. **Understand**: How architecture design affects network behavior +1. **Build**: Compose layers into complete network architectures using function composition principles +2. **Use**: Apply networks to classification and regression tasks, visualizing behavior and data flow +3. **Optimize**: Analyze architectural choices, compare design patterns, and understand performance trade-offs ## ๐Ÿ“š What You'll Build -### **Sequential Network** +### Sequential Network Architecture ```python -# Basic network composition +# Function composition in action network = Sequential([ - Dense(784, 128), - ReLU(), - Dense(128, 64), - ReLU(), - Dense(64, 10), - Sigmoid() + Dense(784, 128), # Input transformation + ReLU(), # Nonlinearity + Dense(128, 64), # Feature compression + ReLU(), # More nonlinearity + Dense(64, 10), # Classification head + Sigmoid() # Probability outputs ]) -# Forward pass -x = Tensor([[1.0, 2.0, 3.0, ...]]) # Input data -output = network(x) # Network prediction +# Single forward pass processes entire batch +x = Tensor([[...]]) # Input batch +predictions = network(x) # End-to-end inference ``` -### **MLP (Multi-Layer Perceptron)** +### Specialized Network Builders ```python -# Create MLP for classification -mlp = create_mlp( - input_size=784, # 28x28 image - hidden_sizes=[128, 64], # Hidden layers - output_size=10, # 10 classes - activation=ReLU, - output_activation=Sigmoid -) -``` - -### **Specialized Networks** -```python -# Classification network -classifier = create_classification_network( - input_size=100, num_classes=2 +# MLP for multi-class classification +classifier = create_mlp( + input_size=784, # Flattened 28x28 images + hidden_sizes=[256, 128], # Two hidden layers + output_size=10, # 10 digit classes + activation=ReLU, # Hidden layer activation + output_activation=Sigmoid # Probability outputs ) -# Regression network +# Regression network for continuous prediction regressor = create_regression_network( - input_size=13, output_size=1 + input_size=13, # Housing features + hidden_sizes=[64, 32], # Progressive compression + output_size=1 # Single price prediction +) + +# Binary classification with appropriate architecture +binary_classifier = create_classification_network( + input_size=100, + num_classes=2, + architecture='deep' # Optimized for binary tasks ) ``` -## ๐ŸŽจ Visualization Features +### Advanced Network Analysis +```python +# Comprehensive architecture visualization +visualize_network_architecture(network) +# Shows: layer types, connections, parameter counts, data flow -This module includes powerful visualizations to help you understand: +# Behavior analysis with real data +analyze_network_behavior(network, sample_data) +# Shows: activation patterns, layer statistics, transformation analysis -### **Network Architecture Visualization** -- **Layer-by-layer structure**: See how layers connect -- **Color-coded layers**: Different colors for Dense, ReLU, Sigmoid, etc. -- **Connection arrows**: Visualize data flow between layers -- **Layer details**: Input/output sizes and parameters - -### **Data Flow Visualization** -- **Shape transformations**: See how tensor shapes change through the network -- **Activation patterns**: Visualize intermediate layer outputs -- **Statistics tracking**: Mean, std, and distribution of activations -- **Layer analysis**: Understand what each layer learns - -### **Network Comparison** -- **Side-by-side analysis**: Compare different architectures -- **Performance metrics**: Output distributions and statistics -- **Architectural insights**: Layer type distributions and complexity - -### **Behavior Analysis** -- **Input-output relationships**: How inputs map to outputs -- **Activation patterns**: Layer-by-layer activation analysis -- **Network depth**: Understanding the role of depth vs width -- **Practical insights**: Real-world application considerations +# Architectural comparison +compare_networks([shallow_net, deep_net, wide_net]) +# Shows: performance characteristics, complexity trade-offs +``` ## ๐Ÿš€ Getting Started ### Prerequisites -- Complete Module 1: Tensor โœ… -- Complete Module 2: Layers โœ… -- Understand basic function composition -- Familiar with matplotlib for visualizations +Ensure you have mastered the foundational building blocks: -### Quick Start ```bash -# Navigate to the networks module -cd modules/networks +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -# Work in the development notebook -jupyter notebook networks_dev.ipynb - -# Or work in the Python file -code networks_dev.py +# Verify all prerequisite modules +tito test --module tensor +tito test --module activations +tito test --module layers ``` -## ๐Ÿ“– Module Structure - -``` -modules/networks/ -โ”œโ”€โ”€ networks_dev.py # Main development file (work here!) -โ”œโ”€โ”€ networks_dev.ipynb # Jupyter notebook version -โ”œโ”€โ”€ tests/ -โ”‚ โ””โ”€โ”€ test_networks.py # Comprehensive tests -โ”œโ”€โ”€ README.md # This file -โ””โ”€โ”€ solutions/ # Reference implementations (if stuck) -``` - -## ๐ŸŽ“ Learning Path - -### Step 1: Sequential Network (Function Composition) -- Understand `f(x) = layer_n(...layer_1(x))` -- Implement basic network composition -- Test with simple examples - -### Step 2: Network Visualization -- Visualize network architectures -- Understand data flow through networks -- Compare different network designs - -### Step 3: Common Architectures -- Build MLPs for different tasks -- Create classification networks -- Design regression networks - -### Step 4: Behavior Analysis -- Analyze network behavior with different inputs -- Understand architectural trade-offs -- See how design affects capability - -### Step 5: Practical Applications -- Build networks for real problems -- Understand classification vs regression -- See how architecture matches task +### Development Workflow +1. **Open the development file**: `modules/source/05_networks/networks_dev.py` +2. **Implement Sequential class**: Build the composition framework for chaining layers +3. **Create network builders**: Implement MLPs and specialized architectures +4. **Add visualization tools**: Build plotting functions for network analysis +5. **Test with real scenarios**: Apply networks to classification and regression tasks +6. **Export and verify**: `tito export --module networks && tito test --module networks` ## ๐Ÿงช Testing Your Implementation -### Module-Level Tests -```bash -# Run comprehensive tests -python -m pytest tests/test_networks.py -v +### Comprehensive Test Suite +Run the full test suite to verify architectural correctness: -# Quick test -python -c "from networks_dev import Sequential; print('โœ… Networks working!')" +```bash +# TinyTorch CLI (recommended) +tito test --module networks + +# Direct pytest execution +python -m pytest tests/ -k networks -v ``` -### Package-Level Tests -```bash -# Export to package -python ../../bin/tito sync +### Test Coverage Areas +- โœ… **Sequential Composition**: Verify layers chain correctly with proper data flow +- โœ… **Network Builders**: Test MLP and specialized network creation functions +- โœ… **Shape Consistency**: Ensure networks handle various input shapes and batch sizes +- โœ… **Visualization Functions**: Verify plotting and analysis tools work correctly +- โœ… **Real-world Applications**: Test networks on classification and regression tasks -# Test integration -python ../../bin/tito test --module networks +### Inline Testing & Visualization +The module includes comprehensive educational feedback and visual analysis: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: Sequential network composition... +โœ… Layers chain correctly with proper data flow +โœ… Forward pass produces expected output shapes +โœ… Network handles batch processing correctly +๐Ÿ“ˆ Progress: Sequential Networks โœ“ + +# Visualization feedback +๐Ÿ“Š Generating network architecture visualization... +๐Ÿ“ˆ Showing data flow through 3-layer MLP +๐Ÿ“Š Layer analysis: 784โ†’128โ†’64โ†’10 parameter flow +``` + +### Manual Testing Examples +```python +from tinytorch.core.tensor import Tensor +from networks_dev import Sequential, create_mlp +from layers_dev import Dense +from activations_dev import ReLU, Sigmoid + +# Test network composition +network = Sequential([ + Dense(10, 5), + ReLU(), + Dense(5, 2), + Sigmoid() +]) + +# Forward pass +x = Tensor([[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]]) +output = network(x) +print(f"Network output: {output.data}, Shape: {output.shape}") + +# Test MLP builder +mlp = create_mlp(input_size=4, hidden_sizes=[8, 4], output_size=2) +test_input = Tensor([[1.0, 2.0, 3.0, 4.0]]) +prediction = mlp(test_input) +print(f"MLP prediction: {prediction.data}") ``` ## ๐ŸŽฏ Key Concepts -### **Function Composition** -- Networks as `f(x) = g(h(x))` -- Each layer is a function -- Composition creates complex behavior +### Real-World Applications +- **Image Classification**: ResNet and VGG architectures use sequential composition of convolutional and dense layers +- **Natural Language Processing**: Transformer architectures compose attention layers with feed-forward networks +- **Recommendation Systems**: Deep collaborative filtering uses MLPs to learn user-item interactions +- **Autonomous Systems**: Neural networks in self-driving cars compose perception, planning, and control layers -### **Architecture Design** -- **Depth**: Number of layers -- **Width**: Number of neurons per layer -- **Activation**: Nonlinearity choices -- **Output**: Task-specific final layer +### Function Composition Theory +- **Mathematical Foundation**: Networks implement nested function composition `f_n(f_{n-1}(...f_1(x)))` +- **Universal Approximation**: MLPs with sufficient width can approximate any continuous function +- **Depth vs Width Trade-offs**: Deep networks learn hierarchical features, wide networks increase expressivity +- **Architectural Inductive Biases**: Network structure encodes assumptions about the problem domain -### **Visualization Benefits** -- **Debugging**: See where things go wrong -- **Understanding**: Visualize complex transformations -- **Design**: Compare different architectures -- **Intuition**: Build mental models of networks +### Visualization and Analysis +- **Architecture Visualization**: Understand network structure through visual representation +- **Data Flow Analysis**: Track how information transforms through each layer +- **Activation Pattern Analysis**: Visualize what each layer learns to represent +- **Comparative Analysis**: Understand trade-offs between different architectural choices -### **Practical Considerations** -- **Input size**: Must match your data -- **Output size**: Must match your task -- **Hidden layers**: Balance complexity vs overfitting -- **Activation functions**: Choose based on task +### Design Patterns and Best Practices +- **Progressive Dimensionality**: Common pattern of gradually reducing dimensions toward output +- **Activation Placement**: Standard practice of activation after each linear transformation +- **Output Layer Design**: Task-specific final layers (sigmoid for binary, softmax for multi-class) +- **Network Depth Guidelines**: Balance between expressivity and training difficulty -## ๐Ÿ” Common Issues +## ๐ŸŽ‰ Ready to Build? -### **Import Errors** -```python -# Make sure you're in the right directory -import sys -sys.path.append('../../') -from modules.layers.layers_dev import Dense -from modules.activations.activations_dev import ReLU, Sigmoid -``` +You're about to master the art of neural architecture design! This is where the magic happensโ€”taking simple mathematical building blocks and composing them into systems capable of recognizing images, understanding language, and making intelligent decisions. -### **Shape Mismatches** -```python -# Check layer sizes match -layer1 = Dense(3, 4) # 3 inputs, 4 outputs -layer2 = Dense(4, 2) # 4 inputs (matches layer1 output), 2 outputs -``` +Every breakthrough in AI, from AlexNet to GPT, started with someone thoughtfully composing layers into powerful architectures. You're about to learn those same composition principles and build networks that can solve real problems! -### **Visualization Issues** -```python -# Make sure matplotlib is installed -pip install matplotlib seaborn +```{grid} 3 +:gutter: 3 +:margin: 2 -# Check if plots are disabled during testing -if _should_show_plots(): - # Your visualization code - pass -``` +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/05_networks/networks_dev.py +:class-title: text-center +:class-body: text-center -## ๐ŸŽ‰ Success Criteria +Interactive development environment -You've successfully completed this module when: -- โœ… All tests pass (`pytest tests/test_networks.py`) -- โœ… You can build and visualize different network architectures -- โœ… You understand how architecture affects network behavior -- โœ… You can create networks for classification and regression tasks -- โœ… Package export works (`tito test --module networks`) +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/05_networks/networks_dev.ipynb +:class-title: text-center +:class-body: text-center -## ๐Ÿš€ What's Next +Google Colab notebook -After completing this module, you're ready for: -- **Module 4: Training** - Learn how networks learn from data -- **Module 5: Data** - Work with real datasets -- **Module 6: Applications** - Solve real-world problems +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/05_networks/networks_dev.py +:class-title: text-center +:class-body: text-center -## ๐Ÿค Getting Help - -- Check the tests for examples of expected behavior -- Look at the solutions/ directory if you're stuck -- Review the pedagogical principles in `docs/pedagogy/` -- Remember: Build โ†’ Use โ†’ Understand! - -## ๐ŸŽจ Visualization Examples - -### Network Architecture -``` -Input โ†’ Dense(784,128) โ†’ ReLU โ†’ Dense(128,64) โ†’ ReLU โ†’ Dense(64,10) โ†’ Sigmoid โ†’ Output -``` - -### Data Flow -``` -(1,784) โ†’ (1,128) โ†’ (1,128) โ†’ (1,64) โ†’ (1,64) โ†’ (1,10) โ†’ (1,10) -``` - -### Layer Analysis -- **Dense layers**: Linear transformations -- **ReLU**: Introduces nonlinearity -- **Sigmoid**: Outputs probabilities - -**Build powerful neural networks with beautiful visualizations!** ๐Ÿš€ \ No newline at end of file +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/06_cnn/README.md b/modules/source/06_cnn/README.md index f7a00d47..ef91750a 100644 --- a/modules/source/06_cnn/README.md +++ b/modules/source/06_cnn/README.md @@ -6,71 +6,216 @@ - **Prerequisites**: Tensor, Activations, Layers, Networks modules - **Next Steps**: Training, Computer Vision modules -**Implement the core building block of modern computer vision: the convolutional layer.** +Implement the core building block of modern computer vision: the convolutional layer. This module teaches you how convolution transforms computer vision from hand-crafted features to learned hierarchical representations that power everything from image recognition to autonomous vehicles. ## ๐ŸŽฏ Learning Objectives -- Understand the convolution operation (sliding window, local connectivity, weight sharing) -- Implement Conv2D with explicit for-loops (single channel, single filter, no stride/pad) -- Visualize how convolution builds feature maps -- Compose Conv2D with other layers to build a simple ConvNet -- (Stretch) Explore stride, padding, pooling, and multi-channel input -## ๐Ÿง  Build โ†’ Use โ†’ Understand -1. **Build**: Implement Conv2D from scratch (for-loop) -2. **Use**: Compose Conv2D with ReLU, Flatten, Dense to build a ConvNet -3. **Understand**: Visualize and analyze how convolution works +By the end of this module, you will be able to: + +- **Understand convolution fundamentals**: Master the sliding window operation, local connectivity, and weight sharing principles +- **Implement Conv2D from scratch**: Build convolutional layers using explicit loops to understand the core operation +- **Visualize feature learning**: See how convolution builds feature maps and hierarchical representations +- **Design CNN architectures**: Compose convolutional layers with pooling and dense layers into complete networks +- **Apply computer vision principles**: Understand how CNNs revolutionized image processing and pattern recognition + +## ๐Ÿง  Build โ†’ Use โ†’ Analyze + +This module follows TinyTorch's **Build โ†’ Use โ†’ Analyze** framework: + +1. **Build**: Implement Conv2D from scratch using explicit for-loops to understand the core convolution operation +2. **Use**: Compose Conv2D with activation functions and other layers to build complete convolutional networks +3. **Analyze**: Visualize learned features, understand architectural choices, and compare CNN performance characteristics ## ๐Ÿ“š What You'll Build -- **Conv2D (for-loop):** The core operation, implemented by you -- **Conv2D Layer:** Wrap your function in a layer class -- **Simple ConvNet:** Compose Conv2D โ†’ ReLU โ†’ Flatten โ†’ Dense -- **Visualization:** See how the filter slides and builds the output -## ๐Ÿ› ๏ธ Provided Functionality -- **Stride and Padding:** Provided as utilities or stretch goals -- **Multi-channel/Filter Support:** Provided or as stretch -- **Pooling (Max/Avg):** Optional, provided or as stretch -- **Flatten Layer:** Provided -- **Visualization:** Provided for learning -- **Tests:** Provided for feedback +### Core Convolution Implementation +```python +# Conv2D layer: the heart of computer vision +conv_layer = Conv2D(in_channels=3, out_channels=16, kernel_size=3) +input_image = Tensor([[[[...]]]]) # (batch, channels, height, width) +feature_maps = conv_layer(input_image) # Learned features -## ๐Ÿค” Why Focus on the For-Loop? -Implementing the convolution for-loop is the best way to understand what makes CNNs powerful. Youโ€™ll see exactly how the filter slides, how local patterns are captured, and why this operation is so efficient for images. Other features (stride, padding, pooling) are important, but the core insight comes from building the basic operation yourself. +# Understanding the operation +print(f"Input shape: {input_image.shape}") # (1, 3, 32, 32) +print(f"Output shape: {feature_maps.shape}") # (1, 16, 30, 30) +print(f"Learned {feature_maps.shape[1]} different feature detectors") +``` + +### Complete CNN Architecture +```python +# Simple CNN for image classification +cnn = Sequential([ + Conv2D(3, 16, kernel_size=3), # Feature extraction + ReLU(), # Nonlinearity + MaxPool2D(kernel_size=2), # Dimensionality reduction + Conv2D(16, 32, kernel_size=3), # Higher-level features + ReLU(), # More nonlinearity + Flatten(), # Prepare for dense layers + Dense(32 * 13 * 13, 128), # Feature integration + ReLU(), + Dense(128, 10), # Classification head + Sigmoid() # Probability outputs +]) + +# End-to-end image classification +image_batch = Tensor([[[[...]]]]) # Batch of images +predictions = cnn(image_batch) # Class probabilities +``` + +### Convolution Operation Details +- **Sliding Window**: Filter moves across input to detect local patterns +- **Weight Sharing**: Same filter applied everywhere for translation invariance +- **Local Connectivity**: Each output depends only on local input region +- **Feature Maps**: Multiple filters learn different feature detectors + +### CNN Building Blocks +- **Conv2D Layer**: Core convolution operation with learnable filters +- **Pooling Layers**: MaxPool and AvgPool for spatial downsampling +- **Flatten Layer**: Converts 2D feature maps to 1D for dense layers +- **Complete Networks**: Integration with existing Dense and activation layers ## ๐Ÿš€ Getting Started + +### Prerequisites +Ensure you have mastered the foundational network building blocks: + ```bash -cd modules/cnn -jupyter notebook cnn_dev.ipynb # or edit cnn_dev.py +# Activate TinyTorch environment +source bin/activate-tinytorch.sh + +# Verify all prerequisite modules +tito test --module tensor +tito test --module activations +tito test --module layers +tito test --module networks ``` -## ๐Ÿ“– Module Structure -``` -modules/cnn/ -โ”œโ”€โ”€ cnn_dev.py # Main development file (work here!) -โ”œโ”€โ”€ cnn_dev.ipynb # Jupyter notebook version -โ”œโ”€โ”€ tests/ -โ”‚ โ””โ”€โ”€ test_cnn.py # Tests for your implementation -โ”œโ”€โ”€ README.md # This file -``` +### Development Workflow +1. **Open the development file**: `modules/source/06_cnn/cnn_dev.py` +2. **Implement convolution operation**: Start with explicit for-loop implementation for understanding +3. **Build Conv2D layer class**: Wrap convolution in reusable layer interface +4. **Add pooling operations**: Implement MaxPool and AvgPool for spatial reduction +5. **Create complete CNNs**: Compose layers into full computer vision architectures +6. **Export and verify**: `tito export --module cnn && tito test --module cnn` ## ๐Ÿงช Testing Your Implementation + +### Comprehensive Test Suite +Run the full test suite to verify computer vision functionality: + ```bash -# Run tests -python -m pytest tests/test_cnn.py -v +# TinyTorch CLI (recommended) +tito test --module cnn + +# Direct pytest execution +python -m pytest tests/ -k cnn -v ``` -## ๐ŸŒŸ Stretch Goals -- Add stride and padding support -- Support multi-channel input/output -- Implement pooling layers -- Visualize learned filters and feature maps +### Test Coverage Areas +- โœ… **Convolution Operation**: Verify sliding window operation and local connectivity +- โœ… **Filter Learning**: Test weight initialization and parameter management +- โœ… **Shape Transformations**: Ensure proper input/output shape handling +- โœ… **Pooling Operations**: Verify spatial downsampling and feature preservation +- โœ… **CNN Integration**: Test complete networks with real image-like data -## ๐Ÿ’ก Key Insight -> **Convolution is a new, fundamental building block.** -> By implementing it yourself, youโ€™ll understand the magic behind modern vision models! +### Inline Testing & Visualization +The module includes comprehensive educational feedback and visual analysis: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: Conv2D implementation... +โœ… Convolution sliding window works correctly +โœ… Weight sharing applied consistently +โœ… Output shapes match expected dimensions +๐Ÿ“ˆ Progress: Conv2D โœ“ + +# Visualization feedback +๐Ÿ“Š Visualizing convolution operation... +๐Ÿ“ˆ Showing filter sliding across input +๐Ÿ“Š Feature map generation: 3โ†’16 channels +``` + +### Manual Testing Examples +```python +from tinytorch.core.tensor import Tensor +from cnn_dev import Conv2D, MaxPool2D, Flatten +from activations_dev import ReLU + +# Test basic convolution +conv = Conv2D(in_channels=1, out_channels=4, kernel_size=3) +input_img = Tensor([[[[1, 2, 3, 4, 5], + [6, 7, 8, 9, 10], + [11, 12, 13, 14, 15], + [16, 17, 18, 19, 20], + [21, 22, 23, 24, 25]]]]) +feature_maps = conv(input_img) +print(f"Input: {input_img.shape}, Features: {feature_maps.shape}") + +# Test complete CNN pipeline +relu = ReLU() +pool = MaxPool2D(kernel_size=2) +flatten = Flatten() + +# Forward pass through CNN layers +activated = relu(feature_maps) +pooled = pool(activated) +flattened = flatten(pooled) +print(f"Final shape: {flattened.shape}") +``` + +## ๐ŸŽฏ Key Concepts + +### Real-World Applications +- **Image Classification**: CNNs power systems like ImageNet winners (AlexNet, ResNet, EfficientNet) +- **Object Detection**: YOLO and R-CNN families use CNN backbones for feature extraction +- **Medical Imaging**: CNNs analyze X-rays, MRIs, and CT scans for diagnostic assistance +- **Autonomous Vehicles**: CNN-based perception systems process camera feeds for navigation + +### Computer Vision Fundamentals +- **Translation Invariance**: Convolution detects patterns regardless of position in image +- **Hierarchical Features**: Early layers detect edges, later layers detect objects and concepts +- **Parameter Efficiency**: Weight sharing dramatically reduces parameters compared to dense layers +- **Spatial Structure**: CNNs preserve and leverage 2D spatial relationships in images + +### Convolution Mathematics +- **Sliding Window Operation**: Filter moves across input with stride and padding parameters +- **Cross-Correlation vs Convolution**: Deep learning typically uses cross-correlation operation +- **Feature Map Computation**: Output[i,j] = sum(input[i:i+k, j:j+k] * filter) +- **Receptive Field**: Region of input that influences each output activation + +### CNN Architecture Patterns +- **Feature Extraction**: Convolution + ReLU + Pooling blocks extract hierarchical features +- **Classification Head**: Flatten + Dense layers perform final classification +- **Progressive Filtering**: Increasing filter count with decreasing spatial dimensions +- **Skip Connections**: Advanced architectures add residual connections for deeper networks ## ๐ŸŽ‰ Ready to Build? -The CNN module brings computer vision to TinyTorch! You're about to implement the core operation that powers modern image recognition, from filters and feature maps to complete convolutional networks. +You're about to implement the technology that revolutionized computer vision! CNNs transformed image processing from hand-crafted features to learned representations, enabling everything from photo tagging to medical diagnosis to autonomous driving. -Take your time, test thoroughly, and enjoy building something that really works! ๐Ÿ”ฅ +Understanding convolution from the ground upโ€”implementing the sliding window operation yourselfโ€”will give you deep insight into why CNNs work so well for visual tasks. Take your time with the core operation, visualize what's happening, and enjoy building the foundation of modern computer vision! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/06_cnn/cnn_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/06_cnn/cnn_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/06_cnn/cnn_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` diff --git a/modules/source/07_dataloader/README.md b/modules/source/07_dataloader/README.md index 7ef7d3c8..ce940ec2 100644 --- a/modules/source/07_dataloader/README.md +++ b/modules/source/07_dataloader/README.md @@ -6,309 +6,236 @@ - **Prerequisites**: Tensor, Layers modules - **Next Steps**: Training, Networks modules -Build the data pipeline foundation of TinyTorch! This module implements efficient data loading, preprocessing, and batching systems - the critical infrastructure that feeds neural networks during training. +Build the data pipeline foundation of TinyTorch! This module implements efficient data loading, preprocessing, and batching systemsโ€”the critical infrastructure that feeds neural networks during training and powers real-world ML systems. ## ๐ŸŽฏ Learning Objectives -By the end of this module, you will: -- โœ… Understand data engineering as the foundation of ML systems -- โœ… Implement reusable dataset abstractions and interfaces -- โœ… Build efficient data loaders with batching and shuffling -- โœ… Create data preprocessing pipelines for normalization -- โœ… Apply systems thinking to data I/O and memory management -- โœ… Have a complete data pipeline ready for neural network training +By the end of this module, you will be able to: -## ๐Ÿ“‹ Module Structure +- **Design data pipeline architectures**: Understand data engineering as the foundation of scalable ML systems +- **Implement reusable dataset abstractions**: Build flexible interfaces that support multiple data sources and formats +- **Create efficient data loaders**: Develop batching, shuffling, and streaming systems for optimal training performance +- **Build preprocessing pipelines**: Implement normalization, augmentation, and transformation systems +- **Apply systems engineering principles**: Handle memory management, I/O optimization, and error recovery in data pipelines -``` -modules/dataloader/ -โ”œโ”€โ”€ README.md # ๐Ÿ“– This file - Module overview -โ”œโ”€โ”€ dataloader_dev.py # ๐Ÿ”ง Main development file -โ”œโ”€โ”€ dataloader_dev.ipynb # ๐Ÿ““ Generated notebook (auto-created) -โ”œโ”€โ”€ tests/ -โ”‚ โ””โ”€โ”€ test_dataloader.py # ๐Ÿงช Automated tests -โ””โ”€โ”€ check_dataloader.py # โœ… Manual verification (coming soon) -``` +## ๐Ÿง  Build โ†’ Use โ†’ Optimize -## ๐Ÿš€ Getting Started +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: -### Step 1: Complete Prerequisites -Make sure you've completed the foundational modules: -```bash -tito test --module setup # Should pass -tito test --module tensor # Should pass -tito test --module layers # Should pass -``` +1. **Build**: Implement dataset abstractions, data loaders, and preprocessing pipelines from engineering principles +2. **Use**: Apply your data system to real CIFAR-10 dataset with complete train/test workflows +3. **Optimize**: Analyze performance characteristics, memory usage, and system bottlenecks for production readiness -### Step 2: Open the Data Development File -```bash -# Start from the dataloader module directory -cd modules/dataloader/ +## ๐Ÿ“š What You'll Build -# Convert to notebook if needed -tito notebooks --module dataloader - -# Open the development notebook -jupyter lab dataloader_dev.ipynb -``` - -### Step 3: Work Through the Implementation -The development file guides you through building: -1. **Dataset base class** - Abstract interface for all datasets -2. **CIFAR-10 implementation** - Real dataset with binary file parsing -3. **DataLoader** - Efficient batching and shuffling system -4. **Normalizer** - Data preprocessing for stable training -5. **Complete pipeline** - Integration of all components - -### Step 4: Export and Test -```bash -# Export your dataloader implementation -tito sync --module dataloader - -# Test your implementation -tito test --module dataloader -``` - -## ๐Ÿ“š What You'll Implement - -### Core Data Infrastructure -You'll build a complete data loading system that supports: - -#### 1. Dataset Abstraction +### Complete Data Pipeline System ```python -# Abstract base class for all datasets -class Dataset: - def __getitem__(self, index): - # Get single sample and label - pass - - def __len__(self): - # Get total number of samples - pass - - def get_num_classes(self): - # Get number of classes - pass - -# Concrete implementation -dataset = CIFAR10Dataset("data/cifar10/", train=True) -image, label = dataset[0] # Get first sample -``` - -#### 2. Real Dataset Loading -```python -# CIFAR-10 dataset with download and parsing -dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True) -print(f"Dataset size: {len(dataset)}") # 50,000 training samples -print(f"Sample shape: {dataset.get_sample_shape()}") # (3, 32, 32) -print(f"Classes: {dataset.get_num_classes()}") # 10 classes -``` - -#### 3. Efficient Data Loading -```python -# DataLoader with batching and shuffling -dataloader = DataLoader(dataset, batch_size=32, shuffle=True) -for batch_images, batch_labels in dataloader: - print(f"Batch shape: {batch_images.shape}") # (32, 3, 32, 32) - print(f"Labels shape: {batch_labels.shape}") # (32,) - # Ready for neural network training! -``` - -#### 4. Data Preprocessing -```python -# Normalizer for stable training -normalizer = Normalizer() -normalizer.fit(training_data) # Compute statistics -normalized_data = normalizer.transform(test_data) # Apply normalization -``` - -#### 5. Complete Pipeline -```python -# One-function pipeline creation +# End-to-end data pipeline creation train_loader, test_loader, normalizer = create_data_pipeline( dataset_path="data/cifar10/", batch_size=32, normalize=True, shuffle=True ) + +# Ready for neural network training +for batch_images, batch_labels in train_loader: + # batch_images.shape: (32, 3, 32, 32) - normalized pixel values + # batch_labels.shape: (32,) - class indices + predictions = model(batch_images) + loss = compute_loss(predictions, batch_labels) + # Continue training loop... ``` -### Technical Requirements -Your data system must: -- Handle multiple dataset types through common interface -- Efficiently load and parse binary data files -- Support batching with configurable batch sizes -- Implement shuffling for training randomization -- Provide data normalization for stable training -- Export to `tinytorch.core.dataloader` +### Dataset Abstraction System +```python +# Flexible interface supporting multiple datasets +class Dataset: + def __getitem__(self, index): + # Return (data, label) for any dataset type + pass + def __len__(self): + # Enable len() and iteration + pass + +# Concrete implementation with real data +dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True) +print(f"Loaded {len(dataset)} real samples") # 50,000 training images +image, label = dataset[0] # Access individual samples +print(f"Sample shape: {image.shape}, Label: {label}") +``` + +### Efficient Data Loading System +```python +# High-performance batching with memory optimization +dataloader = DataLoader( + dataset=dataset, + batch_size=32, # Configurable batch size + shuffle=True, # Training randomization + drop_last=False # Handle incomplete batches +) + +# Pythonic iteration interface +for batch_idx, (batch_data, batch_labels) in enumerate(dataloader): + print(f"Batch {batch_idx}: {batch_data.shape}") + # Automatic batching handles all the complexity +``` + +### Data Preprocessing Pipeline +```python +# Production-ready normalization system +normalizer = Normalizer() + +# Fit on training data (compute statistics once) +normalizer.fit(training_images) +print(f"Mean: {normalizer.mean}, Std: {normalizer.std}") + +# Apply to any dataset (training, validation, test) +normalized_images = normalizer.transform(test_images) +# Ensures consistent preprocessing across data splits +``` + +## ๐Ÿš€ Getting Started + +### Prerequisites +Ensure you have the foundational tensor operations: + +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh + +# Verify prerequisite modules +tito test --module tensor +tito test --module layers +``` + +### Development Workflow +1. **Open the development file**: `modules/source/07_dataloader/dataloader_dev.py` +2. **Implement Dataset abstraction**: Create the base interface for all data sources +3. **Build CIFAR-10 dataset**: Implement real dataset loading with binary file parsing +4. **Create DataLoader system**: Add batching, shuffling, and iteration functionality +5. **Add preprocessing tools**: Implement normalizer and transformation pipeline +6. **Export and verify**: `tito export --module dataloader && tito test --module dataloader` ## ๐Ÿงช Testing Your Implementation -### Progressive Testing with Real Data - -The tests follow the **"Build โ†’ Use โ†’ Understand"** pattern with real CIFAR-10 data: +### Comprehensive Test Suite +Run the full test suite to verify data engineering functionality: ```bash -# Run all tests (downloads real CIFAR-10 data) +# TinyTorch CLI (recommended) tito test --module dataloader -# Run specific test categories -python -m pytest tests/test_dataloader.py::TestDatasetInterface -v # Test abstract interface -python -m pytest tests/test_dataloader.py::TestCIFAR10Dataset -v # Test real data loading -python -m pytest tests/test_dataloader.py::TestDataLoader -v # Test batching real data -python -m pytest tests/test_dataloader.py::TestNormalizer -v # Test normalizing real data -python -m pytest tests/test_dataloader.py::TestDataPipeline -v # Test complete pipeline +# Direct pytest execution +python -m pytest tests/ -k dataloader -v ``` -### Real Data Testing Flow +### Test Coverage Areas +- โœ… **Dataset Interface**: Verify abstract base class and concrete implementations +- โœ… **Real Data Loading**: Test with actual CIFAR-10 dataset (downloads ~170MB) +- โœ… **Batching System**: Ensure correct batch shapes and memory efficiency +- โœ… **Data Preprocessing**: Verify normalization statistics and transformations +- โœ… **Pipeline Integration**: Test complete train/test workflow with real data -Each test builds on the previous component using actual CIFAR-10 data: - -1. **Build Dataset** โ†’ **Test**: Download and load real CIFAR-10 images (50,000 training, 10,000 test) -2. **Build DataLoader** โ†’ **Test**: Batch real images with proper shuffling and iteration -3. **Build Normalizer** โ†’ **Test**: Normalize real pixel values (0-255 range โ†’ standardized) -4. **Build Pipeline** โ†’ **Test**: Complete pipeline with real data flow and preprocessing - -### Why Real Data Testing Matters - -- **Real-world validation**: Tests work with actual data students will use in training -- **Immediate feedback**: See your pipeline working with real images, not fake data -- **Systems thinking**: Understand I/O, memory, and performance with real data distributions -- **Debugging**: Catch issues that only appear with real data (file formats, edge cases) - -**Note**: First test run downloads ~170MB CIFAR-10 dataset with progress bar. Subsequent runs use cached data. - -### Interactive Testing with Visual Feedback +### Inline Testing & Real Data Validation +The module includes comprehensive feedback using real CIFAR-10 data: ```python -# Test in the notebook or Python REPL -from tinytorch.core.dataloader import Dataset, DataLoader, CIFAR10Dataset +# Example inline test output +๐Ÿ”ฌ Unit Test: CIFAR-10 dataset loading... +๐Ÿ“ฅ Downloading CIFAR-10 dataset (170MB)... +โœ… Successfully loaded 50,000 training samples +โœ… Sample shapes correct: (3, 32, 32) +โœ… Labels in valid range: [0, 9] +๐Ÿ“ˆ Progress: CIFAR-10 Dataset โœ“ -# Create and test datasets with real data +# DataLoader testing with real data +๐Ÿ”ฌ Unit Test: DataLoader batching... +โœ… Batch shapes correct: (32, 3, 32, 32) +โœ… Shuffling produces different orders +โœ… Iteration covers all samples exactly once +๐Ÿ“ˆ Progress: DataLoader โœ“ +``` + +### Manual Testing Examples +```python +from tinytorch.core.tensor import Tensor +from dataloader_dev import CIFAR10Dataset, DataLoader, Normalizer + +# Test dataset loading with real data dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True) -print(f"Loaded {len(dataset)} real CIFAR-10 samples") +print(f"Dataset size: {len(dataset)}") +print(f"Classes: {dataset.get_num_classes()}") -# Test data loading -dataloader = DataLoader(dataset, batch_size=16) -for batch_data, batch_labels in dataloader: - print(f"Real batch shape: {batch_data.shape}") # (16, 3, 32, 32) - print(f"Real labels: {batch_labels}") # Actual CIFAR-10 classes - break +# Test data loading pipeline +dataloader = DataLoader(dataset, batch_size=16, shuffle=True) +for batch_images, batch_labels in dataloader: + print(f"Batch shape: {batch_images.shape}") + print(f"Label range: {batch_labels.min()} to {batch_labels.max()}") + break # Just test first batch + +# Test preprocessing pipeline +normalizer = Normalizer() +sample_batch, _ = next(iter(dataloader)) +normalizer.fit(sample_batch) +normalized = normalizer.transform(sample_batch) +print(f"Original range: [{sample_batch.min():.2f}, {sample_batch.max():.2f}]") +print(f"Normalized range: [{normalized.min():.2f}, {normalized.max():.2f}]") ``` -### ๐ŸŽจ Development Visual Feedback +## ๐ŸŽฏ Key Concepts -The development notebook (`dataloader_dev.py`) includes **visual feedback** for learning: +### Real-World Applications +- **Production ML Systems**: Companies like Netflix, Spotify use similar data pipelines for recommendation training +- **Computer Vision**: ImageNet, COCO dataset loaders power research and production vision systems +- **Natural Language Processing**: Text preprocessing pipelines enable language model training +- **Autonomous Systems**: Real-time data streams from sensors require efficient pipeline architectures -```python -# ๐Ÿ‘๏ธ SEE your data - Available in development notebook only -show_cifar10_samples(dataset, num_samples=8, title="My CIFAR-10 Data") -``` +### Data Engineering Principles +- **Interface Design**: Abstract Dataset class enables switching between data sources seamlessly +- **Memory Efficiency**: Streaming data loading prevents memory overflow with large datasets +- **I/O Optimization**: Batching reduces system calls and improves throughput +- **Preprocessing Consistency**: Fit-transform pattern ensures identical preprocessing across data splits -### ๐ŸŽจ Visual Feedback Features (Development Only) +### Systems Performance Considerations +- **Batch Size Trade-offs**: Larger batches improve GPU utilization but increase memory usage +- **Shuffling Strategy**: Random access patterns for training vs sequential for inference +- **Caching and Storage**: Balance between memory usage and I/O performance +- **Error Handling**: Robust handling of corrupted data, network failures, disk issues -The development notebook includes **visual feedback** for learning and debugging: - -- **Download progress bar**: Visual progress indicator during CIFAR-10 download (~170MB) -- **`show_cifar10_samples()`**: Display a grid of CIFAR-10 images with class labels -- **Real image visualization**: See actual airplanes, cars, birds, cats, etc. -- **Batch visualization**: View what your DataLoader is producing -- **Pipeline visualization**: See the complete data flow in action - -**Why Visual Feedback Matters:** -- **Build confidence**: See that your data pipeline is working correctly -- **Debug issues**: Spot problems like incorrect normalization or corrupted images -- **Understand data**: Build intuition about what your model will be learning from -- **Immediate feedback**: Visual confirmation follows the "Build โ†’ Use โ†’ Understand" pattern - -**Note**: Visual feedback is available in the development notebook (`data_dev.py`) for learning purposes. The core package exports only the essential data loading components. - -## ๐ŸŽฏ Success Criteria - -Your data module is complete when: - -1. **All tests pass**: `tito test --module dataloader` -2. **Data classes import correctly**: `from tinytorch.core.dataloader import Dataset, DataLoader` -3. **Dataset loading works**: Can create datasets and access samples -4. **Batching works**: DataLoader produces correct batch shapes -5. **Preprocessing works**: Normalizer computes and applies statistics -6. **Pipeline works**: Complete pipeline creates train/test loaders - -## ๐Ÿ’ก Implementation Tips - -### Start with the Interface -1. **Dataset base class** - Define the abstract interface -2. **Simple test dataset** - Create mock data for testing -3. **Basic DataLoader** - Implement batching without shuffling -4. **Add shuffling** - Randomize sample order -5. **Test frequently** - Verify each component works - -### Design Patterns -```python -class Dataset: - def __getitem__(self, index): - # Return (data, label) tuple - return data_tensor, label_tensor - - def __len__(self): - # Return total number of samples - return self.num_samples - -class DataLoader: - def __iter__(self): - # Yield batches of (batch_data, batch_labels) - for batch in self._create_batches(): - yield batch_data, batch_labels -``` - -### Systems Thinking -- **Memory management**: Don't load entire dataset into RAM -- **I/O efficiency**: Batch file operations when possible -- **Preprocessing**: Compute statistics once, apply many times -- **Interface design**: Make components easily swappable - -### Common Challenges -- **Binary file parsing** - CIFAR-10 uses custom format -- **Batch size handling** - Last batch may be smaller -- **Data type consistency** - Convert to consistent types -- **Error handling** - Provide helpful debugging messages - -## ๐Ÿ”ง Advanced Features (Optional) - -If you finish early, try implementing: -- **Data augmentation** - Random transformations for training -- **Multi-worker loading** - Parallel data loading -- **Caching** - Store processed data for faster access -- **Different datasets** - MNIST, Fashion-MNIST, etc. - -## ๐Ÿš€ Next Steps - -Once you complete the data module: - -1. **Move to Autograd**: `cd modules/autograd/` -2. **Build automatic differentiation**: Enable gradient computation -3. **Combine with data**: Train models on real datasets -4. **Prepare for training**: Ready for the training module - -## ๐Ÿ”— Why Data Engineering Matters - -Data engineering is the foundation of all ML systems: -- **Training loops** need efficient data loading -- **Model performance** depends on data quality -- **Production systems** require scalable data pipelines -- **Research** needs flexible data interfaces - -Your data implementation will power all TinyTorch training! - -## ๐Ÿ“Š Real-World Connection - -The patterns you'll implement are used in: -- **PyTorch DataLoader** - Same interface and concepts -- **TensorFlow tf.data** - Similar pipeline architecture -- **Production ML** - Scalable data processing systems -- **Research** - Flexible experimentation frameworks +### Production ML Pipeline Patterns +- **ETL Design**: Extract (load files), Transform (preprocess), Load (batch) pattern +- **Data Versioning**: Reproducible datasets with consistent preprocessing +- **Pipeline Monitoring**: Track data quality, distribution shifts, processing times +- **Scalability Planning**: Design for growing datasets and distributed processing ## ๐ŸŽ‰ Ready to Build? -The data module is where TinyTorch becomes a real ML system. You're about to create the infrastructure that will feed neural networks, enable training loops, and power production ML pipelines. +You're about to build the data engineering foundation that powers every successful ML system! From startup prototypes to billion-dollar recommendation engines, they all depend on robust data pipelines like the one you're building. -Focus on clean interfaces, efficient implementation, and systems thinking! ๐Ÿ”ฅ \ No newline at end of file +This module teaches you the systems thinking that separates hobby projects from production ML systems. You'll work with real data, handle real performance constraints, and build infrastructure that scales. Take your time, think about edge cases, and enjoy building the backbone of machine learning! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/07_dataloader/dataloader_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/07_dataloader/dataloader_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/07_dataloader/dataloader_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/08_autograd/README.md b/modules/source/08_autograd/README.md index cfd5f1e8..08d57b0f 100644 --- a/modules/source/08_autograd/README.md +++ b/modules/source/08_autograd/README.md @@ -6,287 +6,230 @@ - **Prerequisites**: Tensor, Activations, Layers modules - **Next Steps**: Training, Optimizers modules -**Build the automatic differentiation engine that makes neural network training possible** +Build the automatic differentiation engine that makes neural network training possible. This module implements the mathematical foundation that enables backpropagationโ€”transforming TinyTorch from a static computation library into a dynamic, trainable ML framework. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand how automatic differentiation works through computational graphs -- Implement the Variable class that tracks gradients and operations -- Build backward propagation for gradient computation -- Create differentiable versions of all mathematical operations -- Master the mathematical foundations of backpropagation +By the end of this module, you will be able to: + +- **Master automatic differentiation theory**: Understand computational graphs, chain rule application, and gradient flow +- **Implement gradient tracking systems**: Build the Variable class that automatically computes and accumulates gradients +- **Create differentiable operations**: Extend all mathematical operations to support backward propagation +- **Apply backpropagation algorithms**: Implement the gradient computation that enables neural network optimization +- **Integrate with ML systems**: Connect automatic differentiation with layers, networks, and training algorithms ## ๐Ÿง  Build โ†’ Use โ†’ Analyze -This module follows the TinyTorch pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Analyze** framework: -1. **Build**: Create the Variable class and gradient computation system -2. **Use**: Perform automatic differentiation on complex expressions -3. **Analyze**: Understand how gradients flow through computational graphs and optimize performance +1. **Build**: Implement Variable class and gradient computation system using mathematical differentiation rules +2. **Use**: Apply automatic differentiation to complex expressions and neural network forward passes +3. **Analyze**: Understand computational graph construction, memory usage, and performance characteristics of autodiff systems ## ๐Ÿ“š What You'll Build -### **Variable Class** +### Automatic Differentiation System ```python -# Gradient-tracking wrapper around Tensors +# Variables track gradients automatically x = Variable(5.0, requires_grad=True) y = Variable(3.0, requires_grad=True) -z = x * y + x**2 + +# Complex mathematical expressions +z = x**2 + 2*x*y + y**3 +print(f"f(x,y) = {z.data}") # Forward pass result + +# Automatic gradient computation z.backward() -print(x.grad) # Gradient of z with respect to x -print(y.grad) # Gradient of z with respect to y +print(f"df/dx = {x.grad}") # โˆ‚f/โˆ‚x = 2x + 2y = 16 +print(f"df/dy = {y.grad}") # โˆ‚f/โˆ‚y = 2x + 3yยฒ = 37 ``` -### **Differentiable Operations** +### Neural Network Integration ```python -# All operations track gradients automatically -def f(x, y): - return x**2 + 2*x*y + y**2 +# Seamless integration with existing TinyTorch components +from tinytorch.core.layers import Dense +from tinytorch.core.activations import ReLU + +# Create differentiable network +x = Variable([[1.0, 2.0, 3.0]], requires_grad=True) +layer1 = Dense(3, 4) # Weights automatically become Variables +layer2 = Dense(4, 1) +relu = ReLU() + +# Forward pass builds computational graph +h1 = relu(layer1(x)) +output = layer2(h1) +loss = output.sum() + +# Backward pass computes all gradients +loss.backward() + +# All parameters now have gradients +print(f"Layer 1 weight gradients: {layer1.weights.grad.shape}") +print(f"Layer 2 bias gradients: {layer2.bias.grad.shape}") +print(f"Input gradients: {x.grad.shape}") +``` + +### Computational Graph Construction +```python +# Automatic graph building for complex operations +def complex_function(x, y): + a = x * y # Multiplication node + b = x + y # Addition node + c = a / b # Division node + return c.sin() # Trigonometric node x = Variable(2.0, requires_grad=True) y = Variable(3.0, requires_grad=True) -result = f(x, y) +result = complex_function(x, y) + +# Chain rule applied automatically through entire graph result.backward() -print(f"df/dx = {x.grad}") # Should be 2x + 2y = 10 -print(f"df/dy = {y.grad}") # Should be 2x + 2y = 10 -``` - -### **Neural Network Integration** -```python -# Works seamlessly with existing TinyTorch components -from tinytorch.core.activations import ReLU -from tinytorch.core.layers import Dense - -# Create differentiable network -x = Variable([[1.0, 2.0, 3.0]]) -layer = Dense(3, 2) -relu = ReLU() - -# Forward pass with gradient tracking -output = relu(layer(x)) -loss = output.sum() -loss.backward() - -# Gradients available for all parameters -print(layer.weights.grad) # Weight gradients -print(layer.bias.grad) # Bias gradients +print(f"Complex gradient dx: {x.grad}") +print(f"Complex gradient dy: {y.grad}") ``` ## ๐Ÿš€ Getting Started ### Prerequisites +Ensure you understand the mathematical building blocks: -1. **Activate the virtual environment**: - ```bash - source bin/activate-tinytorch.sh - ``` +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -2. **Start development environment**: - ```bash - tito jupyter - ``` +# Verify prerequisite modules +tito test --module tensor +tito test --module activations +tito test --module layers +``` ### Development Workflow - -1. **Open the development file**: - ```bash - # Then open modules/source/07_autograd/autograd_dev.py - ``` - -2. **Implement the core components**: - - Start with Variable class (gradient tracking) - - Add basic operations (add, multiply, etc.) - - Implement backward propagation - - Add activation function gradients - -3. **Test your implementation**: - ```bash - tito test --module 07_autograd - ``` - -## ๐Ÿ“Š Understanding Automatic Differentiation - -### The Chain Rule in Action - -Automatic differentiation is based on the chain rule: -``` -If z = f(g(x)), then dz/dx = (dz/df) * (df/dx) -``` - -### Computational Graph Example -``` -Expression: f(x, y) = (x + y) * (x - y) - -Forward Pass: -x = 2, y = 3 -a = x + y = 5 -b = x - y = -1 -f = a * b = -5 - -Backward Pass: -df/df = 1 -df/da = b = -1, df/db = a = 5 -da/dx = 1, da/dy = 1 -db/dx = 1, db/dy = -1 -df/dx = df/da * da/dx + df/db * db/dx = (-1)(1) + (5)(1) = 4 -df/dy = df/da * da/dy + df/db * db/dy = (-1)(1) + (5)(-1) = -6 -``` - -### Key Concepts - -| Concept | Description | Example | -|---------|-------------|---------| -| **Variable** | Tensor wrapper with gradient tracking | `Variable(5.0, requires_grad=True)` | -| **Computational Graph** | DAG representing operations | `z = x * y` creates graph | -| **Forward Pass** | Computing function values | `z.data` contains result | -| **Backward Pass** | Computing gradients | `z.backward()` fills gradients | -| **Leaf Node** | Variable created by user | `x = Variable(5.0)` | -| **Gradient Function** | How to compute gradients | `grad_fn` for each operation | +1. **Open the development file**: `modules/source/08_autograd/autograd_dev.py` +2. **Implement Variable class**: Create gradient tracking wrapper around Tensors +3. **Add basic operations**: Implement differentiable arithmetic (add, multiply, power) +4. **Build backward propagation**: Implement chain rule for gradient computation +5. **Extend to all operations**: Add gradients for activations, matrix operations, etc. +6. **Export and verify**: `tito export --module autograd && tito test --module autograd` ## ๐Ÿงช Testing Your Implementation -### Unit Tests +### Comprehensive Test Suite +Run the full test suite to verify mathematical correctness: + ```bash -tito test --module 07_autograd +# TinyTorch CLI (recommended) +tito test --module autograd + +# Direct pytest execution +python -m pytest tests/ -k autograd -v ``` -**Test Coverage**: -- โœ… Variable creation and properties -- โœ… Basic arithmetic operations -- โœ… Gradient computation correctness -- โœ… Chain rule implementation -- โœ… Integration with existing modules +### Test Coverage Areas +- โœ… **Variable Creation**: Test gradient tracking initialization and properties +- โœ… **Basic Operations**: Verify arithmetic operations compute correct gradients +- โœ… **Chain Rule**: Ensure composite functions apply chain rule correctly +- โœ… **Backpropagation**: Test gradient flow through complex computational graphs +- โœ… **Neural Network Integration**: Verify seamless operation with layers and activations -### Manual Testing +### Inline Testing & Mathematical Verification +The module includes comprehensive mathematical validation: ```python -# Test basic gradients -x = Variable(2.0, requires_grad=True) -y = x**2 + 3*x + 1 +# Example inline test output +๐Ÿ”ฌ Unit Test: Variable gradient tracking... +โœ… Variable creation with gradient tracking +โœ… Leaf variables correctly identified +โœ… Gradient accumulation works correctly +๐Ÿ“ˆ Progress: Variable System โœ“ + +# Mathematical verification +๐Ÿ”ฌ Unit Test: Chain rule implementation... +โœ… f(x) = xยฒ โ†’ df/dx = 2x โœ“ +โœ… f(x,y) = xy โ†’ df/dx = y, df/dy = x โœ“ +โœ… Complex compositions follow chain rule โœ“ +๐Ÿ“ˆ Progress: Differentiation Rules โœ“ +``` + +### Manual Testing Examples +```python +from autograd_dev import Variable +import math + +# Test basic differentiation rules +x = Variable(3.0, requires_grad=True) +y = x**2 y.backward() -print(x.grad) # Should be 2*2 + 3 = 7 +print(f"d(xยฒ)/dx at x=3: {x.grad}") # Should be 6 # Test chain rule x = Variable(2.0, requires_grad=True) y = Variable(3.0, requires_grad=True) -z = x * y -w = z + x -w.backward() -print(x.grad) # Should be y + 1 = 4 -print(y.grad) # Should be x = 2 +z = (x + y) * (x - y) # Difference of squares +z.backward() +print(f"d/dx = {x.grad}") # Should be 2x = 4 +print(f"d/dy = {y.grad}") # Should be -2y = -6 + +# Test with transcendental functions +x = Variable(1.0, requires_grad=True) +y = x.exp().log() # Should equal x +y.backward() +print(f"d(exp(log(x)))/dx: {x.grad}") # Should be 1 ``` -## ๐Ÿ“Š Mathematical Foundations +## ๐ŸŽฏ Key Concepts -### Gradient Computation Rules +### Real-World Applications +- **Deep Learning Frameworks**: PyTorch, TensorFlow, JAX all use automatic differentiation for training +- **Scientific Computing**: Automatic differentiation enables gradient-based optimization in physics, chemistry, engineering +- **Financial Modeling**: Risk analysis and portfolio optimization use autodiff for sensitivity analysis +- **Robotics**: Control systems use gradients for trajectory optimization and inverse kinematics -| Operation | Forward | Backward (Gradient) | -|-----------|---------|-------------------| -| Addition | `z = x + y` | `dx = dz, dy = dz` | -| Multiplication | `z = x * y` | `dx = y * dz, dy = x * dz` | -| Power | `z = x^n` | `dx = n * x^(n-1) * dz` | -| Exp | `z = exp(x)` | `dx = exp(x) * dz` | -| Log | `z = log(x)` | `dx = (1/x) * dz` | -| ReLU | `z = max(0, x)` | `dx = (x > 0) * dz` | -| Sigmoid | `z = 1/(1+exp(-x))` | `dx = z * (1-z) * dz` | +### Mathematical Foundations +- **Chain Rule**: โˆ‚f/โˆ‚x = (โˆ‚f/โˆ‚u)(โˆ‚u/โˆ‚x) for composite functions f(u(x)) +- **Computational Graphs**: Directed acyclic graphs representing function composition +- **Forward Mode vs Reverse Mode**: Different autodiff strategies with different computational complexities +- **Gradient Accumulation**: Handling multiple computational paths to same variable -### Advanced Concepts -- **Higher-order gradients**: Gradients of gradients -- **Jacobian matrices**: Gradients for vector functions -- **Hessian matrices**: Second-order derivatives -- **Gradient checkpointing**: Memory optimization +### Automatic Differentiation Theory +- **Dual Numbers**: Mathematical foundation using infinitesimals for forward-mode AD +- **Reverse Accumulation**: Backpropagation as reverse-mode automatic differentiation +- **Higher-Order Derivatives**: Computing gradients of gradients for advanced optimization +- **Jacobian Computation**: Efficient computation of vector-valued function gradients -## ๐Ÿ”ง Integration with TinyTorch +### Implementation Patterns +- **Gradient Function Storage**: Each operation stores its backward function in the computational graph +- **Topological Sorting**: Ordering gradient computation to respect dependencies +- **Memory Management**: Efficient storage and cleanup of intermediate values +- **Numerical Stability**: Handling edge cases in gradient computation -After implementation, your autograd system will enable: +## ๐ŸŽ‰ Ready to Build? -```python -from tinytorch.core.autograd import Variable -from tinytorch.core.layers import Dense -from tinytorch.core.activations import ReLU +You're about to implement the mathematical foundation that makes modern AI possible! Automatic differentiation is the invisible engine that powers every neural network, from simple classifiers to GPT and beyond. -# Create a simple neural network -x = Variable([[1.0, 2.0, 3.0]]) -layer1 = Dense(3, 4) -layer2 = Dense(4, 1) -relu = ReLU() +Understanding autodiff from first principlesโ€”implementing the Variable class and chain rule yourselfโ€”will give you deep insight into how deep learning really works. This is where mathematics meets software engineering to create something truly powerful. Take your time, understand each gradient rule, and enjoy building the heart of machine learning! -# Forward pass -h = relu(layer1(x)) -output = layer2(h) -loss = output.sum() +```{grid} 3 +:gutter: 3 +:margin: 2 -# Backward pass -loss.backward() +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/08_autograd/autograd_dev.py +:class-title: text-center +:class-body: text-center -# All gradients computed automatically! -print(layer1.weights.grad) -print(layer2.weights.grad) -``` +Interactive development environment -## ๐ŸŽฏ Success Criteria +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/08_autograd/autograd_dev.ipynb +:class-title: text-center +:class-body: text-center -Your autograd module is complete when: +Google Colab notebook -1. **All tests pass**: `tito test --module 07_autograd` -2. **Variable imports correctly**: `from tinytorch.core.autograd import Variable` -3. **Basic operations work**: Can create Variables and do arithmetic -4. **Gradients compute correctly**: Backward pass produces correct gradients -5. **Integration works**: Seamlessly works with existing TinyTorch modules +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/08_autograd/autograd_dev.py +:class-title: text-center +:class-body: text-center -## ๐Ÿ’ก Implementation Tips - -### Start with the Basics -1. **Variable class** - Wrap Tensors with gradient tracking -2. **Simple operations** - Start with addition and multiplication -3. **Backward method** - Implement gradient computation -4. **Test frequently** - Verify gradients match analytical solutions - -### Design Patterns -```python -class Variable: - def __init__(self, data, requires_grad=True, grad_fn=None): - # Store data, gradient state, and computation history - - def backward(self, gradient=None): - # Implement backpropagation using chain rule - -def add(a, b): - # Create new Variable with grad_fn that knows how to backprop - def backward_fn(grad): - # Distribute gradient to inputs - return Variable(result, grad_fn=backward_fn) -``` - -### Common Challenges -- **Gradient accumulation** - Handle multiple paths to same Variable -- **Memory management** - Store intermediate values efficiently -- **Numerical stability** - Handle edge cases in gradient computation -- **Graph construction** - Build computation graph correctly - -## ๐Ÿ”ง Advanced Features (Optional) - -If you finish early, try implementing: -- **Higher-order gradients** - Gradients of gradients -- **Gradient checkpointing** - Memory optimization -- **Custom operations** - Define your own differentiable functions -- **Gradient clipping** - Prevent exploding gradients - -## ๐Ÿš€ Next Steps - -Once you complete the autograd module: - -1. **Move to Training**: `cd modules/source/08_training/` -2. **Build optimization algorithms**: Implement SGD, Adam, etc. -3. **Create training loops**: Put it all together -4. **Train real models**: Use your autograd system for actual ML! - -## ๐Ÿ”— Why Autograd Matters - -Automatic differentiation is the foundation of modern ML: -- **Neural networks** require gradients for backpropagation -- **Optimization** needs gradients for parameter updates -- **Research** benefits from easy gradient computation -- **Production** systems rely on efficient autodiff - -This module transforms TinyTorch from a static computation library into a dynamic, trainable ML framework! \ No newline at end of file +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/09_optimizers/README.md b/modules/source/09_optimizers/README.md index 14b878ce..48ce55ab 100644 --- a/modules/source/09_optimizers/README.md +++ b/modules/source/09_optimizers/README.md @@ -6,208 +6,237 @@ - **Prerequisites**: Tensor, Autograd modules - **Next Steps**: Training, MLOps modules -**Build intelligent optimization algorithms that enable effective neural network training** +Build intelligent optimization algorithms that enable effective neural network training. This module implements the learning algorithms that power modern AIโ€”from basic gradient descent to advanced adaptive methods that make training large-scale models possible. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand gradient descent and how optimizers use gradients to update parameters -- Implement SGD with momentum for accelerated convergence -- Build Adam optimizer with adaptive learning rates for modern deep learning -- Master learning rate scheduling strategies for training stability -- See how optimizers enable complete neural network training workflows +By the end of this module, you will be able to: -## ๐Ÿง  Build โ†’ Use โ†’ Analyze +- **Master gradient-based optimization theory**: Understand how gradients guide parameter updates and the mathematical foundations of learning +- **Implement core optimization algorithms**: Build SGD, momentum, and Adam optimizers from mathematical first principles +- **Design learning rate strategies**: Create scheduling systems that balance convergence speed with training stability +- **Apply optimization in practice**: Use optimizers effectively in complete training workflows with real neural networks +- **Analyze optimization dynamics**: Compare algorithm behavior, convergence patterns, and performance characteristics -This module follows the TinyTorch pedagogical framework: +## ๐Ÿง  Build โ†’ Use โ†’ Optimize -1. **Build**: Core optimization algorithms (SGD, Adam, scheduling) -2. **Use**: Apply optimizers to train neural networks effectively -3. **Analyze**: Compare optimizer behavior and convergence patterns +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: + +1. **Build**: Implement gradient descent, SGD with momentum, Adam optimizer, and learning rate scheduling from mathematical foundations +2. **Use**: Apply optimization algorithms to train neural networks and solve real optimization problems +3. **Optimize**: Analyze convergence behavior, compare algorithm performance, and tune hyperparameters for optimal training ## ๐Ÿ“š What You'll Build -### **Gradient Descent Foundation** +### Core Optimization Algorithms ```python -# Basic gradient descent step +# Gradient descent foundation def gradient_descent_step(parameter, learning_rate): parameter.data = parameter.data - learning_rate * parameter.grad.data -``` -### **SGD with Momentum** -```python -# Accelerated convergence -sgd = SGD([w1, w2, bias], learning_rate=0.01, momentum=0.9) -sgd.zero_grad() -loss.backward() -sgd.step() -``` +# SGD with momentum for accelerated convergence +sgd = SGD(parameters=[w1, w2, bias], learning_rate=0.01, momentum=0.9) +sgd.zero_grad() # Clear previous gradients +loss.backward() # Compute new gradients +sgd.step() # Update parameters -### **Adam Optimizer** -```python -# Adaptive learning rates -adam = Adam([w1, w2, bias], learning_rate=0.001, beta1=0.9, beta2=0.999) +# Adam optimizer with adaptive learning rates +adam = Adam(parameters=[w1, w2, bias], learning_rate=0.001, beta1=0.9, beta2=0.999) adam.zero_grad() loss.backward() -adam.step() +adam.step() # Adaptive updates per parameter ``` -### **Learning Rate Scheduling** +### Learning Rate Scheduling Systems ```python # Strategic learning rate adjustment scheduler = StepLR(optimizer, step_size=10, gamma=0.1) -scheduler.step() # Reduce learning rate every 10 epochs -``` - -### **Complete Training Integration** -```python -# Modern training loop -optimizer = Adam(model.parameters(), learning_rate=0.001) -scheduler = StepLR(optimizer, step_size=20, gamma=0.5) +# Training loop with scheduling for epoch in range(num_epochs): for batch in dataloader: optimizer.zero_grad() loss = criterion(model(batch.inputs), batch.targets) loss.backward() optimizer.step() - scheduler.step() + + scheduler.step() # Adjust learning rate each epoch + print(f"Epoch {epoch}, LR: {scheduler.get_last_lr()}") ``` -## ๐Ÿ”ฌ Core Concepts +### Complete Training Integration +```python +# Modern training workflow +model = Sequential([Dense(784, 128), ReLU(), Dense(128, 10)]) +optimizer = Adam(model.parameters(), learning_rate=0.001) +scheduler = StepLR(optimizer, step_size=20, gamma=0.5) -### **Gradient Descent Theory** -- **Mathematical foundation**: ฮธ = ฮธ - ฮฑโˆ‡L(ฮธ) -- **Learning rate**: Balance between convergence speed and stability -- **Convergence**: How optimizers reach optimal parameters +# Training loop with optimization +for epoch in range(num_epochs): + for batch_inputs, batch_targets in dataloader: + # Forward pass + predictions = model(batch_inputs) + loss = criterion(predictions, batch_targets) + + # Optimization step + optimizer.zero_grad() # Clear gradients + loss.backward() # Compute gradients + optimizer.step() # Update parameters + + scheduler.step() # Adjust learning rate +``` -### **Momentum Acceleration** -- **Velocity accumulation**: v_t = ฮฒv_{t-1} + โˆ‡L(ฮธ) -- **Oscillation dampening**: Smooth progress in consistent directions -- **Acceleration**: Build up speed toward minimum +### Optimization Algorithm Implementations +- **Gradient Descent**: Basic parameter update rule using gradients +- **SGD with Momentum**: Velocity accumulation for smoother convergence +- **Adam Optimizer**: Adaptive learning rates with bias correction +- **Learning Rate Scheduling**: Strategic adjustment during training -### **Adaptive Learning Rates** -- **First moment**: Exponential moving average of gradients -- **Second moment**: Exponential moving average of squared gradients -- **Bias correction**: Handle initialization bias in moment estimates +## ๐Ÿš€ Getting Started -### **Learning Rate Scheduling** -- **Step decay**: Reduce learning rate at fixed intervals -- **Convergence strategy**: Start fast, then refine with smaller steps -- **Training stability**: Prevent overshooting near optimum +### Prerequisites +Ensure you understand the mathematical foundations: -## ๐ŸŽฎ What You'll Experience +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -### **Immediate Feedback** -- **Test each optimizer**: See parameter updates in real-time -- **Compare convergence**: SGD vs Adam on same problem -- **Visualize learning**: Watch parameters converge to optimal values +# Verify prerequisite modules +tito test --module tensor +tito test --module autograd +``` -### **Real Training Workflow** -- **Complete training loop**: From gradients to parameter updates -- **Learning rate scheduling**: Strategic adjustment during training -- **Modern best practices**: Industry-standard optimization patterns +### Development Workflow +1. **Open the development file**: `modules/source/09_optimizers/optimizers_dev.py` +2. **Implement gradient descent**: Start with basic parameter update mechanics +3. **Build SGD with momentum**: Add velocity accumulation for acceleration +4. **Create Adam optimizer**: Implement adaptive learning rates with moment estimation +5. **Add learning rate scheduling**: Build strategic learning rate adjustment systems +6. **Export and verify**: `tito export --module optimizers && tito test --module optimizers` -### **Mathematical Insights** -- **Gradient interpretation**: How gradients guide parameter updates -- **Momentum physics**: Velocity and acceleration in optimization -- **Adaptive scaling**: Different learning rates for different parameters +## ๐Ÿงช Testing Your Implementation -## ๐Ÿ”ง Technical Implementation +### Comprehensive Test Suite +Run the full test suite to verify optimization algorithm correctness: -### **State Management** -- **Momentum buffers**: Track velocity for each parameter -- **Moment estimates**: First and second moments for Adam -- **Step counting**: Track iterations for bias correction +```bash +# TinyTorch CLI (recommended) +tito test --module optimizers -### **Numerical Stability** -- **Epsilon handling**: Prevent division by zero -- **Overflow protection**: Handle large gradients gracefully -- **Precision**: Balance between float32 and numerical accuracy +# Direct pytest execution +python -m pytest tests/ -k optimizers -v +``` -### **Memory Efficiency** -- **Lazy initialization**: Create buffers only when needed -- **Parameter tracking**: Use object IDs for state management -- **Gradient management**: Proper gradient zeroing and accumulation +### Test Coverage Areas +- โœ… **Algorithm Implementation**: Verify SGD, momentum, and Adam compute correct parameter updates +- โœ… **Mathematical Correctness**: Test against analytical solutions for convex optimization +- โœ… **State Management**: Ensure proper momentum and moment estimation tracking +- โœ… **Learning Rate Scheduling**: Verify step decay and scheduling functionality +- โœ… **Training Integration**: Test optimizers in complete neural network training workflows -## ๐Ÿ“ˆ Performance Characteristics +### Inline Testing & Convergence Analysis +The module includes comprehensive mathematical validation and convergence visualization: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: SGD with momentum... +โœ… Parameter updates follow momentum equations +โœ… Velocity accumulation works correctly +โœ… Convergence achieved on test function +๐Ÿ“ˆ Progress: SGD with Momentum โœ“ -### **SGD with Momentum** -- **Memory**: O(P) for momentum buffers (P = number of parameters) -- **Computation**: O(P) per step -- **Convergence**: Linear in convex case, good for large batch training +# Optimization analysis +๐Ÿ”ฌ Unit Test: Adam optimizer... +โœ… First moment estimation (m_t) computed correctly +โœ… Second moment estimation (v_t) computed correctly +โœ… Bias correction applied properly +โœ… Adaptive learning rates working +๐Ÿ“ˆ Progress: Adam Optimizer โœ“ +``` -### **Adam Optimizer** -- **Memory**: O(2P) for first and second moment buffers -- **Computation**: O(P) per step with additional operations -- **Convergence**: Fast initial progress, good for most deep learning +### Manual Testing Examples +```python +from optimizers_dev import SGD, Adam, StepLR +from autograd_dev import Variable -### **Learning Rate Scheduling** -- **Overhead**: Minimal computational cost -- **Impact**: Significant improvement in final performance -- **Flexibility**: Adaptable to different training scenarios +# Test SGD on simple quadratic function +x = Variable(10.0, requires_grad=True) +sgd = SGD([x], learning_rate=0.1, momentum=0.9) -## ๐Ÿ”— Integration with TinyTorch +for step in range(100): + sgd.zero_grad() + loss = x**2 # Minimize f(x) = xยฒ + loss.backward() + sgd.step() + if step % 10 == 0: + print(f"Step {step}: x = {x.data:.4f}, loss = {loss.data:.4f}") -### **Dependencies** -- **Tensor**: Core data structure for parameters -- **Autograd**: Gradient computation for parameter updates -- **Variables**: Parameter containers with gradient tracking +# Test Adam convergence +x = Variable([2.0, -3.0], requires_grad=True) +adam = Adam([x], learning_rate=0.01) -### **Enables** -- **Training Module**: Complete training loops with loss functions -- **Advanced Training**: Distributed training, mixed precision -- **Research**: Novel optimization algorithms and strategies +for step in range(50): + adam.zero_grad() + loss = (x[0]**2 + x[1]**2).sum() # Minimize ||x||ยฒ + loss.backward() + adam.step() + if step % 10 == 0: + print(f"Step {step}: x = {x.data}, loss = {loss.data:.6f}") +``` -## ๐ŸŽฏ Real-World Applications +## ๐ŸŽฏ Key Concepts -### **Computer Vision** -- **ImageNet training**: ResNet, VGG, Vision Transformers -- **Object detection**: YOLO, R-CNN optimization -- **Segmentation**: U-Net, Mask R-CNN training +### Real-World Applications +- **Large Language Models**: GPT, BERT training relies on Adam optimization for stable convergence +- **Computer Vision**: ResNet, Vision Transformer training uses SGD with momentum for best final performance +- **Recommendation Systems**: Online learning systems use adaptive optimizers for continuous model updates +- **Reinforcement Learning**: Policy gradient methods depend on careful optimizer choice and learning rate tuning -### **Natural Language Processing** -- **Language models**: GPT, BERT, T5 training -- **Machine translation**: Transformer optimization -- **Text generation**: Large language model training +### Mathematical Foundations +- **Gradient Descent**: ฮธ_{t+1} = ฮธ_t - ฮฑโˆ‡L(ฮธ_t) where ฮฑ is learning rate and โˆ‡L is loss gradient +- **Momentum**: v_{t+1} = ฮฒv_t + โˆ‡L(ฮธ_t), ฮธ_{t+1} = ฮธ_t - ฮฑv_{t+1} for accelerated convergence +- **Adam**: Combines momentum with adaptive learning rates using first and second moment estimates +- **Learning Rate Scheduling**: Strategic decay schedules balance exploration and exploitation -### **Scientific Computing** -- **Physics simulations**: Neural ODE optimization -- **Reinforcement learning**: Policy gradient methods -- **Generative models**: GAN, VAE training +### Optimization Theory +- **Convex Optimization**: Guarantees global minimum for convex loss functions +- **Non-convex Optimization**: Neural networks have complex loss landscapes with local minima +- **Convergence Analysis**: Understanding when and why optimization algorithms reach good solutions +- **Hyperparameter Sensitivity**: Learning rate is often the most critical hyperparameter -## ๐Ÿš€ What's Next - -After mastering optimizers, you'll be ready for: - -1. **Training Module**: Complete training loops with loss functions and metrics -2. **Advanced Optimizers**: RMSprop, AdaGrad, learning rate warm-up -3. **Distributed Training**: Multi-GPU optimization strategies -4. **MLOps**: Production optimization monitoring and tuning - -## ๐Ÿ’ก Key Insights - -### **Optimization is Critical** -- **Make or break**: Good optimizer choice determines training success -- **Hyperparameter sensitivity**: Learning rate is the most important hyperparameter -- **Architecture dependent**: Different models prefer different optimizers - -### **Modern Defaults** -- **Adam**: Default choice for most deep learning applications -- **SGD with momentum**: Still preferred for some computer vision tasks -- **Learning rate scheduling**: Almost always improves final performance - -### **Systems Thinking** -- **Memory trade-offs**: Adam uses more memory but often trains faster -- **Convergence patterns**: Understanding when and why optimizers work -- **Debugging**: Optimizer issues are common in training failures - -**Ready to build the intelligent algorithms that power modern AI training?** - -Your optimizers will be the engine that transforms gradients into intelligence! +### Performance Characteristics +- **SGD**: Memory efficient, works well with large batches, good final performance +- **Adam**: Fast initial convergence, works with small batches, requires more memory +- **Learning Rate Schedules**: Often crucial for achieving best performance +- **Algorithm Selection**: Problem-dependent choice based on data, model, and computational constraints ## ๐ŸŽ‰ Ready to Build? -The optimizers module is where learning happens! You're about to implement the algorithms that guide neural networks toward optimal solutions, from basic gradient descent to modern adaptive methods. +You're about to implement the algorithms that power all of modern AI! From the neural networks that recognize your voice to the language models that write code, they all depend on the optimization algorithms you're building. -Take your time, test thoroughly, and enjoy building something that really works! ๐Ÿ”ฅ +Understanding these algorithms from first principlesโ€”implementing momentum physics and adaptive learning rates yourselfโ€”will give you deep insight into why some training works and some doesn't. Take your time with the mathematics, test thoroughly, and enjoy building the intelligence behind intelligent systems! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/09_optimizers/optimizers_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/09_optimizers/optimizers_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/09_optimizers/optimizers_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` diff --git a/modules/source/10_training/README.md b/modules/source/10_training/README.md index e3849fa0..792f2f5c 100644 --- a/modules/source/10_training/README.md +++ b/modules/source/10_training/README.md @@ -6,275 +6,282 @@ - **Prerequisites**: Tensor, Activations, Layers, Networks, DataLoader, Autograd, Optimizers modules - **Next Steps**: Compression, Kernels, Benchmarking, MLOps modules -**Build the complete training pipeline that brings all TinyTorch components together** +Build the complete training pipeline that brings all TinyTorch components together. This capstone module orchestrates data loading, model forward passes, loss computation, backpropagation, and optimization into the end-to-end training workflows that power modern AI systems. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand loss functions and how they guide neural network training -- Implement essential loss functions: MSE, CrossEntropy, and BinaryCrossEntropy -- Build evaluation metrics for classification and regression tasks -- Create a complete training loop that orchestrates the entire training process -- Master training workflows with validation, logging, and progress tracking +By the end of this module, you will be able to: + +- **Design complete training architectures**: Orchestrate all ML components into cohesive training systems +- **Implement essential loss functions**: Build MSE, CrossEntropy, and BinaryCrossEntropy from mathematical foundations +- **Create evaluation frameworks**: Develop metrics systems for classification, regression, and model performance assessment +- **Build production training loops**: Implement robust training workflows with validation, logging, and progress tracking +- **Master training dynamics**: Understand convergence, overfitting, generalization, and optimization in real scenarios ## ๐Ÿง  Build โ†’ Use โ†’ Optimize -This module follows the TinyTorch pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: -1. **Build**: Loss functions, metrics, and training orchestration components -2. **Use**: Train complete neural networks on real datasets -3. **Optimize**: Analyze training dynamics and improve performance +1. **Build**: Implement loss functions, evaluation metrics, and complete training orchestration systems +2. **Use**: Train end-to-end neural networks on real datasets with full pipeline automation +3. **Optimize**: Analyze training dynamics, debug convergence issues, and optimize training performance for production ## ๐Ÿ“š What You'll Build -### **Loss Functions** +### Complete Training Pipeline ```python -# Regression loss -mse = MeanSquaredError() -loss = mse(predictions, targets) +# End-to-end training system +from tinytorch.core.training import Trainer +from tinytorch.core.losses import CrossEntropyLoss +from tinytorch.core.metrics import Accuracy -# Multi-class classification loss -ce = CrossEntropyLoss() -loss = ce(logits, class_indices) - -# Binary classification loss -bce = BinaryCrossEntropyLoss() -loss = bce(logits, binary_labels) -``` - -### **Evaluation Metrics** -```python -# Classification accuracy -accuracy = Accuracy() -acc = accuracy(predictions, true_labels) # Returns 0.0 to 1.0 - -# Regression metrics -mae = MeanAbsoluteError() -error = mae(predictions, targets) -``` - -### **Complete Training Pipeline** -```python -# Set up training components +# Define complete model architecture model = Sequential([ Dense(784, 128), ReLU(), - Dense(128, 64), ReLU(), + Dense(128, 64), ReLU(), Dense(64, 10), Softmax() ]) -optimizer = Adam(model.parameters, learning_rate=0.001) +# Configure training components +optimizer = Adam(model.parameters(), learning_rate=0.001) loss_fn = CrossEntropyLoss() metrics = [Accuracy()] -# Create trainer -trainer = Trainer(model, optimizer, loss_fn, metrics) +# Create and configure trainer +trainer = Trainer( + model=model, + optimizer=optimizer, + loss_fn=loss_fn, + metrics=metrics +) -# Train the model +# Train with comprehensive monitoring history = trainer.fit( - train_dataloader, - val_dataloader, - epochs=10, + train_dataloader=train_loader, + val_dataloader=val_loader, + epochs=50, verbose=True ) ``` -### **Training with Real Data** +### Loss Function Library ```python -# Load dataset -from tinytorch.core.dataloader import SimpleDataset, DataLoader +# Regression loss for continuous targets +mse_loss = MeanSquaredError() +regression_loss = mse_loss(predictions, continuous_targets) -# Create dataset -train_dataset = SimpleDataset(size=1000, num_features=784, num_classes=10) +# Multi-class classification loss +ce_loss = CrossEntropyLoss() +classification_loss = ce_loss(logits, class_indices) + +# Binary classification loss +bce_loss = BinaryCrossEntropyLoss() +binary_loss = bce_loss(sigmoid_outputs, binary_labels) + +# All losses support batch processing and gradient computation +loss.backward() # Automatic differentiation integration +``` + +### Evaluation Metrics System +```python +# Classification performance measurement +accuracy = Accuracy() +acc_score = accuracy(predictions, true_labels) # Returns 0.0 to 1.0 + +# Regression error measurement +mae = MeanAbsoluteError() +error = mae(predictions, targets) + +# Extensible metric framework +class CustomMetric: + def __call__(self, y_pred, y_true): + # Implement custom evaluation logic + return custom_score + +metrics = [Accuracy(), CustomMetric()] +trainer = Trainer(model, optimizer, loss_fn, metrics) +``` + +### Real-World Training Workflows +```python +# Train on CIFAR-10 with full pipeline +from tinytorch.core.dataloader import CIFAR10Dataset, DataLoader + +# Load and prepare data +train_dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True) train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True) +val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False) -# Train on real data -history = trainer.fit(train_loader, epochs=50) +# Configure CNN for computer vision +cnn_model = Sequential([ + Conv2D(3, 16, kernel_size=3), ReLU(), + MaxPool2D(kernel_size=2), + Conv2D(16, 32, kernel_size=3), ReLU(), + Flatten(), + Dense(32 * 13 * 13, 128), ReLU(), + Dense(128, 10) +]) -# Analyze training -print(f"Final training loss: {history['train_loss'][-1]:.4f}") -print(f"Final training accuracy: {history['train_accuracy'][-1]:.4f}") +# Train with monitoring and validation +trainer = Trainer(cnn_model, Adam(cnn_model.parameters()), CrossEntropyLoss(), [Accuracy()]) +history = trainer.fit(train_loader, val_loader, epochs=100) + +# Analyze training results +print(f"Final train accuracy: {history['train_accuracy'][-1]:.4f}") +print(f"Final val accuracy: {history['val_accuracy'][-1]:.4f}") ``` ## ๐Ÿš€ Getting Started ### Prerequisites -- Complete Modules 1-8: Setup through Optimizers โœ… -- Understand backpropagation and gradient descent -- Familiar with classification and regression tasks +Ensure you have completed the entire TinyTorch foundation: -### Quick Start ```bash -# Navigate to the training module -cd modules/source/09_training +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -# Open the development notebook -jupyter lab training_dev.py - -# Or use the TinyTorch CLI -tito module info training -tito module test training +# Verify all prerequisite modules (this is the capstone!) +tito test --module tensor +tito test --module activations +tito test --module layers +tito test --module networks +tito test --module dataloader +tito test --module autograd +tito test --module optimizers ``` -## ๐Ÿ“– Core Concepts +### Development Workflow +1. **Open the development file**: `modules/source/10_training/training_dev.py` +2. **Implement loss functions**: Build MSE, CrossEntropy, and BinaryCrossEntropy with proper gradients +3. **Create metrics system**: Develop Accuracy and extensible evaluation framework +4. **Build Trainer class**: Orchestrate training loop with validation and monitoring +5. **Test end-to-end training**: Apply complete pipeline to real datasets and problems +6. **Export and verify**: `tito export --module training && tito test --module training` -### **Loss Functions: The Training Signal** -Loss functions measure how far our predictions are from the true values: +## ๐Ÿงช Testing Your Implementation -- **MSE**: For regression tasks, penalizes large errors heavily -- **CrossEntropy**: For classification, works with softmax outputs -- **BinaryCrossEntropy**: For binary classification, works with sigmoid outputs +### Comprehensive Test Suite +Run the full test suite to verify complete training system functionality: -### **Metrics: Human-Interpretable Performance** -Metrics provide understandable measures of model performance: +```bash +# TinyTorch CLI (recommended) +tito test --module training -- **Accuracy**: Fraction of correct predictions -- **Precision**: Of positive predictions, how many were correct? -- **Recall**: Of actual positives, how many were found? +# Direct pytest execution +python -m pytest tests/ -k training -v +``` -### **Training Loop: Orchestrating Learning** -The training loop coordinates all components: +### Test Coverage Areas +- โœ… **Loss Function Implementation**: Verify mathematical correctness and gradient computation +- โœ… **Metrics System**: Test accuracy calculation and extensible framework +- โœ… **Training Loop Orchestration**: Ensure proper coordination of all components +- โœ… **End-to-End Training**: Verify complete workflows on real datasets +- โœ… **Convergence Analysis**: Test training dynamics and optimization behavior -1. **Forward Pass**: Model makes predictions -2. **Loss Computation**: Measure prediction quality -3. **Backward Pass**: Compute gradients -4. **Parameter Update**: Improve model weights -5. **Validation**: Monitor generalization performance - -### **Training Dynamics** -Understanding how training behaves: - -- **Overfitting**: Model memorizes training data -- **Underfitting**: Model too simple to learn patterns -- **Convergence**: Loss stops decreasing -- **Validation**: Monitoring generalization - -## ๐Ÿ”ฌ Advanced Features - -### **Training Monitoring** +### Inline Testing & Training Analysis +The module includes comprehensive training validation and convergence monitoring: ```python -# Track training progress -history = trainer.fit(train_loader, val_loader, epochs=100) +# Example inline test output +๐Ÿ”ฌ Unit Test: CrossEntropy loss function... +โœ… Mathematical correctness verified +โœ… Gradient computation working +โœ… Batch processing supported +๐Ÿ“ˆ Progress: Loss Functions โœ“ -# Plot training curves -import matplotlib.pyplot as plt -plt.plot(history['train_loss'], label='Training Loss') -plt.plot(history['val_loss'], label='Validation Loss') -plt.legend() -plt.show() +# Training monitoring +๐Ÿ”ฌ Unit Test: Complete training pipeline... +โœ… Trainer orchestrates all components correctly +โœ… Training loop converges on test problem +โœ… Validation monitoring working +๐Ÿ“ˆ Progress: End-to-End Training โœ“ + +# Real dataset training +๐Ÿ“Š Training on CIFAR-10 subset... +Epoch 1/10: train_loss=2.345, train_acc=0.234, val_loss=2.123, val_acc=0.278 +Epoch 5/10: train_loss=1.456, train_acc=0.567, val_loss=1.543, val_acc=0.523 +โœ… Model converging successfully ``` -### **Custom Metrics** +### Manual Testing Examples ```python -# Create custom metrics -class F1Score: - def __call__(self, y_pred, y_true): - # Implement F1 score calculation - pass +from training_dev import Trainer, CrossEntropyLoss, Accuracy +from networks_dev import Sequential +from layers_dev import Dense +from activations_dev import ReLU, Softmax +from optimizers_dev import Adam -# Use in training -trainer = Trainer(model, optimizer, loss_fn, metrics=[Accuracy(), F1Score()]) +# Test complete training on synthetic data +model = Sequential([Dense(4, 8), ReLU(), Dense(8, 3), Softmax()]) +optimizer = Adam(model.parameters(), learning_rate=0.01) +loss_fn = CrossEntropyLoss() +metrics = [Accuracy()] + +trainer = Trainer(model, optimizer, loss_fn, metrics) + +# Create simple dataset +from dataloader_dev import SimpleDataset, DataLoader +train_dataset = SimpleDataset(size=1000, num_features=4, num_classes=3) +train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True) + +# Train and monitor +history = trainer.fit(train_loader, epochs=20, verbose=True) +print(f"Training completed. Final accuracy: {history['train_accuracy'][-1]:.4f}") ``` -### **Training Strategies** -```python -# Learning rate scheduling -scheduler = StepLR(optimizer, step_size=10, gamma=0.1) +## ๐ŸŽฏ Key Concepts -# Early stopping -class EarlyStopping: - def __init__(self, patience=10): - self.patience = patience - self.best_loss = float('inf') - self.counter = 0 - - def __call__(self, val_loss): - if val_loss < self.best_loss: - self.best_loss = val_loss - self.counter = 0 - else: - self.counter += 1 - return self.counter >= self.patience -``` +### Real-World Applications +- **Production ML Systems**: Companies like Netflix, Google use similar training pipelines for recommendation and search systems +- **Research Workflows**: Academic researchers use training frameworks like this for experimental model development +- **MLOps Platforms**: Production training systems extend these patterns with distributed computing and monitoring +- **Edge AI Training**: Federated learning systems use similar orchestration patterns across distributed devices -## ๐Ÿ› ๏ธ Real-World Applications +### Training System Architecture +- **Loss Functions**: Mathematical objectives that define what the model should learn +- **Metrics**: Human-interpretable measures of model performance for monitoring and decision-making +- **Training Loop**: Orchestration pattern that coordinates data loading, forward passes, backward passes, and optimization +- **Validation Strategy**: Techniques for monitoring generalization and preventing overfitting -### **Computer Vision** -```python -# Image classification pipeline -model = Sequential([ - Conv2D((3, 3)), ReLU(), - flatten, - Dense(128), ReLU(), - Dense(10), Softmax() -]) +### Machine Learning Engineering +- **Training Dynamics**: Understanding convergence, overfitting, underfitting, and optimization landscapes +- **Hyperparameter Tuning**: Systematic approaches to learning rate, batch size, and architecture selection +- **Debugging Training**: Common failure modes and diagnostic techniques for training issues +- **Production Considerations**: Scalability, monitoring, reproducibility, and deployment readiness -trainer = Trainer(model, Adam(model.parameters), CrossEntropyLoss(), [Accuracy()]) -history = trainer.fit(cifar10_loader, epochs=50) -``` +### Systems Integration Patterns +- **Component Orchestration**: How to coordinate multiple ML components into cohesive systems +- **Error Handling**: Robust handling of training failures, data issues, and convergence problems +- **Monitoring and Logging**: Tracking training progress, performance metrics, and system health +- **Extensibility**: Design patterns that enable easy addition of new losses, metrics, and training strategies -### **Natural Language Processing** -```python -# Text classification -model = Sequential([ - Dense(vocab_size, 128), ReLU(), - Dense(128, 64), ReLU(), - Dense(64, num_classes), Softmax() -]) +## ๐ŸŽ‰ Ready to Build? -trainer = Trainer(model, SGD(model.parameters), CrossEntropyLoss(), [Accuracy()]) -history = trainer.fit(text_loader, epochs=20) -``` +You're about to complete the TinyTorch framework by building the training system that brings everything together! This is where all your hard work on tensors, layers, networks, data loading, gradients, and optimization culminates in a complete ML system. -### **Regression Tasks** -```python -# House price prediction -model = Sequential([ - Dense(features, 64), ReLU(), - Dense(64, 32), ReLU(), - Dense(32, 1) # Single output for regression -]) +Training is the heart of machine learningโ€”it's where models learn from data and become intelligent. You're building the same patterns used to train GPT, train computer vision models, and power production AI systems. Take your time, understand how all the pieces fit together, and enjoy creating something truly powerful! -trainer = Trainer(model, Adam(model.parameters), MeanSquaredError(), []) -history = trainer.fit(housing_loader, epochs=100) -``` +```{grid} 3 +:gutter: 3 +:margin: 2 -## ๐Ÿ“ˆ Performance Optimization +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/10_training/training_dev.py +:class-title: text-center +:class-body: text-center -### **Batch Size Selection** -- **Small batches**: More updates, noisier gradients -- **Large batches**: Fewer updates, smoother gradients -- **Sweet spot**: Usually 32-256 depending on dataset +Interactive development environment -### **Learning Rate Tuning** -- **Too high**: Training diverges or oscillates -- **Too low**: Training is slow or gets stuck -- **Adaptive methods**: Adam often works well out of the box +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/10_training/training_dev.ipynb +:class-title: text-center +:class-body: text-center -### **Regularization** -- **Dropout**: Randomly disable neurons during training -- **Weight decay**: L2 regularization on parameters -- **Early stopping**: Stop when validation performance plateaus +Google Colab notebook -## ๐ŸŽฏ Module Completion +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/10_training/training_dev.py +:class-title: text-center +:class-body: text-center -### **What You've Built** -โœ… **Complete loss function library**: MSE, CrossEntropy, BinaryCrossEntropy -โœ… **Evaluation metrics**: Accuracy and extensible metric framework -โœ… **Training orchestration**: Full-featured Trainer class -โœ… **Real-world pipeline**: Train models on actual datasets -โœ… **Monitoring tools**: Track training progress and performance - -### **Skills Developed** -โœ… **Training loop design**: Coordinate all training components -โœ… **Loss function implementation**: Measure prediction quality -โœ… **Metric computation**: Evaluate model performance -โœ… **Training dynamics**: Understand convergence and optimization -โœ… **Production workflows**: Build scalable training pipelines - -### **Next Steps** -1. **Export your training module**: `tito export training` -2. **Train a complete model**: Use all TinyTorch components together -3. **Explore advanced topics**: Regularization, scheduling, ensembles -4. **Build production pipelines**: Scale training to larger datasets - -**Ready for the final stretch?** Your training module completes the core TinyTorch framework. Next up: compression, kernels, and MLOps! ๐Ÿš€ \ No newline at end of file +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/11_compression/README.md b/modules/source/11_compression/README.md index ff54df1a..7c0b0ac6 100644 --- a/modules/source/11_compression/README.md +++ b/modules/source/11_compression/README.md @@ -6,179 +6,272 @@ - **Prerequisites**: Networks, Training modules - **Next Steps**: Kernels, MLOps modules +Build model compression systems that make neural networks smaller, faster, and more efficient for real-world deployment. This module teaches the optimization techniques that bridge the gap between research-quality models and production-ready AI systems. + ## ๐ŸŽฏ Learning Objectives -- Understand model size and deployment constraints in real systems -- Implement magnitude-based pruning to remove unimportant weights -- Master quantization for 75% memory reduction (FP32 โ†’ INT8) -- Build knowledge distillation for training compact models -- Create structured pruning to optimize network architectures -- Compare compression techniques and their trade-offs -## ๐Ÿง  Overview -This module teaches students to make neural networks smaller, faster, and more efficient for real-world deployment. Students implement four core compression techniques and learn to balance accuracy with efficiency. +By the end of this module, you will be able to: -## Educational Flow +- **Understand deployment constraints**: Analyze model size, memory usage, and computational requirements for real-world systems +- **Implement pruning techniques**: Build magnitude-based and structured pruning to remove unimportant weights and neurons +- **Master quantization methods**: Reduce memory usage by 75% through FP32 โ†’ INT8 precision reduction +- **Apply knowledge distillation**: Train compact models using larger teacher models for better performance +- **Design compression strategies**: Combine techniques optimally for different deployment scenarios and constraints -### Step 1: Understanding Model Size -- **Concept**: Parameter counting and memory footprint analysis -- **Implementation**: `CompressionMetrics` class for model analysis -- **Learning**: Foundation for compression decision-making +## ๐Ÿง  Build โ†’ Use โ†’ Optimize -### Step 2: Magnitude-Based Pruning -- **Concept**: Remove weights with smallest absolute values -- **Implementation**: `prune_weights_by_magnitude()` and sparsity calculation -- **Learning**: Sparsity patterns and accuracy vs compression trade-offs +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: -### Step 3: Quantization Experiments -- **Concept**: Reduce precision from FP32 to INT8 for memory efficiency -- **Implementation**: `quantize_layer_weights()` with scale/offset mapping -- **Learning**: Numerical precision impact on model performance +1. **Build**: Implement pruning, quantization, knowledge distillation, and structured optimization from engineering principles +2. **Use**: Apply compression techniques to real neural networks with accuracy vs efficiency analysis +3. **Optimize**: Combine compression methods strategically for production deployment scenarios with specific constraints -### Step 4: Knowledge Distillation -- **Concept**: Large models teach small models through soft targets -- **Implementation**: `DistillationLoss` with temperature scaling -- **Learning**: Advanced training techniques for compact models +## ๐Ÿ“š What You'll Build -### Step 5: Structured Pruning -- **Concept**: Remove entire neurons/channels, not just weights -- **Implementation**: `prune_layer_neurons()` with importance scoring -- **Learning**: Architecture optimization and cascade effects +### Model Compression Analysis System +```python +# Comprehensive model analysis for compression planning +metrics = CompressionMetrics() -### Step 6: Comprehensive Comparison -- **Concept**: Combine techniques for maximum efficiency -- **Implementation**: Integrated compression pipeline -- **Learning**: Systems thinking for production deployment +# Analyze original model +original_size = metrics.calculate_model_size(model) +param_count = metrics.count_parameters(model) +weight_dist = metrics.analyze_weight_distribution(model) -## Key Components - -### CompressionMetrics -- **Purpose**: Analyze model size and parameter distribution -- **Methods**: `count_parameters()`, `calculate_model_size()`, `analyze_weight_distribution()` -- **Usage**: Foundation for compression target selection - -### Pruning Functions -- **Purpose**: Remove unimportant weights and neurons -- **Methods**: `prune_weights_by_magnitude()`, `prune_model_by_magnitude()`, `calculate_sparsity()` -- **Usage**: Reduce model size while maintaining performance - -### Quantization Functions -- **Purpose**: Reduce memory usage through lower precision -- **Methods**: `quantize_layer_weights()`, `dequantize_layer_weights()` -- **Usage**: 75% memory reduction for mobile deployment - -### Knowledge Distillation -- **Purpose**: Train compact models with teacher guidance -- **Methods**: `DistillationLoss`, `train_with_distillation()` -- **Usage**: Achieve better small model performance - -### Structured Pruning -- **Purpose**: Remove entire neurons for actual speedup -- **Methods**: `prune_layer_neurons()`, `compute_neuron_importance()` -- **Usage**: Architecture optimization and hardware efficiency - -## Real-World Applications - -### Mobile AI Deployment -- **Constraint**: Models must be < 10MB for smartphone apps -- **Solution**: Combine pruning and quantization for 90% size reduction -- **Examples**: Google Translate offline, mobile camera AI - -### Edge Computing -- **Constraint**: Severe memory and compute limitations -- **Solution**: Structured pruning for actual inference speedup -- **Examples**: IoT sensors, smart cameras, voice assistants - -### Cost Optimization -- **Constraint**: Expensive cloud inference at scale -- **Solution**: Reduce model size for lower compute costs -- **Examples**: Production recommendation systems, search engines - -### Battery Efficiency -- **Constraint**: Wearable devices need long battery life -- **Solution**: Quantization and pruning for energy savings -- **Examples**: Smartwatches, fitness trackers, AR glasses - -## Industry Connections - -### MobileNet Architecture -- **Concept**: Depthwise separable convolutions for efficiency -- **Connection**: Structured optimization for mobile deployment -- **Learning**: Architecture design affects compression potential - -### DistilBERT -- **Concept**: 60% smaller than BERT with 97% performance -- **Connection**: Knowledge distillation for language models -- **Learning**: Teacher-student training for different domains - -### TinyML Movement -- **Concept**: ML on microcontrollers (< 1MB models) -- **Connection**: Extreme compression for embedded systems -- **Learning**: Efficiency requirements for edge deployment - -### Neural Architecture Search -- **Concept**: Automated model design for efficiency -- **Connection**: Structured pruning as architecture optimization -- **Learning**: Automated techniques for compression - -## Assessment Criteria - -### Technical Implementation (40%) -- Correctly implement 4 compression techniques -- Handle edge cases and error conditions -- Provide comprehensive statistics and analysis - -### Understanding Trade-offs (30%) -- Explain accuracy vs efficiency spectrum -- Identify appropriate techniques for different constraints -- Analyze compression effectiveness quantitatively - -### Real-World Application (30%) -- Connect compression to deployment scenarios -- Understand hardware and system constraints -- Design compression strategies for specific use cases - -## Next Steps - -### Module 11: Kernels -- **Connection**: Hardware-aware optimization builds on compression -- **Skills**: GPU kernels, SIMD operations, memory optimization -- **Application**: Implement efficient compressed model inference - -### Module 12: Benchmarking -- **Connection**: Measure compression effectiveness systematically -- **Skills**: Performance profiling, accuracy measurement, A/B testing -- **Application**: Evaluate compression trade-offs in production - -### Module 13: MLOps -- **Connection**: Deploy compressed models in production systems -- **Skills**: Model versioning, monitoring, continuous optimization -- **Application**: Production-ready compressed model deployment - -## File Structure -``` -10_compression/ -โ”œโ”€โ”€ compression_dev.py # Main development notebook -โ”œโ”€โ”€ module.yaml # Module configuration -โ”œโ”€โ”€ README.md # This file -โ””โ”€โ”€ tests/ # Additional test files (if needed) +print(f"Original model: {original_size:.2f} MB, {param_count:,} parameters") +print(f"Weight distribution: mean={weight_dist['mean']:.4f}, std={weight_dist['std']:.4f}") ``` -## Getting Started +### Pruning Systems for Model Sparsity +```python +# Magnitude-based pruning: remove smallest weights +pruned_model = prune_model_by_magnitude(model, sparsity=0.5) # Remove 50% of weights +sparsity = calculate_sparsity(pruned_model) +print(f"Achieved sparsity: {sparsity:.2%}") -1. **Review Dependencies**: Ensure modules 01, 02, 04, 05, 10 are complete -2. **Open Development File**: `compression_dev.py` -3. **Follow Educational Flow**: Work through Steps 1-6 sequentially -4. **Test Thoroughly**: Run all inline tests as you progress -5. **Export to Package**: Use `tito export 10_compression` when complete +# Structured pruning: remove entire neurons/channels +optimized_model = prune_layer_neurons(model, layer_idx=0, neurons_to_remove=32) +print(f"Removed 32 neurons from layer 0") -## Key Takeaways +# Sparsity analysis and performance impact +original_acc = evaluate_model(model, test_loader) +pruned_acc = evaluate_model(pruned_model, test_loader) +print(f"Accuracy: {original_acc:.4f} โ†’ {pruned_acc:.4f} ({pruned_acc-original_acc:+.4f})") +``` -Students completing this module will: -- **Understand** the efficiency requirements of production AI systems -- **Implement** four essential compression techniques from scratch -- **Analyze** accuracy vs efficiency trade-offs quantitatively -- **Apply** compression strategies to real neural networks -- **Connect** compression to mobile, edge, and production deployment -- **Prepare** for advanced optimization and production deployment modules +### Quantization for Memory Efficiency +```python +# Quantize model weights from FP32 to INT8 +quantized_model = quantize_model_weights(model) +compressed_size = metrics.calculate_model_size(quantized_model) -This module bridges the gap between research-quality models and production-ready AI systems, teaching the essential skills for deploying AI in resource-constrained environments. \ No newline at end of file +print(f"Size reduction: {original_size:.2f} MB โ†’ {compressed_size:.2f} MB") +print(f"Compression ratio: {original_size/compressed_size:.1f}x smaller") + +# Test quantization impact on accuracy +quantized_acc = evaluate_model(quantized_model, test_loader) +print(f"Quantization accuracy impact: {quantized_acc-original_acc:+.4f}") +``` + +### Knowledge Distillation for Compact Models +```python +# Train small model using large teacher model +teacher_model = load_pretrained_large_model() +student_model = create_compact_model(compression_ratio=0.25) # 4x smaller + +# Distillation training with temperature scaling +distillation_loss = DistillationLoss(temperature=4.0, alpha=0.7) + +# Training loop with teacher guidance +for batch_inputs, batch_labels in train_loader: + teacher_outputs = teacher_model(batch_inputs) + student_outputs = student_model(batch_inputs) + + # Combined loss: distillation + task loss + loss = distillation_loss(student_outputs, teacher_outputs, batch_labels) + loss.backward() + optimizer.step() + +print(f"Student model size: {metrics.calculate_model_size(student_model):.2f} MB") +print(f"Student accuracy: {evaluate_model(student_model, test_loader):.4f}") +``` + +### Comprehensive Compression Pipeline +```python +# End-to-end compression with multiple techniques +def compress_for_mobile_deployment(model, target_size_mb=5.0): + """Compress model for mobile deployment under 5MB constraint""" + + # Step 1: Structured pruning for architecture optimization + model = prune_redundant_neurons(model, importance_threshold=0.1) + + # Step 2: Magnitude-based pruning for sparsity + model = prune_model_by_magnitude(model, sparsity=0.6) + + # Step 3: Quantization for memory reduction + model = quantize_model_weights(model) + + # Step 4: Verify size constraint + final_size = CompressionMetrics().calculate_model_size(model) + print(f"Final compressed model: {final_size:.2f} MB") + + return model + +mobile_model = compress_for_mobile_deployment(trained_model) +``` + +## ๐Ÿš€ Getting Started + +### Prerequisites +Ensure you have mastered the training foundation: + +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh + +# Verify prerequisite modules +tito test --module networks +tito test --module training +``` + +### Development Workflow +1. **Open the development file**: `modules/source/11_compression/compression_dev.py` +2. **Implement compression metrics**: Build model analysis tools for size and parameter counting +3. **Create pruning algorithms**: Implement magnitude-based and structured pruning techniques +4. **Build quantization system**: Add FP32 โ†’ INT8 weight quantization with scale/offset mapping +5. **Add knowledge distillation**: Implement teacher-student training for compact models +6. **Export and verify**: `tito export --module compression && tito test --module compression` + +## ๐Ÿงช Testing Your Implementation + +### Comprehensive Test Suite +Run the full test suite to verify compression system functionality: + +```bash +# TinyTorch CLI (recommended) +tito test --module compression + +# Direct pytest execution +python -m pytest tests/ -k compression -v +``` + +### Test Coverage Areas +- โœ… **Compression Metrics**: Verify accurate model size and parameter analysis +- โœ… **Pruning Algorithms**: Test magnitude-based and structured pruning correctness +- โœ… **Quantization System**: Ensure proper FP32 โ†” INT8 conversion and accuracy preservation +- โœ… **Knowledge Distillation**: Verify teacher-student training and loss computation +- โœ… **Integrated Compression**: Test combined techniques on real neural networks + +### Inline Testing & Compression Analysis +The module includes comprehensive compression validation and performance analysis: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: Model compression metrics... +โœ… Parameter counting accurate +โœ… Model size calculation correct +โœ… Weight distribution analysis working +๐Ÿ“ˆ Progress: Compression Analysis โœ“ + +# Pruning validation +๐Ÿ”ฌ Unit Test: Magnitude-based pruning... +โœ… Smallest weights identified correctly +โœ… Sparsity calculation accurate +โœ… Model functionality preserved +๐Ÿ“ˆ Progress: Pruning Systems โœ“ + +# Quantization testing +๐Ÿ”ฌ Unit Test: Weight quantization... +โœ… FP32 โ†’ INT8 conversion correct +โœ… Dequantization recovers values +โœ… 75% memory reduction achieved +๐Ÿ“ˆ Progress: Quantization โœ“ +``` + +### Manual Testing Examples +```python +from compression_dev import CompressionMetrics, prune_model_by_magnitude, quantize_model_weights +from networks_dev import Sequential +from layers_dev import Dense +from activations_dev import ReLU + +# Create test model +model = Sequential([ + Dense(784, 128), ReLU(), + Dense(128, 64), ReLU(), + Dense(64, 10) +]) + +# Analyze original model +metrics = CompressionMetrics() +original_size = metrics.calculate_model_size(model) +original_params = metrics.count_parameters(model) +print(f"Original: {original_size:.2f} MB, {original_params:,} parameters") + +# Test pruning +pruned_model = prune_model_by_magnitude(model, sparsity=0.5) +pruned_size = metrics.calculate_model_size(pruned_model) +print(f"After 50% pruning: {pruned_size:.2f} MB ({original_size/pruned_size:.1f}x smaller)") + +# Test quantization +quantized_model = quantize_model_weights(model) +quantized_size = metrics.calculate_model_size(quantized_model) +print(f"After quantization: {quantized_size:.2f} MB ({original_size/quantized_size:.1f}x smaller)") +``` + +## ๐ŸŽฏ Key Concepts + +### Real-World Applications +- **Mobile AI**: Smartphone apps require models under 10MB for fast download and inference +- **Edge Computing**: IoT devices have severe memory constraints requiring aggressive compression +- **Cloud Cost Optimization**: Reducing model size decreases inference costs at scale +- **Autonomous Systems**: Real-time requirements demand efficient models for safety-critical applications + +### Compression Techniques +- **Magnitude-based Pruning**: Remove weights with smallest absolute values to create sparse networks +- **Structured Pruning**: Remove entire neurons/channels for actual hardware speedup benefits +- **Quantization**: Reduce precision from FP32 to INT8 for 75% memory reduction +- **Knowledge Distillation**: Transfer knowledge from large teacher to small student models + +### Production Deployment Considerations +- **Hardware Constraints**: Different devices have different memory, compute, and energy limitations +- **Accuracy vs Efficiency Trade-offs**: Balancing model performance with deployment requirements +- **Inference Speed**: Compression techniques that actually improve runtime performance +- **Model Serving**: Considerations for batch processing, latency, and throughput + +### Systems Engineering Patterns +- **Compression Pipeline Design**: Sequential application of techniques for maximum benefit +- **Performance Profiling**: Measuring actual improvements in memory, speed, and energy usage +- **Quality Assurance**: Maintaining model accuracy while achieving compression targets +- **Deployment Validation**: Testing compressed models in realistic production scenarios + +## ๐ŸŽ‰ Ready to Build? + +You're about to master the optimization techniques that make AI practical for real-world deployment! From the smartphone in your pocket to autonomous vehicles, they all depend on compressed models that balance intelligence with efficiency. + +This module teaches you the systems engineering that separates research prototypes from production AI. You'll learn to think like a deployment engineer, balancing accuracy against constraints and building systems that work in the real world. Take your time, understand the trade-offs, and enjoy building AI that actually ships! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/11_compression/compression_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/11_compression/compression_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/11_compression/compression_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/12_kernels/README.md b/modules/source/12_kernels/README.md index d93d1adf..d1ed1c3b 100644 --- a/modules/source/12_kernels/README.md +++ b/modules/source/12_kernels/README.md @@ -6,143 +6,283 @@ - **Prerequisites**: All previous modules (01-11), especially Compression - **Next Steps**: Benchmarking, MLOps modules -**Bridge the gap between algorithmic optimization and hardware-level performance engineering** +Bridge the gap between algorithmic optimization and hardware-level performance engineering. This module teaches the systems programming skills that make ML frameworks fastโ€”moving beyond NumPy's black box to understand how computation really works on modern hardware. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Understand how to implement custom ML operations beyond NumPy -- Apply SIMD vectorization and CPU optimization techniques -- Optimize memory layout and access patterns for cache efficiency -- Implement GPU-style parallel computing concepts -- Build comprehensive performance profiling and benchmarking tools -- Create hardware-optimized operations for quantized and pruned models +By the end of this module, you will be able to: -## ๐Ÿ”— Connection to Previous Modules - -### What You Already Know -- **Compression (Module 10)**: *What* to optimize (model size, computation) -- **Layers (Module 03)**: Basic matrix multiplication with `matmul()` -- **Training (Module 09)**: High-level optimization workflows -- **Networks (Module 04)**: How operations compose into architectures - -### The Performance Gap -Students understand **algorithmic optimization** but not **hardware optimization**: -- โœ… **Algorithmic**: Pruning, quantization, knowledge distillation -- โŒ **Hardware**: Memory layout, vectorization, parallel processing +- **Master hardware-aware programming**: Understand CPU cache hierarchies, SIMD vectorization, and memory layout optimization +- **Implement custom ML operations**: Build matrix multiplication, activations, and batch processing from scratch with performance awareness +- **Apply parallel computing principles**: Use multi-core processing and GPU-style parallelism for ML workloads +- **Optimize compressed models**: Create hardware-efficient operations for quantized and pruned neural networks +- **Build performance engineering workflows**: Develop profiling, benchmarking, and optimization methodologies ## ๐Ÿง  Build โ†’ Use โ†’ Optimize -This module follows the **"Build โ†’ Use โ†’ Optimize"** pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Optimize** framework: -### 1. **Build**: Custom Operations -- Move beyond NumPy's black box implementations -- Implement specialized matrix multiplication and activations -- Understand the computational patterns underlying ML - -### 2. **Use**: Performance Optimization -- Apply SIMD vectorization for CPU optimization -- Implement cache-friendly memory layouts -- Build GPU-style parallel computing concepts - -### 3. **Optimize**: Real-World Integration -- Profile and benchmark performance improvements -- Integrate with compressed models from Module 10 -- Apply systematic evaluation to validate optimizations +1. **Build**: Implement custom ML operations with hardware awareness, moving beyond NumPy to understand computational patterns +2. **Use**: Apply SIMD vectorization, cache optimization, and parallel processing to real ML workloads +3. **Optimize**: Profile performance systematically, integrate with compressed models, and achieve measurable speedups ## ๐Ÿ“š What You'll Build -### Core Operations -- **Matrix Multiplication**: Custom `matmul_baseline()` and `cache_friendly_matmul()` -- **Activation Functions**: Vectorized `vectorized_relu()` and `parallel_relu()` -- **Batch Processing**: `parallel_batch_processing()` with multiprocessing -- **Quantized Operations**: `quantized_matmul()` and `quantized_relu()` +### Hardware-Optimized Core Operations +```python +# Custom matrix multiplication with cache awareness +import numba +from multiprocessing import Pool -### Performance Tools -- **Profiling**: `profile_operation()` for detailed timing analysis -- **Benchmarking**: `benchmark_operation()` for statistical validation -- **Memory Analysis**: Cache-friendly data layout optimization -- **Parallel Computing**: Multi-core processing patterns +# Baseline implementation for understanding +def matmul_baseline(A, B): + """Reference implementation showing the core computation""" + rows_A, cols_A = A.shape + rows_B, cols_B = B.shape + C = np.zeros((rows_A, cols_B)) + + for i in range(rows_A): + for j in range(cols_B): + for k in range(cols_A): + C[i, j] += A[i, k] * B[k, j] + return C -## ๐Ÿ› ๏ธ Key Components +# Cache-friendly optimized version +def cache_friendly_matmul(A, B): + """Optimized for memory access patterns and cache efficiency""" + # Implementation with blocked matrix multiplication + # and memory-friendly access patterns + pass -### Hardware-Optimized Operations -- **Purpose**: Implement custom ML operations with hardware awareness -- **Methods**: `matmul_baseline()`, `vectorized_relu()`, `cache_friendly_matmul()` -- **Learning**: Understanding computational patterns beyond NumPy +# Performance comparison +baseline_time = profile_operation(matmul_baseline, A, B) +optimized_time = profile_operation(cache_friendly_matmul, A, B) +speedup = baseline_time / optimized_time +print(f"Speedup: {speedup:.2f}x") +``` -### Parallel Processing Framework -- **Purpose**: Multi-core optimization for batch operations -- **Methods**: `parallel_batch_processing()`, `parallel_relu()` -- **Learning**: GPU-style parallel computing concepts +### SIMD Vectorized Operations +```python +# Vectorized activation functions +@numba.jit(nopython=True) +def vectorized_relu(x): + """SIMD-optimized ReLU using numba compilation""" + return np.maximum(0, x) -### Quantization Integration -- **Purpose**: Hardware-optimized operations for compressed models -- **Methods**: `quantized_matmul()`, `quantized_relu()` -- **Learning**: Bridging compression and performance optimization +# Parallel batch processing +def parallel_batch_processing(batch_data, operation, num_workers=4): + """Multi-core processing for batch operations""" + with Pool(num_workers) as pool: + results = pool.map(operation, batch_data) + return np.array(results) -### Performance Profiling -- **Purpose**: Systematic measurement and validation of optimizations -- **Methods**: `profile_operation()`, `benchmark_operation()` -- **Learning**: Evidence-based performance engineering +# Compare single-threaded vs parallel +single_time = profile_operation(sequential_relu, large_batch) +parallel_time = profile_operation(parallel_relu, large_batch) +efficiency = single_time / (parallel_time * num_cores) +print(f"Parallel efficiency: {efficiency:.2f}") +``` -## ๐ŸŒŸ Real-World Applications +### Quantized Operation Optimization +```python +# Hardware-optimized quantized operations +def quantized_matmul(A_int8, B_int8, scale_A, scale_B, zero_point_A, zero_point_B): + """INT8 matrix multiplication with proper scaling""" + # Use INT32 accumulator to prevent overflow + C_int32 = np.dot(A_int8.astype(np.int32), B_int8.astype(np.int32)) + + # Apply scaling and zero-point corrections + scale_C = scale_A * scale_B + C_float = scale_C * (C_int32 - zero_point_corrections) + + return C_float -### Industry Examples -- **Google TPUs**: Custom hardware for ML operations -- **Intel MKL**: Optimized math libraries for CPU performance -- **NVIDIA cuDNN**: GPU-accelerated neural network operations -- **Apple Neural Engine**: Hardware-specific ML acceleration +# Measure memory and compute benefits +fp32_memory = measure_memory_usage(model_fp32) +int8_memory = measure_memory_usage(model_int8) +memory_reduction = fp32_memory / int8_memory +print(f"Memory reduction: {memory_reduction:.1f}x") +``` -### Performance Patterns -- **Memory Layout**: Row-major vs column-major access patterns -- **Vectorization**: SIMD instructions for parallel computation -- **Cache Optimization**: Data locality for memory hierarchy -- **Parallel Processing**: Multi-core utilization strategies +### Performance Profiling Framework +```python +# Comprehensive operation profiling +class PerformanceProfiler: + def __init__(self): + self.results = {} + + def profile_operation(self, operation, *args, num_runs=100): + """Statistical profiling with multiple runs""" + times = [] + for _ in range(num_runs): + start = time.perf_counter() + result = operation(*args) + end = time.perf_counter() + times.append(end - start) + + return { + 'mean_time': np.mean(times), + 'std_time': np.std(times), + 'min_time': np.min(times), + 'max_time': np.max(times) + } + + def compare_operations(self, baseline_op, optimized_op, *args): + """Compare two implementations statistically""" + baseline_stats = self.profile_operation(baseline_op, *args) + optimized_stats = self.profile_operation(optimized_op, *args) + + speedup = baseline_stats['mean_time'] / optimized_stats['mean_time'] + significance = self.statistical_significance(baseline_stats, optimized_stats) + + return {'speedup': speedup, 'significant': significance} +``` ## ๐Ÿš€ Getting Started -### Prerequisites Check +### Prerequisites +Ensure you have mastered the optimization foundation: + ```bash -tito test --module compression # Should pass -tito status --module 11_kernels # Check module status +# Activate TinyTorch environment +source bin/activate-tinytorch.sh + +# Verify all prerequisite modules +tito test --module compression # Essential for integration +tito test --module training # Understanding of ML workflows ``` ### Development Workflow -```bash -cd modules/source/11_kernels -jupyter notebook kernels_dev.py # or edit directly -``` +1. **Open the development file**: `modules/source/12_kernels/kernels_dev.py` +2. **Implement baseline operations**: Build reference implementations for understanding +3. **Add hardware optimizations**: Apply SIMD, cache optimization, and parallelization +4. **Create quantized operations**: Build INT8 and hardware-efficient variants +5. **Build profiling tools**: Develop systematic performance measurement +6. **Export and verify**: `tito export --module kernels && tito test --module kernels` + +## ๐Ÿงช Testing Your Implementation + +### Comprehensive Test Suite +Run the full test suite to verify performance optimization functionality: -### Testing Your Implementation ```bash -# Test inline (within notebook) -# Run comprehensive tests +# TinyTorch CLI (recommended) tito test --module kernels + +# Direct pytest execution +python -m pytest tests/ -k kernels -v ``` -## ๐Ÿ“– Module Structure -``` -modules/source/11_kernels/ -โ”œโ”€โ”€ kernels_dev.py # Main development file -โ”œโ”€โ”€ README.md # This overview -โ””โ”€โ”€ module.yaml # Module configuration +### Test Coverage Areas +- โœ… **Operation Correctness**: Verify optimized operations produce identical results to baselines +- โœ… **Performance Improvements**: Measure and validate actual speedups from optimizations +- โœ… **Hardware Utilization**: Test SIMD usage, cache efficiency, and parallel scaling +- โœ… **Quantization Integration**: Verify INT8 operations maintain accuracy while improving performance +- โœ… **Profiling Accuracy**: Ensure performance measurement tools provide reliable statistics + +### Inline Testing & Performance Analysis +The module includes comprehensive performance validation and optimization verification: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: Cache-friendly matrix multiplication... +โœ… Correctness: Results match NumPy reference +โœ… Performance: 2.3x speedup over baseline +โœ… Memory efficiency: 40% reduction in cache misses +๐Ÿ“ˆ Progress: Optimized Matrix Operations โœ“ + +# SIMD vectorization testing +๐Ÿ”ฌ Unit Test: Vectorized ReLU implementation... +โœ… SIMD utilization: 8-wide vectors detected +โœ… Throughput: 4.1x improvement over scalar code +โœ… Batch scaling: Linear performance with data size +๐Ÿ“ˆ Progress: Vectorized Operations โœ“ + +# Quantization optimization +๐Ÿ”ฌ Unit Test: INT8 quantized operations... +โœ… Accuracy preservation: <0.1% degradation +โœ… Memory reduction: 4x smaller model size +โœ… Compute speedup: 2.8x faster inference +๐Ÿ“ˆ Progress: Quantized Kernels โœ“ ``` -## ๐Ÿ”— Integration Points +### Manual Testing Examples +```python +from kernels_dev import matmul_baseline, cache_friendly_matmul, PerformanceProfiler +import numpy as np -### Input from Previous Modules -- **Tensor operations** โ†’ Custom implementations -- **Compressed models** โ†’ Hardware-optimized execution -- **Training workflows** โ†’ Performance-critical operations +# Create test matrices +A = np.random.randn(1000, 500).astype(np.float32) +B = np.random.randn(500, 800).astype(np.float32) -### Output to Next Modules -- **Benchmarking** โ†’ Operations to evaluate systematically -- **MLOps** โ†’ Production-ready optimized operations -- **Complete system** โ†’ End-to-end performance optimization +# Compare implementations +profiler = PerformanceProfiler() +baseline_result = matmul_baseline(A, B) +optimized_result = cache_friendly_matmul(A, B) -## ๐ŸŽ“ Educational Philosophy +# Verify correctness +np.testing.assert_allclose(baseline_result, optimized_result, rtol=1e-5) +print("โœ… Optimized implementation matches baseline") -This module bridges the gap between **algorithmic understanding** and **systems performance**. Students learn that optimization isn't just about better algorithmsโ€”it's about understanding how algorithms interact with hardware to achieve real-world performance gains. +# Measure performance +comparison = profiler.compare_operations(matmul_baseline, cache_friendly_matmul, A, B) +print(f"Speedup: {comparison['speedup']:.2f}x") +print(f"Statistically significant: {comparison['significant']}") +``` -By the end, you'll think like a **performance engineer**, not just a machine learning practitioner. \ No newline at end of file +## ๐ŸŽฏ Key Concepts + +### Real-World Applications +- **PyTorch/TensorFlow**: Production ML frameworks use similar kernel optimization techniques +- **Intel MKL/OpenBLAS**: Optimized math libraries employ cache-friendly algorithms and SIMD instructions +- **NVIDIA cuDNN**: GPU libraries optimize memory access patterns and parallel computation +- **Google TPUs**: Custom hardware accelerators use similar quantization and optimization principles + +### Hardware Performance Fundamentals +- **CPU Cache Hierarchy**: L1/L2/L3 cache optimization through memory access pattern design +- **SIMD Vectorization**: Single Instruction Multiple Data processing for parallel computation +- **Memory Layout**: Row-major vs column-major access patterns and cache line utilization +- **Parallel Computing**: Multi-core CPU utilization and GPU-style parallel programming patterns + +### Optimization Techniques +- **Algorithmic Optimization**: Choosing algorithms that match hardware characteristics +- **Memory Optimization**: Cache-friendly data structures and access patterns +- **Vectorization**: SIMD instruction utilization for parallel arithmetic operations +- **Quantization Integration**: Hardware-efficient low-precision computation + +### Performance Engineering Methodology +- **Profiling-Driven Optimization**: Measure first, optimize second, validate third +- **Statistical Validation**: Ensuring performance improvements are statistically significant +- **Bottleneck Analysis**: Identifying and addressing the most impactful performance constraints +- **Hardware-Software Co-design**: Understanding hardware capabilities and designing software accordingly + +## ๐ŸŽ‰ Ready to Build? + +You're about to learn the systems programming skills that make modern ML frameworks fast! This is where computer science meets practical engineeringโ€”understanding how algorithms interact with hardware to achieve real performance gains. + +From smartphone AI to data center training, all efficient ML systems depend on the optimization techniques you're about to master. You'll think like a performance engineer, understanding not just what to compute but how to compute it efficiently. Take your time, profile everything, and enjoy building systems that are both intelligent and fast! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/12_kernels/kernels_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/12_kernels/kernels_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/12_kernels/kernels_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/13_benchmarking/README.md b/modules/source/13_benchmarking/README.md index d0bd1ff5..5f1bc343 100644 --- a/modules/source/13_benchmarking/README.md +++ b/modules/source/13_benchmarking/README.md @@ -4,154 +4,275 @@ - **Difficulty**: โญโญโญโญ Advanced - **Time Estimate**: 6-8 hours - **Prerequisites**: All previous modules (01-12), especially Kernels -- **Next Steps**: MLOps module (13) +- **Next Steps**: MLOps module (14) -**Learn to systematically evaluate ML systems using industry-standard benchmarking methodology** +Learn to systematically evaluate ML systems using industry-standard benchmarking methodology. This module teaches you to measure performance reliably, validate optimization claims, and create professional evaluation reports that meet research and industry standards. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Design systematic benchmarking experiments for ML systems -- Apply MLPerf-inspired patterns to evaluate model performance -- Implement statistical validation for benchmark results -- Create professional performance reports and comparisons -- Apply systematic evaluation to real ML projects +By the end of this module, you will be able to: -## ๐Ÿ”— Connection to Previous Modules - -### What You Already Know -- **Kernels (Module 11)**: *How* to optimize individual operations -- **Training (Module 09)**: End-to-end model training workflows -- **Compression (Module 10)**: Model optimization techniques -- **Networks (Module 04)**: Model architectures and complexity - -### The Evaluation Gap -Students understand **how to build** ML systems but not **how to evaluate** them systematically: -- โœ… **Implementation**: Can build tensors, layers, networks, optimizers -- โŒ **Evaluation**: Don't know how to measure performance reliably -- โœ… **Optimization**: Can implement kernels and compression -- โŒ **Validation**: Can't prove optimizations actually work +- **Design systematic benchmarking experiments**: Apply MLPerf-inspired methodology to evaluate ML system performance +- **Implement statistical validation**: Ensure benchmark results are statistically significant and reproducible +- **Create professional performance reports**: Generate industry-standard documentation for optimization claims +- **Apply evaluation methodology**: Systematically compare models, optimizations, and architectural choices +- **Debug performance systematically**: Use benchmarking to identify bottlenecks and validate improvements ## ๐Ÿง  Build โ†’ Use โ†’ Analyze -This module follows the **"Build โ†’ Use โ†’ Analyze"** pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Analyze** framework: -### 1. **Build**: Benchmarking Framework -- Understand the four-component MLPerf architecture -- Learn different benchmark scenarios (latency, throughput, server) -- Implement statistical validation for meaningful results +1. **Build**: Implement comprehensive benchmarking framework with MLPerf-inspired architecture and statistical validation +2. **Use**: Apply systematic evaluation to TinyTorch models, optimizations, and performance claims +3. **Analyze**: Generate professional reports, validate optimization effectiveness, and prepare results for presentations -### 2. **Use**: Systematic Evaluation -- Apply benchmarking to your TinyTorch models -- Compare different approaches systematically -- Validate optimization claims with proper methodology +## ๐Ÿ“š What You'll Build -### 3. **Analyze**: Professional Reporting -- Generate industry-standard performance reports -- Present results with statistical confidence -- Prepare for capstone project presentations - -## ๐ŸŽ“ Why This Matters - -### **Industry Reality** -Real ML engineers spend significant time on: -- **A/B testing**: Comparing model variants in production -- **Performance optimization**: Proving optimizations actually work -- **Research validation**: Demonstrating improvements over baselines -- **System design**: Choosing between architectural alternatives - -### **Professional Applications** -This module prepares you for: -- **ML project evaluation**: Systematic comparison against baselines -- **Performance presentations**: Professional reporting of results -- **Statistical validation**: Proving your improvements are significant -- **Research methodology**: Reproducible evaluation practices - -## ๐Ÿš€ Key Concepts - -### **MLPerf-Inspired Architecture** -- **System Under Test (SUT)**: Your ML model/system -- **Dataset**: Standardized evaluation data -- **Model**: The specific architecture being tested -- **Load Generator**: Controls how evaluation queries are sent - -### **Benchmark Scenarios** -- **Single-Stream**: Measures latency (mobile/edge use cases) -- **Server**: Measures throughput (production server use cases) -- **Offline**: Measures batch processing (data center use cases) - -### **Statistical Validation** -- **Confidence intervals**: Ensuring results are meaningful -- **Multiple runs**: Accounting for variability -- **Significance testing**: Proving improvements are real -- **Pitfall detection**: Avoiding common benchmarking mistakes - -## ๐Ÿ”ง What You'll Build - -### **1. TinyTorchPerf Framework** +### MLPerf-Inspired Benchmarking Framework ```python -from tinytorch.benchmarking import TinyTorchPerf +# Professional ML system evaluation +from tinytorch.core.benchmarking import TinyTorchPerf, StatisticalValidator -# Professional ML benchmarking +# Configure benchmark system benchmark = TinyTorchPerf() -benchmark.set_model(your_model) -benchmark.set_dataset('cifar10') +benchmark.set_model(your_trained_model) +benchmark.set_dataset('cifar10', subset_size=1000) +benchmark.set_metrics(['latency', 'throughput', 'accuracy']) -# Run different scenarios -results = benchmark.run_all_scenarios() +# Run comprehensive evaluation +results = benchmark.run_all_scenarios([ + 'single_stream', # Latency-focused (mobile/edge) + 'server', # Throughput-focused (production) + 'offline' # Batch processing (data center) +]) + +print(f"Single-stream latency: {results['single_stream']['latency']:.2f}ms") +print(f"Server throughput: {results['server']['throughput']:.0f} samples/sec") +print(f"Offline batch time: {results['offline']['batch_time']:.2f}s") ``` -### **2. Statistical Validator** +### Statistical Validation System ```python # Ensure statistically valid results -validator = StatisticalValidator() -validation = validator.validate_results(results) -if validation.significant: - print("โœ… Improvement is statistically significant") +validator = StatisticalValidator(confidence_level=0.95, min_runs=30) + +# Compare two models with statistical rigor +baseline_model = load_model("baseline_v1") +optimized_model = load_model("optimized_v2") + +comparison = validator.compare_models( + baseline_model, + optimized_model, + test_dataset, + metrics=['latency', 'accuracy'] +) + +if comparison['latency']['significant']: + speedup = comparison['latency']['improvement'] + confidence = comparison['latency']['confidence_interval'] + print(f"โœ… Speedup: {speedup:.2f}x (95% CI: {confidence[0]:.2f}-{confidence[1]:.2f})") +else: + print("โŒ Performance difference not statistically significant") ``` -### **3. Performance Reporter** +### Comprehensive Performance Reporter ```python -# Generate professional reports +# Generate professional evaluation reports +from tinytorch.core.benchmarking import PerformanceReporter + reporter = PerformanceReporter() -report = reporter.generate_report(results) -report.save_as_html("my_capstone_results.html") +report = reporter.generate_comprehensive_report({ + 'models': [baseline_model, optimized_model, compressed_model], + 'datasets': ['cifar10', 'imagenet_subset'], + 'scenarios': ['mobile', 'server', 'edge'], + 'optimizations': ['baseline', 'quantized', 'pruned', 'kernels'] +}) + +# Export professional documentation +report.save_as_html("performance_evaluation.html") +report.save_as_pdf("performance_evaluation.pdf") +report.save_summary_table("results_summary.csv") + +# Generate presentation slides +report.create_presentation_slides("optimization_results.pptx") ``` -## ๐Ÿ“ˆ Real-World Applications +### Real-World Evaluation Scenarios +```python +# Mobile deployment evaluation +mobile_benchmark = TinyTorchPerf() +mobile_benchmark.configure_mobile_scenario( + max_latency_ms=100, + battery_constraints=True, + memory_limit_mb=50 +) -### **Immediate Use Cases** -- **ML projects**: Systematic evaluation of your model implementations -- **Module integration**: Validate that your TinyTorch components work together -- **Performance optimization**: Prove your kernels actually improve performance +mobile_results = mobile_benchmark.evaluate_model(compressed_model) +mobile_feasible = mobile_results['meets_constraints'] -### **Career Applications** -- **Research**: Proper experimental methodology for papers -- **Industry**: A/B testing and performance optimization -- **Open source**: Contributing benchmarks to ML libraries +# Production server evaluation +server_benchmark = TinyTorchPerf() +server_benchmark.configure_server_scenario( + target_throughput=1000, # requests/second + max_latency_p99=50, # 99th percentile latency + concurrent_users=100 +) -## ๐ŸŽฏ Success Metrics +server_results = server_benchmark.evaluate_model(optimized_model) +production_ready = server_results['meets_sla'] +``` -By the end of this module, you should be able to: -- [ ] Design a systematic benchmark for any ML system -- [ ] Apply MLPerf principles to your own evaluations -- [ ] Generate statistically valid performance comparisons -- [ ] Create professional reports suitable for presentations -- [ ] Identify and avoid common benchmarking pitfalls +## ๐Ÿš€ Getting Started -## ๐Ÿ”„ Connection to Module 13 (MLOps) +### Prerequisites +Ensure you have built the complete TinyTorch system: -**Benchmarking** โ†’ **Production Monitoring** -- Benchmarking establishes baselines for production systems -- Monitoring detects when production deviates from benchmarks -- Both use similar metrics and statistical validation +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh -## ๐Ÿ“š Resources +# Verify prerequisite modules (comprehensive system needed) +tito test --module kernels # Performance optimization +tito test --module compression # Model optimization +tito test --module training # End-to-end training +``` -- [MLPerf Inference Rules](https://github.com/mlcommons/inference_policies) -- [Statistical Methods for ML Evaluation](https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/) -- [A/B Testing for ML Systems](https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15) +### Development Workflow +1. **Open the development file**: `modules/source/13_benchmarking/benchmarking_dev.py` +2. **Implement benchmarking framework**: Build MLPerf-inspired evaluation system +3. **Add statistical validation**: Ensure reproducible and significant results +4. **Create performance reporters**: Generate professional documentation +5. **Test evaluation scenarios**: Apply to real models and optimization claims +6. **Export and verify**: `tito export --module benchmarking && tito test --module benchmarking` ---- +## ๐Ÿงช Testing Your Implementation -**๐ŸŽ‰ Ready to become a systematic ML evaluator? Let's build professional benchmarking skills!** \ No newline at end of file +### Comprehensive Test Suite +Run the full test suite to verify benchmarking system functionality: + +```bash +# TinyTorch CLI (recommended) +tito test --module benchmarking + +# Direct pytest execution +python -m pytest tests/ -k benchmarking -v +``` + +### Test Coverage Areas +- โœ… **Benchmarking Framework**: Verify MLPerf-inspired evaluation system works correctly +- โœ… **Statistical Validation**: Test confidence intervals, significance testing, and reproducibility +- โœ… **Performance Reporting**: Ensure professional report generation and data visualization +- โœ… **Scenario Testing**: Validate mobile, server, and offline evaluation scenarios +- โœ… **Integration Testing**: Test with real TinyTorch models and optimizations + +### Inline Testing & Evaluation Validation +The module includes comprehensive benchmarking validation and methodology verification: +```python +# Example inline test output +๐Ÿ”ฌ Unit Test: MLPerf-inspired benchmark framework... +โœ… Single-stream scenario working correctly +โœ… Server scenario measures throughput accurately +โœ… Offline scenario handles batch processing +๐Ÿ“ˆ Progress: Benchmarking Framework โœ“ + +# Statistical validation testing +๐Ÿ”ฌ Unit Test: Statistical significance testing... +โœ… Confidence intervals computed correctly +โœ… Multiple comparison correction applied +โœ… Minimum sample size requirements enforced +๐Ÿ“ˆ Progress: Statistical Validation โœ“ + +# Report generation testing +๐Ÿ”ฌ Unit Test: Performance report generation... +โœ… HTML reports generated with proper formatting +โœ… Summary tables include all required metrics +โœ… Visualization charts display correctly +๐Ÿ“ˆ Progress: Professional Reporting โœ“ +``` + +### Manual Testing Examples +```python +from benchmarking_dev import TinyTorchPerf, StatisticalValidator +from networks_dev import Sequential +from layers_dev import Dense +from activations_dev import ReLU + +# Create test models +baseline_model = Sequential([Dense(784, 128), ReLU(), Dense(128, 10)]) +optimized_model = compress_model(baseline_model, compression_ratio=0.5) + +# Set up benchmarking +benchmark = TinyTorchPerf() +benchmark.set_dataset('synthetic', size=1000, input_shape=(784,), num_classes=10) + +# Run evaluation +baseline_results = benchmark.evaluate_model(baseline_model) +optimized_results = benchmark.evaluate_model(optimized_model) + +print(f"Baseline latency: {baseline_results['latency']:.2f}ms") +print(f"Optimized latency: {optimized_results['latency']:.2f}ms") +print(f"Speedup: {baseline_results['latency']/optimized_results['latency']:.2f}x") + +# Statistical validation +validator = StatisticalValidator() +comparison = validator.compare_models(baseline_model, optimized_model, test_data) +print(f"Statistically significant: {comparison['significant']}") +``` + +## ๐ŸŽฏ Key Concepts + +### Real-World Applications +- **MLPerf Benchmarks**: Industry-standard evaluation methodology for ML systems and hardware +- **Production A/B Testing**: Statistical validation of model improvements in live systems +- **Research Paper Evaluation**: Rigorous experimental methodology for academic publication +- **Hardware Evaluation**: Systematic comparison of ML accelerators and deployment platforms + +### Evaluation Methodology +- **Systematic Experimentation**: Controlled variables, multiple runs, and statistical validation +- **Scenario-Based Testing**: Mobile, server, and edge deployment evaluation patterns +- **Performance Metrics**: Latency, throughput, accuracy, memory usage, and energy consumption +- **Statistical Rigor**: Confidence intervals, significance testing, and reproducibility requirements + +### Professional Reporting +- **Industry Standards**: MLPerf-style reporting with comprehensive metrics and statistical validation +- **Visual Communication**: Charts, tables, and graphs that clearly communicate performance results +- **Executive Summaries**: High-level findings suitable for technical and business stakeholders +- **Reproducibility**: Complete methodology documentation for result verification + +### Benchmarking Best Practices +- **Baseline Establishment**: Proper reference points for meaningful comparisons +- **Environment Control**: Consistent hardware, software, and data conditions +- **Statistical Power**: Sufficient sample sizes for reliable conclusions +- **Bias Avoidance**: Careful experimental design to prevent misleading results + +## ๐ŸŽ‰ Ready to Build? + +You're about to master the evaluation methodology that separates rigorous engineering from wishful thinking! This module teaches you to validate claims, measure improvements systematically, and communicate results professionally. + +Every major breakthrough in MLโ€”from ImageNet winners to production systemsโ€”depends on systematic evaluation like what you're building. You'll learn to think like a performance scientist, ensuring your optimizations actually work and proving it with statistical rigor. Take your time, be thorough, and enjoy building the foundation of evidence-based ML engineering! + +```{grid} 3 +:gutter: 3 +:margin: 2 + +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/13_benchmarking/benchmarking_dev.py +:class-title: text-center +:class-body: text-center + +Interactive development environment + +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/13_benchmarking/benchmarking_dev.ipynb +:class-title: text-center +:class-body: text-center + +Google Colab notebook + +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/13_benchmarking/benchmarking_dev.py +:class-title: text-center +:class-body: text-center + +Browse the code on GitHub +``` \ No newline at end of file diff --git a/modules/source/14_mlops/README.md b/modules/source/14_mlops/README.md index f358c1a1..06b1b3ba 100644 --- a/modules/source/14_mlops/README.md +++ b/modules/source/14_mlops/README.md @@ -4,324 +4,423 @@ - **Difficulty**: โญโญโญโญโญ Expert - **Time Estimate**: 10-12 hours - **Prerequisites**: All previous modules (01-13) - Complete TinyTorch ecosystem -- **Next Steps**: **Final capstone module** - Deploy your complete ML system! +- **Next Steps**: **๐ŸŽ“ Course completion** - Deploy your complete ML system! -**Build production-ready ML systems with deployment, monitoring, and continuous learning** +Build production-ready ML systems with deployment, monitoring, and continuous learning. This capstone module integrates everything you've built into production-grade systems that can handle real-world challenges and scale to enterprise requirements. ## ๐ŸŽฏ Learning Objectives -After completing this module, you will: -- Build complete MLOps pipelines from model development to production -- Implement model versioning and registry systems for lifecycle management -- Create production-ready model serving and inference endpoints -- Design monitoring systems for model performance and data drift detection -- Apply A/B testing methodology for safe model deployment -- Implement continuous learning systems for model improvement -- Integrate all TinyTorch components into production-ready systems +By the end of this module, you will be able to: + +- **Design complete MLOps architectures**: Orchestrate model development, deployment, and operations into production-ready systems +- **Implement model lifecycle management**: Build versioning, registry, and deployment automation for reliable model operations +- **Create production serving systems**: Deploy scalable, reliable model inference endpoints with monitoring and observability +- **Build continuous learning pipelines**: Implement automated retraining, A/B testing, and model improvement workflows +- **Apply enterprise MLOps practices**: Use industry-standard patterns for model governance, security, and compliance ## ๐Ÿง  Build โ†’ Use โ†’ Deploy -This module follows the TinyTorch **"Build โ†’ Use โ†’ Deploy"** pedagogical framework: +This module follows TinyTorch's **Build โ†’ Use โ†’ Deploy** framework: -1. **Build**: Complete MLOps infrastructure and production systems -2. **Use**: Deploy and operate ML systems in production environments -3. **Deploy**: Create end-to-end ML pipelines ready for real-world deployment - -## ๐Ÿ”— Connection to Previous Modules - -### The Complete TinyTorch Ecosystem -MLOps is the **capstone module** that brings together everything you've built: - -- **00_setup**: System configuration and development environment -- **01_tensor**: Data structures and operations -- **02_activations**: Nonlinear functions for neural networks -- **03_layers**: Building blocks of neural networks -- **04_networks**: Complete neural network architectures -- **05_cnn**: Convolutional networks for image processing -- **06_dataloader**: Data loading and preprocessing pipelines -- **07_autograd**: Automatic differentiation for training -- **08_optimizers**: Training algorithms and optimization -- **09_training**: Complete training pipelines and workflows -- **10_compression**: Model optimization for deployment -- **11_kernels**: Hardware-optimized operations -- **12_benchmarking**: Performance measurement and evaluation - -### The Production Gap -Students understand **how to build** and **how to optimize** ML systems but not **how to deploy** them: -- โœ… **Development**: Can build complete ML systems from scratch -- โœ… **Optimization**: Can compress, accelerate, and benchmark models -- โŒ **Production**: Don't know how to deploy, monitor, and maintain systems -- โŒ **Operations**: Can't handle model versioning, A/B testing, or continuous learning +1. **Build**: Implement complete MLOps infrastructure including model registry, serving, monitoring, and continuous learning systems +2. **Use**: Deploy and operate ML systems in production environments with real-world constraints and requirements +3. **Deploy**: Create end-to-end ML pipelines that demonstrate mastery of the entire TinyTorch ecosystem ## ๐Ÿ“š What You'll Build -### **Model Management System** +### Complete Model Lifecycle Management ```python -# Model versioning and registry +# Enterprise-grade model registry and versioning +from tinytorch.core.mlops import ModelRegistry, ModelMetadata + +# Model registry with comprehensive metadata registry = ModelRegistry("production") -model_v1 = registry.register_model(trained_model, version="1.0.0") -model_v2 = registry.register_model(compressed_model, version="2.0.0") +metadata = ModelMetadata( + name="image_classifier_v2", + version="2.1.0", + training_data="cifar10_v3", + compression_applied=True, + performance_metrics={'accuracy': 0.94, 'latency_ms': 23}, + compliance_approved=True +) -# Version comparison -comparison = registry.compare_models("1.0.0", "2.0.0") +# Register model with full lifecycle tracking +model_id = registry.register_model( + model=optimized_model, + metadata=metadata, + artifacts=['weights.pt', 'config.json', 'benchmark_report.html'] +) + +# Model comparison and governance +comparison = registry.compare_models("2.0.0", "2.1.0") +deployment_approval = registry.approve_for_production(model_id) ``` -### **Production Serving System** +### Production Serving Infrastructure ```python -# Model serving endpoint -server = ModelServer(model_v2, port=8080) -server.start() +# Scalable model serving with monitoring +from tinytorch.core.mlops import ModelServer, LoadBalancer, HealthChecker -# Inference endpoint -endpoint = InferenceEndpoint(server) -prediction = endpoint.predict(input_data) +# Configure production server +server = ModelServer( + model_id=model_id, + max_concurrent_requests=100, + timeout_ms=500, + auto_scaling=True, + health_check_interval=30 +) + +# Load balancing across multiple instances +load_balancer = LoadBalancer( + servers=[server1, server2, server3], + strategy='round_robin', + health_aware=True +) + +# Inference endpoint with comprehensive logging +@server.endpoint('/predict') +def predict(request): + start_time = time.time() + + # Input validation and preprocessing + validated_input = validate_input(request.data) + preprocessed_input = preprocess(validated_input) + + # Model inference + prediction = model.predict(preprocessed_input) + + # Logging and monitoring + latency = (time.time() - start_time) * 1000 + logger.log_prediction(request.id, prediction, latency) + monitor.track_inference(latency, prediction.confidence) + + return jsonify({'prediction': prediction.tolist(), 'confidence': prediction.confidence}) ``` -### **Monitoring & Observability** +### Advanced Monitoring and Observability ```python -# Model performance monitoring -monitor = ModelMonitor(model_v2) -monitor.track_latency(prediction_time) -monitor.track_accuracy(predictions, true_labels) +# Comprehensive production monitoring +from tinytorch.core.mlops import ModelMonitor, DriftDetector, AlertManager -# Data drift detection -drift_detector = DriftDetector(reference_data) -drift_detected = drift_detector.detect_drift(new_data) +# Multi-dimensional monitoring system +monitor = ModelMonitor(model_id) +monitor.track_performance_metrics(['latency', 'throughput', 'accuracy']) +monitor.track_business_metrics(['conversion_rate', 'user_satisfaction']) +monitor.track_infrastructure_metrics(['cpu_usage', 'memory_usage', 'error_rate']) + +# Advanced drift detection +drift_detector = DriftDetector( + reference_dataset=training_data, + detection_methods=['statistical', 'adversarial', 'embedding_drift'], + alert_threshold=0.05 +) + +# Real-time alerting system +alert_manager = AlertManager() +alert_manager.configure_alerts({ + 'latency_p99_ms': {'threshold': 100, 'severity': 'critical'}, + 'accuracy_drop': {'threshold': 0.02, 'severity': 'high'}, + 'drift_score': {'threshold': 0.05, 'severity': 'medium'}, + 'error_rate': {'threshold': 0.01, 'severity': 'high'} +}) ``` -### **A/B Testing Framework** +### A/B Testing and Experimentation ```python -# Safe model deployment -ab_test = ABTestManager() -ab_test.add_variant("control", model_v1, traffic_split=0.8) -ab_test.add_variant("treatment", model_v2, traffic_split=0.2) +# Production-grade experimentation framework +from tinytorch.core.mlops import ExperimentManager, TrafficSplitter -# Experiment tracking -results = ab_test.run_experiment(test_data) +# Configure A/B test +experiment = ExperimentManager("image_classifier_optimization") +experiment.add_variant("control", model_v2_0, traffic_percentage=70) +experiment.add_variant("treatment", model_v2_1, traffic_percentage=30) + +# Statistical experiment design +experiment.configure_statistical_parameters( + significance_level=0.05, + minimum_detectable_effect=0.01, + power=0.8, + expected_runtime_days=14 +) + +# Traffic splitting with session consistency +traffic_splitter = TrafficSplitter(experiment) + +@server.endpoint('/predict') +def predict_with_experiment(request): + # Determine experiment variant + variant = traffic_splitter.assign_variant(request.user_id) + model = experiment.get_model(variant) + + # Make prediction and log experiment data + prediction = model.predict(request.data) + experiment.log_outcome(request.user_id, variant, prediction, request.ground_truth) + + return prediction + +# Automated experiment analysis +experiment_results = experiment.analyze_results() +if experiment_results.significant_improvement: + experiment.promote_winner() ``` -### **Continuous Learning System** +### Continuous Learning and Automation ```python -# Automated retraining -learner = ContinuousLearner(model_v2) -learner.add_training_data(new_data) -improved_model = learner.retrain_if_needed() +# Automated model improvement pipeline +from tinytorch.core.mlops import ContinuousLearner, AutoMLPipeline -# Automated deployment -pipeline = MLOpsPipeline() -pipeline.train_model(new_data) -pipeline.validate_model(validation_data) -pipeline.deploy_model(improved_model) +# Continuous learning system +learner = ContinuousLearner( + base_model=current_production_model, + retraining_schedule='weekly', + data_freshness_threshold=7, # days + performance_threshold_drop=0.02 +) + +# Automated pipeline orchestration +pipeline = AutoMLPipeline() +pipeline.configure_stages([ + 'data_validation', + 'feature_engineering', + 'model_training', + 'model_evaluation', + 'compression_optimization', + 'performance_validation', + 'a_b_testing', + 'production_deployment' +]) + +# Trigger automated improvement +@learner.schedule('weekly') +def automated_model_improvement(): + # Collect new training data + new_data = data_collector.get_recent_data(days=7) + + # Validate data quality + if data_validator.validate(new_data): + # Retrain model with new data + improved_model = pipeline.train_improved_model( + base_model=current_production_model, + additional_data=new_data + ) + + # Automated evaluation + if pipeline.meets_production_criteria(improved_model): + # Deploy to A/B test + experiment_manager.deploy_candidate(improved_model) ``` -## ๐ŸŽ“ Educational Structure - -### **Step 1: Model Management & Versioning** -- **Concept**: Model lifecycle management and version control -- **Implementation**: ModelRegistry, ModelVersioning, ModelSerializer -- **Learning**: Track model evolution and manage production deployments - -### **Step 2: Production Serving & Deployment** -- **Concept**: Scalable model serving and inference endpoints -- **Implementation**: ModelServer, InferenceEndpoint, BatchInference -- **Learning**: Deploy models for real-time and batch inference - -### **Step 3: Monitoring & Observability** -- **Concept**: Production model monitoring and performance tracking -- **Implementation**: ModelMonitor, PerformanceTracker, DriftDetector -- **Learning**: Detect issues and maintain model quality in production - -### **Step 4: A/B Testing & Experimentation** -- **Concept**: Safe deployment through controlled experiments -- **Implementation**: ABTestManager, ExperimentTracker, ModelComparator -- **Learning**: Validate model improvements with statistical rigor - -### **Step 5: Continuous Learning & Automation** -- **Concept**: Automated model improvement and retraining -- **Implementation**: ContinuousLearner, AutoRetrainer, DataPipeline -- **Learning**: Build self-improving ML systems - -### **Step 6: Complete MLOps Pipeline** -- **Concept**: End-to-end production ML system orchestration -- **Implementation**: MLOpsPipeline, DeploymentManager, ProductionValidator -- **Learning**: Integrate all components into production-ready systems - -## ๐ŸŒ Real-World Applications - -### **Production ML Systems** -- **Netflix**: Recommendation system deployment and A/B testing -- **Uber**: Real-time demand prediction and dynamic pricing -- **Spotify**: Music recommendation and playlist generation -- **Google**: Search ranking and ad serving systems - -### **Model Lifecycle Management** -- **Airbnb**: Price prediction model versioning and deployment -- **Facebook**: News feed algorithm updates and rollbacks -- **Amazon**: Product recommendation system evolution -- **Tesla**: Autonomous driving model deployment and monitoring - -### **Monitoring & Observability** -- **Stripe**: Fraud detection system monitoring -- **Zillow**: Home price prediction accuracy tracking -- **LinkedIn**: Job recommendation performance monitoring -- **Twitter**: Content moderation model drift detection - -### **Continuous Learning** -- **YouTube**: Video recommendation system adaptation -- **Instagram**: Content filtering continuous improvement -- **Snapchat**: Face filter quality enhancement -- **TikTok**: Content discovery algorithm evolution - -## ๐Ÿ”ง Technical Architecture - -### **Production Requirements** +### Enterprise Integration and Governance ```python -# Performance requirements -- Latency: < 100ms inference time -- Throughput: > 1000 requests/second -- Availability: 99.9% uptime -- Scalability: Handle traffic spikes +# Production ML system with enterprise features +from tinytorch.core.mlops import MLOpsPlatform, GovernanceEngine -# Reliability requirements -- Model versioning: Track all model changes -- Rollback capability: Revert to previous versions -- Monitoring: Real-time performance tracking -- Alerting: Automated issue detection +# Complete MLOps platform +platform = MLOpsPlatform() +platform.configure_enterprise_features({ + 'model_governance': True, + 'audit_logging': True, + 'compliance_tracking': True, + 'role_based_access': True, + 'encryption_at_rest': True, + 'encryption_in_transit': True +}) + +# Governance and compliance +governance = GovernanceEngine() +governance.configure_policies({ + 'model_approval_required': True, + 'bias_testing_required': True, + 'performance_monitoring_required': True, + 'data_lineage_tracking': True, + 'model_explainability_required': True +}) + +# Complete deployment with governance +deployment = platform.deploy_model( + model=approved_model, + environment='production', + governance_checks=governance.get_required_checks(), + monitoring_config=monitor.get_config(), + serving_config=server.get_config() +) ``` -### **Integration with TinyTorch Components** +## ๐Ÿš€ Getting Started + +### Prerequisites +Ensure you have completed the entire TinyTorch journey: + +```bash +# Activate TinyTorch environment +source bin/activate-tinytorch.sh + +# Verify complete ecosystem (this is the final capstone!) +tito test --module tensor # Foundation +tito test --module activations # Neural network components +tito test --module layers # Building blocks +tito test --module networks # Architectures +tito test --module cnn # Computer vision +tito test --module dataloader # Data engineering +tito test --module autograd # Automatic differentiation +tito test --module optimizers # Learning algorithms +tito test --module training # End-to-end training +tito test --module compression # Model optimization +tito test --module kernels # Performance optimization +tito test --module benchmarking # Evaluation methodology +``` + +### Development Workflow +1. **Open the development file**: `modules/source/14_mlops/mlops_dev.py` +2. **Implement model lifecycle management**: Build registry, versioning, and metadata systems +3. **Create production serving**: Develop scalable inference endpoints with monitoring +4. **Add monitoring and observability**: Build comprehensive tracking and alerting systems +5. **Build experimentation framework**: Implement A/B testing and statistical validation +6. **Create continuous learning**: Develop automated improvement and deployment pipelines +7. **Complete capstone project**: Integrate entire TinyTorch ecosystem into production system + +## ๐Ÿงช Testing Your Implementation + +### Comprehensive Test Suite +Run the full test suite to verify complete MLOps system functionality: + +```bash +# TinyTorch CLI (recommended) +tito test --module mlops + +# Direct pytest execution +python -m pytest tests/ -k mlops -v +``` + +### Test Coverage Areas +- โœ… **Model Lifecycle Management**: Verify registry, versioning, and metadata tracking +- โœ… **Production Serving**: Test scalable inference endpoints and load balancing +- โœ… **Monitoring Systems**: Ensure comprehensive tracking and alerting functionality +- โœ… **A/B Testing Framework**: Validate experimental design and statistical analysis +- โœ… **Continuous Learning**: Test automated retraining and deployment workflows +- โœ… **Enterprise Integration**: Verify governance, security, and compliance features + +### Inline Testing & Production Validation +The module includes comprehensive MLOps validation and enterprise readiness verification: ```python -# Complete system integration +# Example inline test output +๐Ÿ”ฌ Unit Test: Model lifecycle management... +โœ… Model registry stores and retrieves models correctly +โœ… Versioning system tracks model evolution +โœ… Metadata management supports governance requirements +๐Ÿ“ˆ Progress: Model Lifecycle โœ“ + +# Production serving testing +๐Ÿ”ฌ Unit Test: Production inference endpoints... +โœ… Server handles concurrent requests correctly +โœ… Load balancing distributes traffic evenly +โœ… Health checks detect and route around failures +๐Ÿ“ˆ Progress: Production Serving โœ“ + +# Monitoring and observability +๐Ÿ”ฌ Unit Test: Production monitoring systems... +โœ… Performance metrics tracked accurately +โœ… Drift detection identifies data changes +โœ… Alert system triggers on threshold violations +๐Ÿ“ˆ Progress: Monitoring & Observability โœ“ + +# End-to-end integration +๐Ÿ”ฌ Unit Test: Complete MLOps pipeline... +โœ… All TinyTorch components integrate successfully +โœ… Production deployment meets enterprise requirements +โœ… Continuous learning pipeline operates automatically +๐Ÿ“ˆ Progress: Complete MLOps System โœ“ +``` + +### Capstone Project Validation +```python +# Complete system integration test +from tinytorch.core.mlops import MLOpsPlatform from tinytorch.core.training import Trainer from tinytorch.core.compression import quantize_model from tinytorch.core.kernels import optimize_inference -from tinytorch.core.benchmarking import benchmark_model -from tinytorch.core.mlops import MLOpsPipeline -# End-to-end pipeline -pipeline = MLOpsPipeline() -trained_model = pipeline.train_with_trainer(Trainer, data) -compressed_model = pipeline.compress_model(quantize_model, trained_model) -optimized_model = pipeline.optimize_inference(optimize_inference, compressed_model) -benchmark_results = pipeline.benchmark_model(benchmark_model, optimized_model) -deployed_model = pipeline.deploy_model(optimized_model) +# End-to-end pipeline validation +platform = MLOpsPlatform() + +# Train model using TinyTorch training system +trainer = Trainer(model, optimizer, loss_fn) +trained_model = trainer.fit(train_loader, val_loader, epochs=50) + +# Optimize using compression and kernels +compressed_model = quantize_model(trained_model) +optimized_model = optimize_inference(compressed_model) + +# Deploy to production with full MLOps +deployment = platform.deploy_complete_system( + model=optimized_model, + monitoring=True, + a_b_testing=True, + continuous_learning=True +) + +print(f"โœ… Complete TinyTorch system deployed successfully!") +print(f"๐Ÿ“Š Model accuracy: {deployment.metrics['accuracy']:.4f}") +print(f"โšก Inference latency: {deployment.metrics['latency_ms']:.2f}ms") +print(f"๐Ÿš€ Production endpoint: {deployment.endpoint_url}") ``` -## ๐ŸŽฏ Key Skills Developed +## ๐ŸŽฏ Key Concepts -### **Systems Engineering** -- **Architecture design**: Scalable, reliable ML system design -- **Performance optimization**: Low-latency, high-throughput systems -- **Reliability engineering**: Fault-tolerant and self-healing systems -- **Monitoring & observability**: Comprehensive system health tracking +### Real-World Applications +- **Netflix**: Recommendation system deployment with A/B testing and continuous learning +- **Uber**: Real-time demand prediction with monitoring and automated retraining +- **Spotify**: Music recommendation MLOps with experimentation and personalization +- **Tesla**: Autonomous driving model deployment with safety monitoring and over-the-air updates -### **ML Engineering** -- **Model lifecycle management**: Version control and deployment strategies -- **Production deployment**: Safe, scalable model serving -- **Continuous learning**: Automated model improvement workflows -- **Experiment design**: A/B testing and statistical validation +### MLOps Architecture Patterns +- **Model Registry**: Centralized model versioning, metadata, and artifact management +- **Serving Infrastructure**: Scalable, reliable model inference with load balancing and health monitoring +- **Observability**: Comprehensive monitoring of model performance, data quality, and system health +- **Experimentation**: Statistical A/B testing for safe model deployment and improvement validation -### **DevOps & Platform Engineering** -- **CI/CD pipelines**: Automated testing and deployment -- **Infrastructure as code**: Reproducible deployment environments -- **Container orchestration**: Scalable model serving infrastructure -- **Monitoring & alerting**: Proactive issue detection and resolution +### Production ML Engineering +- **Deployment Automation**: CI/CD pipelines for model deployment with safety checks and rollback capabilities +- **Performance Optimization**: Integration of compression, quantization, and hardware optimization +- **Reliability Engineering**: Fault tolerance, disaster recovery, and high availability design +- **Security and Governance**: Model security, audit trails, and compliance with regulations -## ๐Ÿ† Capstone Project: Complete ML System +### Continuous Learning Systems +- **Automated Retraining**: Data-driven model improvement with performance monitoring +- **Feedback Loops**: Online learning and adaptation based on production performance +- **Quality Assurance**: Automated testing and validation before production deployment +- **Business Impact**: Connecting ML improvements to business metrics and outcomes -### **Project Overview** -Build a complete, production-ready ML system that demonstrates mastery of the entire TinyTorch ecosystem. +## ๐ŸŽ‰ Ready to Build? -### **Project Components** -1. **Data Pipeline**: Automated data ingestion and preprocessing -2. **Model Training**: Automated training with hyperparameter optimization -3. **Model Optimization**: Compression and kernel optimization -4. **Benchmarking**: Performance evaluation and comparison -5. **Deployment**: Production serving with monitoring -6. **Continuous Learning**: Automated retraining and improvement +๐ŸŽ“ **Congratulations!** You've reached the capstone module of TinyTorch! This is where everything comes togetherโ€”all the tensors, layers, networks, data loading, training, optimization, and evaluation you've built will be integrated into a production-ready ML system. -### **Deliverables** -- **Trained Model**: High-quality model trained on real data -- **Compressed Model**: Optimized for production deployment -- **Serving Endpoint**: Production-ready inference API -- **Monitoring Dashboard**: Real-time performance tracking -- **A/B Testing Framework**: Safe deployment validation -- **Continuous Learning Pipeline**: Automated improvement system +You're about to build the same MLOps infrastructure that powers the AI systems you use every day. From recommendation engines to autonomous vehicles, they all depend on the deployment patterns, monitoring systems, and continuous learning pipelines you're implementing. -## ๐Ÿ”ฎ Industry Connections +Take your time, think about the big picture, and enjoy creating a complete ML system that's ready for the real world. This is your moment to demonstrate mastery of the entire ML engineering stack! ๐Ÿš€ -### **MLOps Platforms** -- **MLflow**: Model lifecycle management and experiment tracking -- **Kubeflow**: Kubernetes-based ML workflows and pipelines -- **TensorFlow Extended (TFX)**: End-to-end ML platform -- **Amazon SageMaker**: AWS managed ML platform -- **Google AI Platform**: Google Cloud ML services -- **Azure ML**: Microsoft's comprehensive ML platform +```{grid} 3 +:gutter: 3 +:margin: 2 -### **Production ML Systems** -- **TensorFlow Serving**: High-performance model serving -- **PyTorch Serve**: PyTorch model deployment -- **ONNX Runtime**: Cross-platform inference optimization -- **Apache Kafka**: Real-time data streaming -- **Prometheus**: Monitoring and alerting -- **Grafana**: Visualization and dashboards +{grid-item-card} ๐Ÿš€ Launch Builder +:link: https://mybinder.org/v2/gh/VJProductions/TinyTorch/main?filepath=modules/source/14_mlops/mlops_dev.py +:class-title: text-center +:class-body: text-center -### **Career Preparation** -- **ML Engineer**: Production ML system development -- **MLOps Engineer**: ML infrastructure and operations -- **Data Engineer**: ML data pipeline development -- **Platform Engineer**: ML platform and tooling -- **Site Reliability Engineer**: Production system reliability -- **ML Researcher**: Advanced ML system research +Interactive development environment -## ๐Ÿš€ What's Next +{grid-item-card} ๐Ÿ““ Open in Colab +:link: https://colab.research.google.com/github/VJProductions/TinyTorch/blob/main/modules/source/14_mlops/mlops_dev.ipynb +:class-title: text-center +:class-body: text-center -### **Beyond TinyTorch** -Your MLOps skills prepare you for: -- **Production ML roles**: Industry-ready deployment expertise -- **Advanced ML systems**: Distributed training, federated learning -- **ML platform development**: Building ML infrastructure and tools -- **Research applications**: Reproducible, scalable research systems +Google Colab notebook -### **Continuous Learning** -- **Advanced MLOps**: Multi-model systems, federated learning -- **ML Security**: Model privacy, security, and governance -- **AutoML**: Automated machine learning systems -- **Edge ML**: Deployment on edge devices and IoT systems +{grid-item-card} ๐Ÿ‘€ View Source +:link: https://github.com/VJProductions/TinyTorch/blob/main/modules/source/14_mlops/mlops_dev.py +:class-title: text-center +:class-body: text-center -## ๐Ÿ“ File Structure -``` -13_mlops/ -โ”œโ”€โ”€ mlops_dev.py # Main development notebook -โ”œโ”€โ”€ module.yaml # Module configuration -โ”œโ”€โ”€ README.md # This file -โ”œโ”€โ”€ deployments/ # Deployment configurations -โ”‚ โ”œโ”€โ”€ docker/ # Container configurations -โ”‚ โ”œโ”€โ”€ kubernetes/ # K8s deployment configs -โ”‚ โ””โ”€โ”€ monitoring/ # Monitoring configurations -โ””โ”€โ”€ tests/ # Additional test files - โ””โ”€โ”€ test_mlops.py # External tests -``` - -## ๐ŸŽฏ Getting Started - -1. **Review Prerequisites**: Ensure all modules 01-13 are complete -2. **Open Development File**: `mlops_dev.py` -3. **Follow Educational Flow**: Work through Steps 1-6 sequentially -4. **Build Capstone Project**: Complete end-to-end ML system -5. **Test Production System**: Validate deployment and monitoring -6. **Export to Package**: Use `tito export 13_mlops` when complete - -## ๐ŸŽ‰ Final Achievement - -Students completing this module will: -- **Master production ML systems**: End-to-end deployment expertise -- **Understand ML operations**: Complete MLOps lifecycle management -- **Build scalable systems**: Production-ready ML infrastructure -- **Apply best practices**: Industry-standard deployment and monitoring -- **Demonstrate expertise**: Complete TinyTorch ecosystem mastery -- **Prepare for careers**: Industry-ready ML engineering skills - -**Congratulations!** You've built a complete ML framework from scratch and learned to deploy it in production. You're now ready to tackle real-world ML systems with confidence and expertise! - -This module represents the culmination of your TinyTorch journey - from basic tensors to production-ready ML systems. You've gained the skills to build, optimize, and deploy ML systems that can handle real-world challenges and scale to production requirements. \ No newline at end of file +Browse the code on GitHub +``` \ No newline at end of file