mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-06 06:48:26 -05:00
Implement README-to-chapter conversion for cleaner website workflow
This commit is contained in:
Vijay Janapa Reddi
@@ -17,42 +17,42 @@ parts:
|
||||
|
||||
- caption: Foundation
|
||||
chapters:
|
||||
- file: book/chapters/01-setup.ipynb
|
||||
- file: chapters/01-setup
|
||||
title: "1. Setup"
|
||||
- file: book/chapters/02-tensor.ipynb
|
||||
- file: chapters/02-tensor
|
||||
title: "2. Tensors"
|
||||
- file: book/chapters/03-activations.ipynb
|
||||
- file: chapters/03-activations
|
||||
title: "3. Activations"
|
||||
|
||||
- caption: Building Blocks
|
||||
chapters:
|
||||
- file: book/chapters/04-layers.ipynb
|
||||
- file: chapters/04-layers
|
||||
title: "4. Layers"
|
||||
- file: book/chapters/05-networks.ipynb
|
||||
- file: chapters/05-networks
|
||||
title: "5. Networks"
|
||||
- file: book/chapters/06-cnn.ipynb
|
||||
- file: chapters/06-cnn
|
||||
title: "6. CNNs"
|
||||
|
||||
- caption: Training Systems
|
||||
chapters:
|
||||
- file: book/chapters/07-dataloader.ipynb
|
||||
- file: chapters/07-dataloader
|
||||
title: "7. DataLoader"
|
||||
- file: book/chapters/08-autograd.ipynb
|
||||
- file: chapters/08-autograd
|
||||
title: "8. Autograd"
|
||||
- file: book/chapters/09-optimizers.ipynb
|
||||
- file: chapters/09-optimizers
|
||||
title: "9. Optimizers"
|
||||
- file: book/chapters/10-training.ipynb
|
||||
- file: chapters/10-training
|
||||
title: "10. Training"
|
||||
|
||||
- caption: Production & Performance
|
||||
chapters:
|
||||
- file: book/chapters/11-compression.ipynb
|
||||
- file: chapters/11-compression
|
||||
title: "11. Compression"
|
||||
- file: book/chapters/12-kernels.ipynb
|
||||
- file: chapters/12-kernels
|
||||
title: "12. Kernels"
|
||||
- file: book/chapters/13-benchmarking.ipynb
|
||||
- file: chapters/13-benchmarking
|
||||
title: "13. Benchmarking"
|
||||
- file: book/chapters/14-mlops.ipynb
|
||||
- file: chapters/14-mlops
|
||||
title: "14. MLOps"
|
||||
|
||||
- caption: Appendices
|
||||
|
||||
@@ -1,78 +1,189 @@
|
||||
# Setup - TinyTorch System Configuration
|
||||
---
|
||||
title: "Setup & Environment"
|
||||
description: "Development environment setup and basic TinyTorch functionality"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: ['Understand the nbdev notebook-to-Python workflow', 'Write your first TinyTorch code with `#| export` directives', 'Implement system information collection and developer profiles', 'Run tests and use the CLI tools', 'Get comfortable with the development rhythm']
|
||||
---
|
||||
|
||||
Welcome to TinyTorch! This setup module configures your personal TinyTorch installation and teaches you the NBGrader workflow.
|
||||
# Setup Module
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 1: 01 Setup](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Configure your personal TinyTorch installation with custom information
|
||||
- Learn to query system information using Python modules
|
||||
- Master the NBGrader workflow: implement → test → export
|
||||
- Create functions that become part of your tinytorch package
|
||||
- Understand solution blocks, hidden tests, and automated grading
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐ | <strong>Time:</strong> 1-2 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐ Beginner
|
||||
- **Time Estimate**: 1-2 hours
|
||||
- **Prerequisites**: Basic Python knowledge
|
||||
- **Next Steps**: Tensor module
|
||||
|
||||
Welcome to TinyTorch! This is your first module in the Machine Learning Systems course.
|
||||
|
||||
## Overview
|
||||
|
||||
The setup module teaches you the complete TinyTorch development workflow while introducing fundamental programming concepts. You'll learn to write code with NBDev directives, implement classes and functions, and understand the module-to-package export system.
|
||||
|
||||
## Learning Goals
|
||||
|
||||
- Understand the nbdev notebook-to-Python workflow
|
||||
- Write your first TinyTorch code with `#| export` directives
|
||||
- Implement system information collection and developer profiles
|
||||
- Run tests and use the CLI tools
|
||||
- Get comfortable with the development rhythm
|
||||
|
||||
## Files
|
||||
|
||||
- `setup_dev.py` - Main development file (Jupytext format with full educational content)
|
||||
- `setup_dev.ipynb` - Jupyter notebook version (auto-generated and executed)
|
||||
- `tinytorch_flame.txt` - ASCII art file containing the TinyTorch flame design
|
||||
- `tests/test_setup.py` - Comprehensive pytest test suite
|
||||
- `README.md` - This file
|
||||
|
||||
## What You'll Implement
|
||||
|
||||
### 1. Basic Functions
|
||||
- `hello_tinytorch()` - Display ASCII art and welcome message
|
||||
- `add_numbers()` - Basic arithmetic (foundation of ML operations)
|
||||
|
||||
### 2. System Information Class
|
||||
- `SystemInfo` - Collect and display Python version, platform, and machine info
|
||||
- Compatibility checking for minimum requirements
|
||||
|
||||
### 3. Developer Profile Class
|
||||
- `DeveloperProfile` - Personalized developer information and signatures
|
||||
- ASCII art customization and file loading
|
||||
- Professional code attribution system
|
||||
|
||||
## Usage
|
||||
|
||||
### Python Script
|
||||
```python
|
||||
from setup_dev import hello_tinytorch, add_numbers, SystemInfo, DeveloperProfile
|
||||
|
||||
# Display welcome message
|
||||
hello_tinytorch()
|
||||
|
||||
# Basic arithmetic
|
||||
result = add_numbers(2, 3)
|
||||
|
||||
# System information
|
||||
info = SystemInfo()
|
||||
print(f"System: {info}")
|
||||
print(f"Compatible: {info.is_compatible()}")
|
||||
|
||||
# Developer profile
|
||||
profile = DeveloperProfile()
|
||||
print(profile.get_full_profile())
|
||||
```
|
||||
|
||||
### Jupyter Notebook
|
||||
Open `setup_dev.ipynb` and work through the educational content step by step.
|
||||
|
||||
## The Big Picture: Why Configuration Matters in ML Systems
|
||||
Configuration is the foundation of any production ML system. In this module, you'll learn:
|
||||
## Testing
|
||||
|
||||
### 1. **System Awareness**
|
||||
Real ML systems need to understand their environment:
|
||||
- **Hardware constraints**: Memory, CPU cores, GPU availability
|
||||
- **Software dependencies**: Python version, library compatibility
|
||||
- **Platform differences**: Linux servers, macOS development, Windows deployment
|
||||
Run the comprehensive test suite using pytest:
|
||||
|
||||
### 2. **Reproducibility**
|
||||
Configuration enables reproducible ML:
|
||||
- **Environment documentation**: Exactly what system was used
|
||||
- **Dependency management**: Precise versions and requirements
|
||||
- **Debugging support**: System info helps troubleshoot issues
|
||||
```bash
|
||||
# Using the TinyTorch CLI (recommended)
|
||||
tito test --module setup
|
||||
|
||||
### 3. **Professional Development**
|
||||
Proper configuration shows engineering maturity:
|
||||
- **Attribution**: Your work is properly credited
|
||||
- **Collaboration**: Others can understand and extend your setup
|
||||
- **Maintenance**: Systems can be updated and maintained
|
||||
|
||||
### 4. **ML Systems Context**
|
||||
This connects to broader ML engineering:
|
||||
- **Model deployment**: Different environments need different configs
|
||||
- **Monitoring**: System metrics help track performance
|
||||
- **Scaling**: Understanding hardware helps optimize training
|
||||
|
||||
Let's build the foundation of your ML systems engineering skills!
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/01_setup/setup_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
# Or directly with pytest
|
||||
python -m pytest tests/test_setup.py -v
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/01_setup/setup_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### Test Coverage
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
The test suite includes **20 comprehensive tests** covering:
|
||||
- ✅ **Function execution** - All functions run without errors
|
||||
- ✅ **Output validation** - Correct content and formatting
|
||||
- ✅ **Arithmetic operations** - Basic, negative, and floating-point math
|
||||
- ✅ **System information** - Platform detection and compatibility
|
||||
- ✅ **Developer profiles** - Default and custom configurations
|
||||
- ✅ **ASCII art handling** - File loading and fallback behavior
|
||||
- ✅ **Error recovery** - Graceful handling of missing files
|
||||
- ✅ **Integration testing** - All components work together
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Activate the virtual environment**:
|
||||
```bash
|
||||
source bin/activate-tinytorch.sh
|
||||
```
|
||||
|
||||
2. **Test the setup module**:
|
||||
```bash
|
||||
tito test --module setup
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
This module teaches the core TinyTorch development cycle:
|
||||
|
||||
1. **Write code** in the notebook using `#| export` directives
|
||||
2. **Export code** with `tito sync --module setup`
|
||||
3. **Run tests** with `tito test --module setup`
|
||||
4. **Check progress** with `tito info`
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **NBDev workflow** - Write in notebooks, export to Python packages
|
||||
- **Export directives** - Use `#| export` to mark code for export
|
||||
- **Module → Package mapping** - This module exports to `tinytorch/core/utils.py`
|
||||
- **Teaching vs. Building** - Learn by modules, build by function
|
||||
- **Student implementation** - TODO sections with instructor solutions hidden
|
||||
|
||||
## Personalization Features
|
||||
|
||||
### ASCII Art Customization
|
||||
The ASCII art is loaded from `tinytorch_flame.txt`. You can customize it by:
|
||||
|
||||
1. **Edit the file directly** - Modify `tinytorch_flame.txt` with your own ASCII art
|
||||
2. **Custom parameter** - Pass your own ASCII art to `DeveloperProfile`
|
||||
3. **Create your own design** - Your initials, logo, or motivational art
|
||||
|
||||
### Developer Profile Customization
|
||||
```python
|
||||
my_profile = DeveloperProfile(
|
||||
name="Your Name",
|
||||
affiliation="Your University",
|
||||
email="your.email@example.com",
|
||||
github_username="yourgithub",
|
||||
ascii_art="Your custom ASCII art here!"
|
||||
)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/01_setup/setup_dev.py
|
||||
:class-header: bg-light
|
||||
## What You'll Learn
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
This comprehensive module introduces:
|
||||
- **NBDev educational patterns** - `#| export` directives and NBGrader solution markers
|
||||
- **File I/O operations** - Loading ASCII art with error handling
|
||||
- **Object-oriented programming** - Classes, methods, and properties
|
||||
- **System programming** - Platform detection and compatibility
|
||||
- **Testing with pytest** - Professional test structure and assertions
|
||||
- **Code organization** - Module structure and package exports
|
||||
- **The TinyTorch development workflow** - Complete cycle from code to tests
|
||||
|
||||
````
|
||||
## Next Steps
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
Once you've completed this module and all tests pass, you're ready to move on to the **tensor module** where you'll build the core data structures that power TinyTorch neural networks!
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
The skills you learn here - the development workflow, testing patterns, and code organization - will be used throughout every module in TinyTorch.
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="right-next" href="../chapters/02_tensor.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,237 @@
|
||||
# Tensor - Core Data Structure
|
||||
---
|
||||
title: "Tensor"
|
||||
description: "Core tensor data structure and operations"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: ["✅ Understand what tensors are and why they're essential for ML", '✅ Implement a complete Tensor class with core operations', '✅ Handle tensor shapes, data types, and memory management', '✅ Implement element-wise operations and reductions', '✅ Have a solid foundation for building neural networks']
|
||||
---
|
||||
|
||||
Welcome to the Tensor module! This is where TinyTorch really begins. You'll implement the fundamental data structure that powers all ML systems.
|
||||
# 🔥 Module: Tensor
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 2: 02 Tensor](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand tensors as N-dimensional arrays with ML-specific operations
|
||||
- Implement a complete Tensor class with arithmetic operations
|
||||
- Handle shape management, data types, and memory layout
|
||||
- Build the foundation for neural networks and automatic differentiation
|
||||
- Master the NBGrader workflow with comprehensive testing
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐ | <strong>Time:</strong> 4-6 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐ Intermediate
|
||||
- **Time Estimate**: 4-6 hours
|
||||
- **Prerequisites**: Setup module
|
||||
- **Next Steps**: Activations, Layers
|
||||
|
||||
Build the foundation of TinyTorch! This module implements the core Tensor class - the fundamental data structure that powers all neural networks and machine learning operations.
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
By the end of this module, you will:
|
||||
- ✅ Understand what tensors are and why they're essential for ML
|
||||
- ✅ Implement a complete Tensor class with core operations
|
||||
- ✅ Handle tensor shapes, data types, and memory management
|
||||
- ✅ Implement element-wise operations and reductions
|
||||
- ✅ Have a solid foundation for building neural networks
|
||||
|
||||
## 📋 Module Structure
|
||||
|
||||
```
|
||||
modules/tensor/
|
||||
├── README.md # 📖 This file - Module overview
|
||||
├── tensor_dev.ipynb # 📓 Main development notebook
|
||||
├── test_tensor.py # 🧪 Automated tests
|
||||
└── check_tensor.py # ✅ Manual verification (coming soon)
|
||||
```
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
## Build → Use → Understand
|
||||
1. **Build**: Create the Tensor class with core operations
|
||||
2. **Use**: Perform tensor arithmetic and transformations
|
||||
3. **Understand**: How tensors form the foundation of ML systems
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/02_tensor/tensor_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### Step 1: Complete Prerequisites
|
||||
Make sure you've completed the setup module:
|
||||
```bash
|
||||
python bin/tito.py test --module setup # Should pass
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/02_tensor/tensor_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### Step 2: Open the Tensor Notebook
|
||||
```bash
|
||||
# Start from the tensor module directory
|
||||
cd modules/tensor/
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Open the development notebook
|
||||
jupyter lab tensor_dev.ipynb
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/02_tensor/tensor_dev.py
|
||||
:class-header: bg-light
|
||||
### Step 3: Work Through the Implementation
|
||||
The notebook guides you through building:
|
||||
1. **Basic Tensor class** - Constructor and properties
|
||||
2. **Shape management** - Understanding tensor dimensions
|
||||
3. **Arithmetic operations** - Addition, multiplication, etc.
|
||||
4. **Utility methods** - Reshape, transpose, sum, mean
|
||||
5. **Error handling** - Robust edge case management
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
### Step 4: Export and Test
|
||||
```bash
|
||||
# Export your tensor implementation
|
||||
python bin/tito.py sync
|
||||
|
||||
# Test your implementation
|
||||
python bin/tito.py test --module tensor
|
||||
```
|
||||
|
||||
````
|
||||
## 📚 What You'll Implement
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
### Core Tensor Class
|
||||
You'll build a complete `Tensor` class that supports:
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
#### 1. Construction and Properties
|
||||
```python
|
||||
# Creating tensors
|
||||
a = Tensor([1, 2, 3]) # 1D tensor
|
||||
b = Tensor([[1, 2], [3, 4]]) # 2D tensor
|
||||
c = Tensor(5.0) # Scalar tensor
|
||||
|
||||
# Properties
|
||||
print(a.shape) # (3,)
|
||||
print(b.size) # 4
|
||||
print(c.dtype) # float32
|
||||
```
|
||||
|
||||
#### 2. Arithmetic Operations
|
||||
```python
|
||||
# Element-wise operations
|
||||
result = a + b # Addition
|
||||
result = a * 2 # Scalar multiplication
|
||||
result = a @ b # Matrix multiplication (bonus)
|
||||
```
|
||||
|
||||
#### 3. Utility Methods
|
||||
```python
|
||||
# Shape manipulation
|
||||
reshaped = b.reshape(1, 4) # Change shape
|
||||
transposed = b.transpose() # Swap dimensions
|
||||
|
||||
# Reductions
|
||||
total = a.sum() # Sum all elements
|
||||
mean_val = a.mean() # Average value
|
||||
max_val = a.max() # Maximum value
|
||||
```
|
||||
|
||||
### Technical Requirements
|
||||
Your Tensor class must:
|
||||
- Handle multiple data types (int, float)
|
||||
- Support N-dimensional arrays
|
||||
- Implement proper error checking
|
||||
- Work with NumPy arrays internally
|
||||
- Export to `tinytorch.core.tensor`
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Automated Tests
|
||||
```bash
|
||||
python bin/tito.py test --module tensor
|
||||
```
|
||||
|
||||
Tests verify:
|
||||
- ✅ Tensor creation (scalars, vectors, matrices)
|
||||
- ✅ Property access (shape, size, dtype)
|
||||
- ✅ Arithmetic operations (all combinations)
|
||||
- ✅ Utility methods (reshape, transpose, reductions)
|
||||
- ✅ Error handling (invalid operations)
|
||||
|
||||
### Interactive Testing
|
||||
```python
|
||||
# Test in the notebook or Python REPL
|
||||
from tinytorch.core.tensor import Tensor
|
||||
|
||||
# Create and test tensors
|
||||
a = Tensor([1, 2, 3])
|
||||
b = Tensor([[1, 2], [3, 4]])
|
||||
print(a + 5) # Should work
|
||||
print(a.sum()) # Should return scalar
|
||||
```
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Your tensor module is complete when:
|
||||
|
||||
1. **All tests pass**: `python bin/tito.py test --module tensor`
|
||||
2. **Tensor imports correctly**: `from tinytorch.core.tensor import Tensor`
|
||||
3. **Basic operations work**: Can create tensors and do arithmetic
|
||||
4. **Properties work**: Shape, size, dtype return correct values
|
||||
5. **Utilities work**: Reshape, transpose, reductions function properly
|
||||
|
||||
## 💡 Implementation Tips
|
||||
|
||||
### Start with the Basics
|
||||
1. **Simple constructor** - Handle lists and NumPy arrays
|
||||
2. **Basic properties** - Shape, size, dtype
|
||||
3. **One operation** - Start with addition
|
||||
4. **Test frequently** - Verify each feature works
|
||||
|
||||
### Design Patterns
|
||||
```python
|
||||
class Tensor:
|
||||
def __init__(self, data, dtype=None):
|
||||
# Convert input to numpy array
|
||||
# Store shape, size, dtype
|
||||
|
||||
def __add__(self, other):
|
||||
# Handle tensor + tensor
|
||||
# Handle tensor + scalar
|
||||
# Return new Tensor
|
||||
|
||||
def sum(self, axis=None):
|
||||
# Reduce along specified axis
|
||||
# Return scalar or tensor
|
||||
```
|
||||
|
||||
### Common Challenges
|
||||
- **Shape compatibility** - Check dimensions for operations
|
||||
- **Data type handling** - Convert inputs consistently
|
||||
- **Memory efficiency** - Don't create unnecessary copies
|
||||
- **Error messages** - Provide helpful debugging info
|
||||
|
||||
## 🔧 Advanced Features (Optional)
|
||||
|
||||
If you finish early, try implementing:
|
||||
- **Broadcasting** - Operations on different-shaped tensors
|
||||
- **Slicing** - `tensor[1:3, :]` syntax
|
||||
- **In-place operations** - `tensor += other`
|
||||
- **Matrix multiplication** - `tensor @ other`
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once you complete the tensor module:
|
||||
|
||||
1. **Move to Autograd**: `cd modules/autograd/`
|
||||
2. **Build automatic differentiation**: Enable gradient computation
|
||||
3. **Combine with tensors**: Make tensors differentiable
|
||||
4. **Prepare for neural networks**: Ready for the MLP module
|
||||
|
||||
## 🔗 Why Tensors Matter
|
||||
|
||||
Tensors are the foundation of all ML systems:
|
||||
- **Neural networks** store weights and activations as tensors
|
||||
- **Training** computes gradients on tensors
|
||||
- **Data processing** represents batches as tensors
|
||||
- **GPU acceleration** operates on tensor primitives
|
||||
|
||||
Your tensor implementation will power everything else in TinyTorch!
|
||||
|
||||
## 🎉 Ready to Build?
|
||||
|
||||
The tensor module is where TinyTorch really begins. You're about to create the fundamental building block that will power neural networks, training loops, and production ML systems.
|
||||
|
||||
Take your time, test thoroughly, and enjoy building something that really works! 🔥
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/01_setup.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/03_activations.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,281 @@
|
||||
# Activations - Nonlinearity in Neural Networks
|
||||
---
|
||||
title: "Activation Functions"
|
||||
description: "Neural network activation functions (ReLU, Sigmoid, Tanh, Softmax)"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Activations module! This is where neural networks get their power through nonlinearity.
|
||||
# 🔥 TinyTorch Activations Module
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 3: 03 Activations](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand why activation functions are essential for neural networks
|
||||
- Implement the four most important activation functions: ReLU, Sigmoid, Tanh, and Softmax
|
||||
- Visualize how activations transform data and enable complex learning
|
||||
- See how activations work with layers to build powerful networks
|
||||
- Master the NBGrader workflow with comprehensive testing
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐ | <strong>Time:</strong> 3-4 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐ Intermediate
|
||||
- **Time Estimate**: 3-4 hours
|
||||
- **Prerequisites**: Tensor module
|
||||
- **Next Steps**: Layers module
|
||||
|
||||
Welcome to the **Activations** module! This is where you'll implement the mathematical functions that give neural networks their power to learn complex patterns.
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
By the end of this module, you will:
|
||||
1. **Understand** why activation functions are essential for neural networks
|
||||
2. **Implement** the three most important activation functions: ReLU, Sigmoid, and Tanh
|
||||
3. **Test** your functions with various inputs to understand their behavior
|
||||
4. **Grasp** the mathematical properties that make each function useful
|
||||
|
||||
## 🧠 Why This Module Matters
|
||||
|
||||
**Without activation functions, neural networks are just linear transformations!**
|
||||
|
||||
```
|
||||
Linear → Linear → Linear = Still just Linear
|
||||
Linear → Activation → Linear = Can learn complex patterns!
|
||||
```
|
||||
|
||||
This module teaches you the mathematical foundations that make deep learning possible.
|
||||
|
||||
## Build → Use → Understand
|
||||
1. **Build**: Activation functions that add nonlinearity
|
||||
2. **Use**: Transform tensors and see immediate results
|
||||
3. **Understand**: How nonlinearity enables complex pattern learning
|
||||
## 🚀 Interactive Learning
|
||||
## 📚 What You'll Build
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
### 1. **ReLU** (Rectified Linear Unit)
|
||||
- **Formula**: `f(x) = max(0, x)`
|
||||
- **Properties**: Simple, sparse, unbounded
|
||||
- **Use case**: Hidden layers (most common)
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
### 2. **Sigmoid**
|
||||
- **Formula**: `f(x) = 1 / (1 + e^(-x))`
|
||||
- **Properties**: Bounded to (0,1), smooth, probabilistic
|
||||
- **Use case**: Binary classification, gates
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/03_activations/activations_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### 3. **Tanh** (Hyperbolic Tangent)
|
||||
- **Formula**: `f(x) = tanh(x)`
|
||||
- **Properties**: Bounded to (-1,1), zero-centered, smooth
|
||||
- **Use case**: Hidden layers, RNNs
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Activate the virtual environment**:
|
||||
```bash
|
||||
source bin/activate-tinytorch.sh
|
||||
```
|
||||
|
||||
2. **Start development environment**:
|
||||
```bash
|
||||
tito jupyter
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
|
||||
1. **Open the development file**:
|
||||
```bash
|
||||
# Then open assignments/source/02_activations/activations_dev.py
|
||||
```
|
||||
|
||||
2. **Implement the functions**:
|
||||
- Start with ReLU (simplest)
|
||||
- Move to Sigmoid (numerical stability challenge)
|
||||
- Finish with Tanh (symmetry properties)
|
||||
|
||||
3. **Visualize your functions**:
|
||||
- Each function has plotting sections
|
||||
- See how your implementation transforms inputs
|
||||
- Compare all functions side-by-side
|
||||
|
||||
4. **Test as you go**:
|
||||
```bash
|
||||
tito test --module activations
|
||||
```
|
||||
|
||||
5. **Export to package**:
|
||||
```bash
|
||||
tito sync
|
||||
```
|
||||
|
||||
### 📊 Visual Learning Features
|
||||
|
||||
This module includes comprehensive plotting sections to help you understand:
|
||||
|
||||
- **Individual Function Plots**: See each activation function's curve
|
||||
- **Implementation Comparison**: Your implementation vs ideal side-by-side
|
||||
- **Mathematical Explanations**: Visual breakdown of function properties
|
||||
- **Error Analysis**: Quantitative feedback on implementation accuracy
|
||||
- **Comprehensive Comparison**: All functions analyzed together
|
||||
|
||||
**Enhanced Features**:
|
||||
- **4-Panel Plots**: Implementation vs ideal, mathematical definition, properties, error analysis
|
||||
- **Real-time Feedback**: Immediate accuracy scores with color-coded status
|
||||
- **Mathematical Insights**: Detailed explanations of function properties
|
||||
- **Numerical Stability Testing**: Verification with extreme values
|
||||
- **Property Verification**: Symmetry, monotonicity, and zero-centering tests
|
||||
|
||||
**Why enhanced plots matter**:
|
||||
- **Visual Debugging**: See exactly where your implementation differs
|
||||
- **Quantitative Feedback**: Get precise error measurements
|
||||
- **Mathematical Understanding**: Connect formulas to visual behavior
|
||||
- **Implementation Confidence**: Know immediately if your code is correct
|
||||
- **Learning Reinforcement**: Multiple visual perspectives of the same concept
|
||||
|
||||
### Implementation Tips
|
||||
|
||||
#### ReLU Implementation
|
||||
```python
|
||||
def forward(self, x: Tensor) -> Tensor:
|
||||
return Tensor(np.maximum(0, x.data))
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/03_activations/activations_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
#### Sigmoid Implementation (Numerical Stability)
|
||||
```python
|
||||
def forward(self, x: Tensor) -> Tensor:
|
||||
# For x >= 0: sigmoid(x) = 1 / (1 + exp(-x))
|
||||
# For x < 0: sigmoid(x) = exp(x) / (1 + exp(x))
|
||||
x_data = x.data
|
||||
result = np.zeros_like(x_data)
|
||||
|
||||
positive_mask = x_data >= 0
|
||||
result[positive_mask] = 1.0 / (1.0 + np.exp(-x_data[positive_mask]))
|
||||
result[~positive_mask] = np.exp(x_data[~positive_mask]) / (1.0 + np.exp(x_data[~positive_mask]))
|
||||
|
||||
return Tensor(result)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/03_activations/activations_dev.py
|
||||
:class-header: bg-light
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
#### Tanh Implementation
|
||||
```python
|
||||
def forward(self, x: Tensor) -> Tensor:
|
||||
return Tensor(np.tanh(x.data))
|
||||
```
|
||||
|
||||
````
|
||||
### Testing Your Implementation
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
1. **Run the tests**:
|
||||
```bash
|
||||
tito test --module activations
|
||||
```
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
2. **Export to package**:
|
||||
```bash
|
||||
tito sync
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
```python
|
||||
# Test all activations
|
||||
from tinytorch.core.tensor import Tensor
|
||||
from modules.activations.activations_dev import ReLU, Sigmoid, Tanh
|
||||
|
||||
x = Tensor([[-2.0, -1.0, 0.0, 1.0, 2.0]])
|
||||
|
||||
relu = ReLU()
|
||||
sigmoid = Sigmoid()
|
||||
tanh = Tanh()
|
||||
|
||||
print("Input:", x.data)
|
||||
print("ReLU:", relu(x).data)
|
||||
print("Sigmoid:", sigmoid(x).data)
|
||||
print("Tanh:", tanh(x).data)
|
||||
```
|
||||
|
||||
## 📊 Understanding Function Properties
|
||||
|
||||
### Range Comparison
|
||||
| Function | Input Range | Output Range | Zero Point |
|
||||
|----------|-------------|--------------|------------|
|
||||
| ReLU | (-∞, ∞) | [0, ∞) | f(0) = 0 |
|
||||
| Sigmoid | (-∞, ∞) | (0, 1) | f(0) = 0.5 |
|
||||
| Tanh | (-∞, ∞) | (-1, 1) | f(0) = 0 |
|
||||
|
||||
### Key Properties
|
||||
- **ReLU**: Sparse (zeros out negatives), unbounded, simple
|
||||
- **Sigmoid**: Probabilistic (0-1 range), smooth, saturating
|
||||
- **Tanh**: Zero-centered, symmetric, stronger gradients than sigmoid
|
||||
|
||||
## 🔧 Integration with TinyTorch
|
||||
|
||||
After implementation, your activations will be available as:
|
||||
|
||||
```python
|
||||
from tinytorch.core.activations import ReLU, Sigmoid, Tanh
|
||||
|
||||
# Use in neural networks
|
||||
relu = ReLU()
|
||||
output = relu(input_tensor)
|
||||
```
|
||||
|
||||
## 🎯 Common Issues & Solutions
|
||||
|
||||
### Issue 1: Sigmoid Overflow
|
||||
**Problem**: `exp()` overflow with large inputs
|
||||
**Solution**: Use numerically stable implementation (see code above)
|
||||
|
||||
### Issue 2: Wrong Output Range
|
||||
**Problem**: Sigmoid/Tanh outputs outside expected range
|
||||
**Solution**: Check your mathematical implementation
|
||||
|
||||
### Issue 3: Shape Mismatch
|
||||
**Problem**: Output shape differs from input shape
|
||||
**Solution**: Ensure element-wise operations preserve shape
|
||||
|
||||
### Issue 4: Import Errors
|
||||
**Problem**: Cannot import after implementation
|
||||
**Solution**: Run `tito sync` to export to package
|
||||
|
||||
## 📈 Performance Considerations
|
||||
|
||||
- **ReLU**: Fastest (simple max operation)
|
||||
- **Sigmoid**: Moderate (exponential computation)
|
||||
- **Tanh**: Moderate (hyperbolic function)
|
||||
|
||||
All implementations use NumPy for vectorized operations.
|
||||
|
||||
## 🚀 What's Next
|
||||
|
||||
After mastering activations, you'll use them in:
|
||||
1. **Layers Module**: Building neural network layers
|
||||
2. **Loss Functions**: Computing training objectives
|
||||
3. **Advanced Architectures**: CNNs, RNNs, and more
|
||||
|
||||
These functions are the mathematical foundation for everything that follows!
|
||||
|
||||
## 📚 Further Reading
|
||||
|
||||
**Mathematical Background**:
|
||||
- [Activation Functions in Neural Networks](https://en.wikipedia.org/wiki/Activation_function)
|
||||
- [Deep Learning Book - Chapter 6](http://www.deeplearningbook.org/)
|
||||
|
||||
**Advanced Topics**:
|
||||
- ReLU variants (Leaky ReLU, ELU, Swish)
|
||||
- Activation function choice and impact
|
||||
- Gradient flow and vanishing gradients
|
||||
|
||||
## 🎉 Success Criteria
|
||||
|
||||
You've mastered this module when:
|
||||
- [ ] All tests pass (`tito test --module activations`)
|
||||
- [ ] You understand why each function is useful
|
||||
- [ ] You can explain the mathematical properties
|
||||
- [ ] You can use activations in neural networks
|
||||
- [ ] You appreciate the importance of nonlinearity
|
||||
|
||||
**Great work! You've built the mathematical foundation of neural networks!** 🎉
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/02_tensor.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/04_layers.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,241 @@
|
||||
# Layers - Building Blocks of Neural Networks
|
||||
---
|
||||
title: "Layers"
|
||||
description: "Neural network layers (Linear, activation layers)"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Layers module! This is where we build the fundamental components that stack together to form neural networks.
|
||||
# 🧱 Module 2: Layers - Neural Network Building Blocks
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 4: 04 Layers](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand how matrix multiplication powers neural networks
|
||||
- Implement naive matrix multiplication from scratch for deep understanding
|
||||
- Build the Dense (Linear) layer - the foundation of all neural networks
|
||||
- Learn weight initialization strategies and their importance
|
||||
- See how layers compose with activations to create powerful networks
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐ | <strong>Time:</strong> 4-5 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐ Intermediate
|
||||
- **Time Estimate**: 4-5 hours
|
||||
- **Prerequisites**: Tensor, Activations modules
|
||||
- **Next Steps**: Networks module
|
||||
|
||||
**Build the fundamental transformations that compose into neural networks**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand layers as functions that transform tensors: `y = f(x)`
|
||||
- Implement Dense layers with linear transformations: `y = Wx + b`
|
||||
- Add activation functions for nonlinearity (ReLU, Sigmoid, Tanh)
|
||||
- See how neural networks are just function composition
|
||||
- Build intuition for neural network architecture before diving into training
|
||||
|
||||
## 🧱 Build → Use → Understand
|
||||
|
||||
This module follows the TinyTorch pedagogical framework:
|
||||
|
||||
1. **Build**: Dense layers and activation functions from scratch
|
||||
2. **Use**: Transform tensors and see immediate results
|
||||
3. **Understand**: How neural networks transform information
|
||||
|
||||
## 📚 What You'll Build
|
||||
|
||||
### **Dense Layer**
|
||||
```python
|
||||
layer = Dense(input_size=3, output_size=2)
|
||||
x = Tensor([[1.0, 2.0, 3.0]])
|
||||
y = layer(x) # Shape: (1, 2)
|
||||
```
|
||||
|
||||
### **Activation Functions**
|
||||
```python
|
||||
relu = ReLU()
|
||||
sigmoid = Sigmoid()
|
||||
tanh = Tanh()
|
||||
|
||||
## Build → Use → Understand
|
||||
1. **Build**: Matrix multiplication and Dense layers from scratch
|
||||
2. **Use**: Create and test layers with real data
|
||||
3. **Understand**: How linear transformations enable feature learning
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/04_layers/layers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
x = Tensor([[-1.0, 0.0, 1.0]])
|
||||
y_relu = relu(x) # [0.0, 0.0, 1.0]
|
||||
y_sigmoid = sigmoid(x) # [0.27, 0.5, 0.73]
|
||||
y_tanh = tanh(x) # [-0.76, 0.0, 0.76]
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/04_layers/layers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### **Neural Networks**
|
||||
```python
|
||||
# 3 → 4 → 2 network
|
||||
layer1 = Dense(input_size=3, output_size=4)
|
||||
activation1 = ReLU()
|
||||
layer2 = Dense(input_size=4, output_size=2)
|
||||
activation2 = Sigmoid()
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Forward pass
|
||||
x = Tensor([[1.0, 2.0, 3.0]])
|
||||
h1 = layer1(x)
|
||||
h1_activated = activation1(h1)
|
||||
h2 = layer2(h1_activated)
|
||||
output = activation2(h2)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/04_layers/layers_dev.py
|
||||
:class-header: bg-light
|
||||
## 🚀 Getting Started
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
### Prerequisites
|
||||
- Complete Module 1: Tensor ✅
|
||||
- Understand basic linear algebra (matrix multiplication)
|
||||
- Familiar with Python classes and methods
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
# Navigate to the layers module
|
||||
cd modules/layers
|
||||
|
||||
# Work in the development notebook
|
||||
jupyter notebook layers_dev.ipynb
|
||||
|
||||
# Or work in the Python file
|
||||
code layers_dev.py
|
||||
```
|
||||
|
||||
````
|
||||
## 📖 Module Structure
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
modules/layers/
|
||||
├── layers_dev.py # Main development file (work here!)
|
||||
├── layers_dev.ipynb # Jupyter notebook version
|
||||
├── tests/
|
||||
│ └── test_layers.py # Comprehensive tests
|
||||
├── README.md # This file
|
||||
└── solutions/ # Reference implementations (if stuck)
|
||||
```
|
||||
|
||||
## 🎓 Learning Path
|
||||
|
||||
### Step 1: Dense Layer (Linear Transformation)
|
||||
- Understand `y = Wx + b`
|
||||
- Implement weight initialization
|
||||
- Handle matrix multiplication and bias addition
|
||||
- Test with single examples and batches
|
||||
|
||||
### Step 2: Activation Functions
|
||||
- Implement ReLU: `max(0, x)`
|
||||
- Implement Sigmoid: `1 / (1 + e^(-x))`
|
||||
- Implement Tanh: `tanh(x)`
|
||||
- Understand why nonlinearity is crucial
|
||||
|
||||
### Step 3: Layer Composition
|
||||
- Chain layers together
|
||||
- Build complete neural networks
|
||||
- See how simple layers create complex functions
|
||||
|
||||
### Step 4: Real-World Application
|
||||
- Build an image classification network
|
||||
- Understand how architecture affects capability
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Module-Level Tests
|
||||
```bash
|
||||
# Run comprehensive tests
|
||||
python -m pytest tests/test_layers.py -v
|
||||
|
||||
# Quick test
|
||||
python -c "from layers_dev import Dense, ReLU; print('✅ Layers working!')"
|
||||
```
|
||||
|
||||
### Package-Level Tests
|
||||
```bash
|
||||
# Export to package
|
||||
python ../../bin/tito.py sync
|
||||
|
||||
# Test integration
|
||||
python ../../bin/tito.py test --module layers
|
||||
```
|
||||
|
||||
## 🎯 Key Concepts
|
||||
|
||||
### **Layers as Functions**
|
||||
- Input: Tensor with some shape
|
||||
- Transformation: Mathematical operation
|
||||
- Output: Tensor with possibly different shape
|
||||
|
||||
### **Linear vs Nonlinear**
|
||||
- Dense layers: Linear transformations
|
||||
- Activation functions: Nonlinear transformations
|
||||
- Composition: Linear + Nonlinear = Complex functions
|
||||
|
||||
### **Neural Networks = Function Composition**
|
||||
```
|
||||
Input → Dense → ReLU → Dense → Sigmoid → Output
|
||||
```
|
||||
|
||||
### **Why This Matters**
|
||||
- **Modularity**: Build complex networks from simple parts
|
||||
- **Reusability**: Same layers work for different problems
|
||||
- **Understanding**: Know how each part contributes to the whole
|
||||
|
||||
## 🔍 Common Issues
|
||||
|
||||
### **Import Errors**
|
||||
```python
|
||||
# Make sure you're in the right directory
|
||||
import sys
|
||||
sys.path.append('../../')
|
||||
from modules.tensor.tensor_dev import Tensor
|
||||
```
|
||||
|
||||
### **Shape Mismatches**
|
||||
```python
|
||||
# Check input/output sizes match
|
||||
layer1 = Dense(input_size=3, output_size=4)
|
||||
layer2 = Dense(input_size=4, output_size=2) # 4 matches output of layer1
|
||||
```
|
||||
|
||||
### **Gradient Issues (Later)**
|
||||
```python
|
||||
# Use proper weight initialization
|
||||
limit = math.sqrt(6.0 / (input_size + output_size))
|
||||
weights = np.random.uniform(-limit, limit, (input_size, output_size))
|
||||
```
|
||||
|
||||
## 🎉 Success Criteria
|
||||
|
||||
You've successfully completed this module when:
|
||||
- ✅ All tests pass (`pytest tests/test_layers.py`)
|
||||
- ✅ You can build a 2-layer neural network
|
||||
- ✅ You understand how layers transform tensors
|
||||
- ✅ You see the connection between layers and neural networks
|
||||
- ✅ Package export works (`tito test --module layers`)
|
||||
|
||||
## 🚀 What's Next
|
||||
|
||||
After completing this module, you're ready for:
|
||||
- **Module 3: Networks** - Compose layers into common architectures
|
||||
- **Module 4: Training** - Learn how networks improve through experience
|
||||
- **Module 5: Applications** - Use networks for real problems
|
||||
|
||||
## 🤝 Getting Help
|
||||
|
||||
- Check the tests for examples of expected behavior
|
||||
- Look at the solutions/ directory if you're stuck
|
||||
- Review the pedagogical principles in `docs/pedagogy/`
|
||||
- Remember: Build → Use → Understand!
|
||||
|
||||
---
|
||||
|
||||
**Great job building the foundation of neural networks!** 🎉
|
||||
|
||||
*This module implements the core insight: neural networks are just function composition of simple building blocks.*
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/03_activations.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/05_networks.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,62 +1,305 @@
|
||||
# Networks - Neural Network Architectures
|
||||
---
|
||||
title: "Networks"
|
||||
description: "Neural network architectures and composition patterns"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Networks module! This is where we compose layers into complete neural network architectures.
|
||||
# 🧠 Module 3: Networks - Neural Network Architectures
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 5: 05 Networks](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐ | <strong>Time:</strong> 5-6 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 5-7 hours
|
||||
- **Prerequisites**: Tensor, Activations, Layers modules
|
||||
- **Next Steps**: Training, CNN modules
|
||||
|
||||
**Compose layers into complete neural network architectures with powerful visualizations**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand networks as function composition: `f(x) = layer_n(...layer_2(layer_1(x)))`
|
||||
- Build the Sequential network architecture for composing layers
|
||||
- Create common network patterns like MLPs (Multi-Layer Perceptrons)
|
||||
- Visualize network architectures and understand their capabilities
|
||||
- Master forward pass inference through complete networks
|
||||
- Build common architectures (MLP, CNN) from layers
|
||||
- Visualize network structure and data flow
|
||||
- See how architecture affects capability
|
||||
- Master forward pass inference (no training yet!)
|
||||
|
||||
> **Note:**
|
||||
> **MLP (Multi-Layer Perceptron) is not a fundamental building block, but a use case of composing Dense layers and activations in sequence.**
|
||||
> In TinyTorch, you will learn to build MLPs by composing primitives, not as a separate module. This approach helps you see that all architectures (MLP, CNN, etc.) are just patterns of composition, not new primitives.
|
||||
|
||||
## 🧠 Build → Use → Understand
|
||||
|
||||
This module follows the TinyTorch pedagogical framework:
|
||||
|
||||
1. **Build**: Compose layers into complete networks
|
||||
2. **Use**: Create different architectures and run inference
|
||||
3. **Understand**: How architecture design affects network behavior
|
||||
|
||||
## 📚 What You'll Build
|
||||
|
||||
### **Sequential Network**
|
||||
```python
|
||||
# Basic network composition
|
||||
network = Sequential([
|
||||
Dense(784, 128),
|
||||
ReLU(),
|
||||
Dense(128, 64),
|
||||
ReLU(),
|
||||
Dense(64, 10),
|
||||
Sigmoid()
|
||||
])
|
||||
|
||||
# Forward pass
|
||||
x = Tensor([[1.0, 2.0, 3.0, ...]]) # Input data
|
||||
output = network(x) # Network prediction
|
||||
```
|
||||
|
||||
|
||||
## Build → Use → Reflect
|
||||
1. **Build**: Sequential networks that compose layers into complete architectures
|
||||
2. **Use**: Create different network patterns and run inference
|
||||
3. **Reflect**: How architecture design affects network behavior and capability
|
||||
|
||||
## What You'll Learn
|
||||
By the end of this module, you'll understand:
|
||||
- How simple layers combine to create complex behaviors
|
||||
- The fundamental Sequential architecture pattern
|
||||
- How to build MLPs with any number of layers
|
||||
- Different network architectures (shallow, deep, wide)
|
||||
- How neural networks approximate complex functions
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/05_networks/networks_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### **MLP (Multi-Layer Perceptron)**
|
||||
```python
|
||||
# Create MLP for classification
|
||||
mlp = create_mlp(
|
||||
input_size=784, # 28x28 image
|
||||
hidden_sizes=[128, 64], # Hidden layers
|
||||
output_size=10, # 10 classes
|
||||
activation=ReLU,
|
||||
output_activation=Sigmoid
|
||||
)
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/05_networks/networks_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### **Specialized Networks**
|
||||
```python
|
||||
# Classification network
|
||||
classifier = create_classification_network(
|
||||
input_size=100, num_classes=2
|
||||
)
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Regression network
|
||||
regressor = create_regression_network(
|
||||
input_size=13, output_size=1
|
||||
)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/05_networks/networks_dev.py
|
||||
:class-header: bg-light
|
||||
## 🎨 Visualization Features
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
This module includes powerful visualizations to help you understand:
|
||||
|
||||
### **Network Architecture Visualization**
|
||||
- **Layer-by-layer structure**: See how layers connect
|
||||
- **Color-coded layers**: Different colors for Dense, ReLU, Sigmoid, etc.
|
||||
- **Connection arrows**: Visualize data flow between layers
|
||||
- **Layer details**: Input/output sizes and parameters
|
||||
|
||||
### **Data Flow Visualization**
|
||||
- **Shape transformations**: See how tensor shapes change through the network
|
||||
- **Activation patterns**: Visualize intermediate layer outputs
|
||||
- **Statistics tracking**: Mean, std, and distribution of activations
|
||||
- **Layer analysis**: Understand what each layer learns
|
||||
|
||||
### **Network Comparison**
|
||||
- **Side-by-side analysis**: Compare different architectures
|
||||
- **Performance metrics**: Output distributions and statistics
|
||||
- **Architectural insights**: Layer type distributions and complexity
|
||||
|
||||
### **Behavior Analysis**
|
||||
- **Input-output relationships**: How inputs map to outputs
|
||||
- **Activation patterns**: Layer-by-layer activation analysis
|
||||
- **Network depth**: Understanding the role of depth vs width
|
||||
- **Practical insights**: Real-world application considerations
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- Complete Module 1: Tensor ✅
|
||||
- Complete Module 2: Layers ✅
|
||||
- Understand basic function composition
|
||||
- Familiar with matplotlib for visualizations
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
# Navigate to the networks module
|
||||
cd modules/networks
|
||||
|
||||
# Work in the development notebook
|
||||
jupyter notebook networks_dev.ipynb
|
||||
|
||||
# Or work in the Python file
|
||||
code networks_dev.py
|
||||
```
|
||||
|
||||
````
|
||||
## 📖 Module Structure
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
modules/networks/
|
||||
├── networks_dev.py # Main development file (work here!)
|
||||
├── networks_dev.ipynb # Jupyter notebook version
|
||||
├── tests/
|
||||
│ └── test_networks.py # Comprehensive tests
|
||||
├── README.md # This file
|
||||
└── solutions/ # Reference implementations (if stuck)
|
||||
```
|
||||
|
||||
## 🎓 Learning Path
|
||||
|
||||
### Step 1: Sequential Network (Function Composition)
|
||||
- Understand `f(x) = layer_n(...layer_1(x))`
|
||||
- Implement basic network composition
|
||||
- Test with simple examples
|
||||
|
||||
### Step 2: Network Visualization
|
||||
- Visualize network architectures
|
||||
- Understand data flow through networks
|
||||
- Compare different network designs
|
||||
|
||||
### Step 3: Common Architectures
|
||||
- Build MLPs for different tasks
|
||||
- Create classification networks
|
||||
- Design regression networks
|
||||
|
||||
### Step 4: Behavior Analysis
|
||||
- Analyze network behavior with different inputs
|
||||
- Understand architectural trade-offs
|
||||
- See how design affects capability
|
||||
|
||||
### Step 5: Practical Applications
|
||||
- Build networks for real problems
|
||||
- Understand classification vs regression
|
||||
- See how architecture matches task
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Module-Level Tests
|
||||
```bash
|
||||
# Run comprehensive tests
|
||||
python -m pytest tests/test_networks.py -v
|
||||
|
||||
# Quick test
|
||||
python -c "from networks_dev import Sequential; print('✅ Networks working!')"
|
||||
```
|
||||
|
||||
### Package-Level Tests
|
||||
```bash
|
||||
# Export to package
|
||||
python ../../bin/tito sync
|
||||
|
||||
# Test integration
|
||||
python ../../bin/tito test --module networks
|
||||
```
|
||||
|
||||
## 🎯 Key Concepts
|
||||
|
||||
### **Function Composition**
|
||||
- Networks as `f(x) = g(h(x))`
|
||||
- Each layer is a function
|
||||
- Composition creates complex behavior
|
||||
|
||||
### **Architecture Design**
|
||||
- **Depth**: Number of layers
|
||||
- **Width**: Number of neurons per layer
|
||||
- **Activation**: Nonlinearity choices
|
||||
- **Output**: Task-specific final layer
|
||||
|
||||
### **Visualization Benefits**
|
||||
- **Debugging**: See where things go wrong
|
||||
- **Understanding**: Visualize complex transformations
|
||||
- **Design**: Compare different architectures
|
||||
- **Intuition**: Build mental models of networks
|
||||
|
||||
### **Practical Considerations**
|
||||
- **Input size**: Must match your data
|
||||
- **Output size**: Must match your task
|
||||
- **Hidden layers**: Balance complexity vs overfitting
|
||||
- **Activation functions**: Choose based on task
|
||||
|
||||
## 🔍 Common Issues
|
||||
|
||||
### **Import Errors**
|
||||
```python
|
||||
# Make sure you're in the right directory
|
||||
import sys
|
||||
sys.path.append('../../')
|
||||
from modules.layers.layers_dev import Dense
|
||||
from modules.activations.activations_dev import ReLU, Sigmoid
|
||||
```
|
||||
|
||||
### **Shape Mismatches**
|
||||
```python
|
||||
# Check layer sizes match
|
||||
layer1 = Dense(3, 4) # 3 inputs, 4 outputs
|
||||
layer2 = Dense(4, 2) # 4 inputs (matches layer1 output), 2 outputs
|
||||
```
|
||||
|
||||
### **Visualization Issues**
|
||||
```python
|
||||
# Make sure matplotlib is installed
|
||||
pip install matplotlib seaborn
|
||||
|
||||
# Check if plots are disabled during testing
|
||||
if _should_show_plots():
|
||||
# Your visualization code
|
||||
pass
|
||||
```
|
||||
|
||||
## 🎉 Success Criteria
|
||||
|
||||
You've successfully completed this module when:
|
||||
- ✅ All tests pass (`pytest tests/test_networks.py`)
|
||||
- ✅ You can build and visualize different network architectures
|
||||
- ✅ You understand how architecture affects network behavior
|
||||
- ✅ You can create networks for classification and regression tasks
|
||||
- ✅ Package export works (`tito test --module networks`)
|
||||
|
||||
## 🚀 What's Next
|
||||
|
||||
After completing this module, you're ready for:
|
||||
- **Module 4: Training** - Learn how networks learn from data
|
||||
- **Module 5: Data** - Work with real datasets
|
||||
- **Module 6: Applications** - Solve real-world problems
|
||||
|
||||
## 🤝 Getting Help
|
||||
|
||||
- Check the tests for examples of expected behavior
|
||||
- Look at the solutions/ directory if you're stuck
|
||||
- Review the pedagogical principles in `docs/pedagogy/`
|
||||
- Remember: Build → Use → Understand!
|
||||
|
||||
## 🎨 Visualization Examples
|
||||
|
||||
### Network Architecture
|
||||
```
|
||||
Input → Dense(784,128) → ReLU → Dense(128,64) → ReLU → Dense(64,10) → Sigmoid → Output
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
```
|
||||
(1,784) → (1,128) → (1,128) → (1,64) → (1,64) → (1,10) → (1,10)
|
||||
```
|
||||
|
||||
### Layer Analysis
|
||||
- **Dense layers**: Linear transformations
|
||||
- **ReLU**: Introduces nonlinearity
|
||||
- **Sigmoid**: Outputs probabilities
|
||||
|
||||
**Build powerful neural networks with beautiful visualizations!** 🚀
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/04_layers.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/06_cnn.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,62 +1,99 @@
|
||||
# CNN - Convolutional Neural Networks
|
||||
---
|
||||
title: "CNN"
|
||||
description: "Convolutional Neural Network layers and operations"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: ['Understand the convolution operation (sliding window, local connectivity, weight sharing)', 'Implement Conv2D with explicit for-loops (single channel, single filter, no stride/pad)', 'Visualize how convolution builds feature maps', 'Compose Conv2D with other layers to build a simple ConvNet', '(Stretch) Explore stride, padding, pooling, and multi-channel input']
|
||||
---
|
||||
|
||||
Welcome to the CNN module! Here you'll implement the core building block of modern computer vision: the convolutional layer.
|
||||
# 🧠 Module X: CNN - Convolutional Neural Networks
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 6: 06 Cnn](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand the convolution operation and its importance in computer vision
|
||||
- Implement Conv2D with explicit for-loops to understand the sliding window mechanism
|
||||
- Build convolutional layers that can detect spatial patterns in images
|
||||
- Compose Conv2D with other layers to build complete convolutional networks
|
||||
- See how convolution enables parameter sharing and translation invariance
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐ | <strong>Time:</strong> 6-8 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 6-8 hours
|
||||
- **Prerequisites**: Tensor, Activations, Layers, Networks modules
|
||||
- **Next Steps**: Training, Computer Vision modules
|
||||
|
||||
**Implement the core building block of modern computer vision: the convolutional layer.**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
- Understand the convolution operation (sliding window, local connectivity, weight sharing)
|
||||
- Implement Conv2D with explicit for-loops (single channel, single filter, no stride/pad)
|
||||
- Visualize how convolution builds feature maps
|
||||
- Compose Conv2D with other layers to build a simple ConvNet
|
||||
- (Stretch) Explore stride, padding, pooling, and multi-channel input
|
||||
|
||||
## 🧠 Build → Use → Understand
|
||||
1. **Build**: Implement Conv2D from scratch (for-loop)
|
||||
2. **Use**: Compose Conv2D with ReLU, Flatten, Dense to build a ConvNet
|
||||
3. **Understand**: Visualize and analyze how convolution works
|
||||
|
||||
## 📚 What You'll Build
|
||||
- **Conv2D (for-loop):** The core operation, implemented by you
|
||||
- **Conv2D Layer:** Wrap your function in a layer class
|
||||
- **Simple ConvNet:** Compose Conv2D → ReLU → Flatten → Dense
|
||||
- **Visualization:** See how the filter slides and builds the output
|
||||
|
||||
## 🛠️ Provided Functionality
|
||||
- **Stride and Padding:** Provided as utilities or stretch goals
|
||||
- **Multi-channel/Filter Support:** Provided or as stretch
|
||||
- **Pooling (Max/Avg):** Optional, provided or as stretch
|
||||
- **Flatten Layer:** Provided
|
||||
- **Visualization:** Provided for learning
|
||||
- **Tests:** Provided for feedback
|
||||
|
||||
## 🤔 Why Focus on the For-Loop?
|
||||
Implementing the convolution for-loop is the best way to understand what makes CNNs powerful. You’ll see exactly how the filter slides, how local patterns are captured, and why this operation is so efficient for images. Other features (stride, padding, pooling) are important, but the core insight comes from building the basic operation yourself.
|
||||
|
||||
## 🚀 Getting Started
|
||||
```bash
|
||||
cd modules/cnn
|
||||
jupyter notebook cnn_dev.ipynb # or edit cnn_dev.py
|
||||
```
|
||||
|
||||
|
||||
## Build → Use → Reflect
|
||||
1. **Build**: Conv2D layer using sliding window convolution from scratch
|
||||
2. **Use**: Transform images and see feature maps emerge
|
||||
3. **Reflect**: How CNNs learn hierarchical spatial patterns
|
||||
|
||||
## What You'll Learn
|
||||
By the end of this module, you'll understand:
|
||||
- How convolution works as a sliding window operation
|
||||
- Why convolution is perfect for spatial data like images
|
||||
- How to build learnable convolutional layers
|
||||
- The CNN pipeline: Conv2D → Activation → Flatten → Dense
|
||||
- How parameter sharing makes CNNs efficient
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/06_cnn/cnn_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
## 📖 Module Structure
|
||||
```
|
||||
modules/cnn/
|
||||
├── cnn_dev.py # Main development file (work here!)
|
||||
├── cnn_dev.ipynb # Jupyter notebook version
|
||||
├── tests/
|
||||
│ └── test_cnn.py # Tests for your implementation
|
||||
├── README.md # This file
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/06_cnn/cnn_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
## 🧪 Testing Your Implementation
|
||||
```bash
|
||||
# Run tests
|
||||
python -m pytest tests/test_cnn.py -v
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/06_cnn/cnn_dev.py
|
||||
:class-header: bg-light
|
||||
## 🌟 Stretch Goals
|
||||
- Add stride and padding support
|
||||
- Support multi-channel input/output
|
||||
- Implement pooling layers
|
||||
- Visualize learned filters and feature maps
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
## 💡 Key Insight
|
||||
> **Convolution is a new, fundamental building block.**
|
||||
> By implementing it yourself, you’ll understand the magic behind modern vision models!
|
||||
|
||||
````
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/05_networks.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/07_dataloader.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,62 +1,343 @@
|
||||
# DataLoader - Data Loading and Preprocessing
|
||||
---
|
||||
title: "DataLoader"
|
||||
description: "Dataset interfaces and data loading pipelines"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: ['✅ Understand data engineering as the foundation of ML systems', '✅ Implement reusable dataset abstractions and interfaces', '✅ Build efficient data loaders with batching and shuffling', '✅ Create data preprocessing pipelines for normalization', '✅ Apply systems thinking to data I/O and memory management', '✅ Have a complete data pipeline ready for neural network training']
|
||||
---
|
||||
|
||||
Welcome to the DataLoader module! This is where you'll learn how to efficiently load, process, and manage data for machine learning systems.
|
||||
# 🔥 Module: DataLoader
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 7: 07 Dataloader](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand data pipelines as the foundation of ML systems
|
||||
- Implement efficient data loading with memory management and batching
|
||||
- Build reusable dataset abstractions for different data types
|
||||
- Master the Dataset and DataLoader pattern used in all ML frameworks
|
||||
- Learn systems thinking for data engineering and I/O optimization
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐ | <strong>Time:</strong> 4-5 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 5-7 hours
|
||||
- **Prerequisites**: Tensor, Layers modules
|
||||
- **Next Steps**: Training, Networks modules
|
||||
|
||||
Build the data pipeline foundation of TinyTorch! This module implements efficient data loading, preprocessing, and batching systems - the critical infrastructure that feeds neural networks during training.
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
By the end of this module, you will:
|
||||
- ✅ Understand data engineering as the foundation of ML systems
|
||||
- ✅ Implement reusable dataset abstractions and interfaces
|
||||
- ✅ Build efficient data loaders with batching and shuffling
|
||||
- ✅ Create data preprocessing pipelines for normalization
|
||||
- ✅ Apply systems thinking to data I/O and memory management
|
||||
- ✅ Have a complete data pipeline ready for neural network training
|
||||
|
||||
## 📋 Module Structure
|
||||
|
||||
```
|
||||
modules/dataloader/
|
||||
├── README.md # 📖 This file - Module overview
|
||||
├── dataloader_dev.py # 🔧 Main development file
|
||||
├── dataloader_dev.ipynb # 📓 Generated notebook (auto-created)
|
||||
├── tests/
|
||||
│ └── test_dataloader.py # 🧪 Automated tests
|
||||
└── check_dataloader.py # ✅ Manual verification (coming soon)
|
||||
```
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
## Build → Use → Reflect
|
||||
1. **Build**: Create dataset classes and data loaders from scratch
|
||||
2. **Use**: Load real datasets and feed them to neural networks
|
||||
3. **Reflect**: How data engineering affects system performance and scalability
|
||||
|
||||
## What You'll Learn
|
||||
By the end of this module, you'll understand:
|
||||
- The Dataset pattern for consistent data access
|
||||
- How DataLoaders enable efficient batch processing
|
||||
- Why batching and shuffling are crucial for ML
|
||||
- How to handle datasets larger than memory
|
||||
- The connection between data engineering and model performance
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/07_dataloader/dataloader_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### Step 1: Complete Prerequisites
|
||||
Make sure you've completed the foundational modules:
|
||||
```bash
|
||||
tito test --module setup # Should pass
|
||||
tito test --module tensor # Should pass
|
||||
tito test --module layers # Should pass
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/07_dataloader/dataloader_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### Step 2: Open the Data Development File
|
||||
```bash
|
||||
# Start from the dataloader module directory
|
||||
cd modules/dataloader/
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Convert to notebook if needed
|
||||
tito notebooks --module dataloader
|
||||
|
||||
# Open the development notebook
|
||||
jupyter lab dataloader_dev.ipynb
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/07_dataloader/dataloader_dev.py
|
||||
:class-header: bg-light
|
||||
### Step 3: Work Through the Implementation
|
||||
The development file guides you through building:
|
||||
1. **Dataset base class** - Abstract interface for all datasets
|
||||
2. **CIFAR-10 implementation** - Real dataset with binary file parsing
|
||||
3. **DataLoader** - Efficient batching and shuffling system
|
||||
4. **Normalizer** - Data preprocessing for stable training
|
||||
5. **Complete pipeline** - Integration of all components
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
### Step 4: Export and Test
|
||||
```bash
|
||||
# Export your dataloader implementation
|
||||
tito sync --module dataloader
|
||||
|
||||
# Test your implementation
|
||||
tito test --module dataloader
|
||||
```
|
||||
|
||||
````
|
||||
## 📚 What You'll Implement
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
### Core Data Infrastructure
|
||||
You'll build a complete data loading system that supports:
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
#### 1. Dataset Abstraction
|
||||
```python
|
||||
# Abstract base class for all datasets
|
||||
class Dataset:
|
||||
def __getitem__(self, index):
|
||||
# Get single sample and label
|
||||
pass
|
||||
|
||||
def __len__(self):
|
||||
# Get total number of samples
|
||||
pass
|
||||
|
||||
def get_num_classes(self):
|
||||
# Get number of classes
|
||||
pass
|
||||
|
||||
# Concrete implementation
|
||||
dataset = CIFAR10Dataset("data/cifar10/", train=True)
|
||||
image, label = dataset[0] # Get first sample
|
||||
```
|
||||
|
||||
#### 2. Real Dataset Loading
|
||||
```python
|
||||
# CIFAR-10 dataset with download and parsing
|
||||
dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True)
|
||||
print(f"Dataset size: {len(dataset)}") # 50,000 training samples
|
||||
print(f"Sample shape: {dataset.get_sample_shape()}") # (3, 32, 32)
|
||||
print(f"Classes: {dataset.get_num_classes()}") # 10 classes
|
||||
```
|
||||
|
||||
#### 3. Efficient Data Loading
|
||||
```python
|
||||
# DataLoader with batching and shuffling
|
||||
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)
|
||||
for batch_images, batch_labels in dataloader:
|
||||
print(f"Batch shape: {batch_images.shape}") # (32, 3, 32, 32)
|
||||
print(f"Labels shape: {batch_labels.shape}") # (32,)
|
||||
# Ready for neural network training!
|
||||
```
|
||||
|
||||
#### 4. Data Preprocessing
|
||||
```python
|
||||
# Normalizer for stable training
|
||||
normalizer = Normalizer()
|
||||
normalizer.fit(training_data) # Compute statistics
|
||||
normalized_data = normalizer.transform(test_data) # Apply normalization
|
||||
```
|
||||
|
||||
#### 5. Complete Pipeline
|
||||
```python
|
||||
# One-function pipeline creation
|
||||
train_loader, test_loader, normalizer = create_data_pipeline(
|
||||
dataset_path="data/cifar10/",
|
||||
batch_size=32,
|
||||
normalize=True,
|
||||
shuffle=True
|
||||
)
|
||||
```
|
||||
|
||||
### Technical Requirements
|
||||
Your data system must:
|
||||
- Handle multiple dataset types through common interface
|
||||
- Efficiently load and parse binary data files
|
||||
- Support batching with configurable batch sizes
|
||||
- Implement shuffling for training randomization
|
||||
- Provide data normalization for stable training
|
||||
- Export to `tinytorch.core.dataloader`
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Progressive Testing with Real Data
|
||||
|
||||
The tests follow the **"Build → Use → Understand"** pattern with real CIFAR-10 data:
|
||||
|
||||
```bash
|
||||
# Run all tests (downloads real CIFAR-10 data)
|
||||
tito test --module dataloader
|
||||
|
||||
# Run specific test categories
|
||||
python -m pytest tests/test_dataloader.py::TestDatasetInterface -v # Test abstract interface
|
||||
python -m pytest tests/test_dataloader.py::TestCIFAR10Dataset -v # Test real data loading
|
||||
python -m pytest tests/test_dataloader.py::TestDataLoader -v # Test batching real data
|
||||
python -m pytest tests/test_dataloader.py::TestNormalizer -v # Test normalizing real data
|
||||
python -m pytest tests/test_dataloader.py::TestDataPipeline -v # Test complete pipeline
|
||||
```
|
||||
|
||||
### Real Data Testing Flow
|
||||
|
||||
Each test builds on the previous component using actual CIFAR-10 data:
|
||||
|
||||
1. **Build Dataset** → **Test**: Download and load real CIFAR-10 images (50,000 training, 10,000 test)
|
||||
2. **Build DataLoader** → **Test**: Batch real images with proper shuffling and iteration
|
||||
3. **Build Normalizer** → **Test**: Normalize real pixel values (0-255 range → standardized)
|
||||
4. **Build Pipeline** → **Test**: Complete pipeline with real data flow and preprocessing
|
||||
|
||||
### Why Real Data Testing Matters
|
||||
|
||||
- **Real-world validation**: Tests work with actual data students will use in training
|
||||
- **Immediate feedback**: See your pipeline working with real images, not fake data
|
||||
- **Systems thinking**: Understand I/O, memory, and performance with real data distributions
|
||||
- **Debugging**: Catch issues that only appear with real data (file formats, edge cases)
|
||||
|
||||
**Note**: First test run downloads ~170MB CIFAR-10 dataset with progress bar. Subsequent runs use cached data.
|
||||
|
||||
### Interactive Testing with Visual Feedback
|
||||
```python
|
||||
# Test in the notebook or Python REPL
|
||||
from tinytorch.core.dataloader import Dataset, DataLoader, CIFAR10Dataset
|
||||
|
||||
# Create and test datasets with real data
|
||||
dataset = CIFAR10Dataset("data/cifar10/", train=True, download=True)
|
||||
print(f"Loaded {len(dataset)} real CIFAR-10 samples")
|
||||
|
||||
# Test data loading
|
||||
dataloader = DataLoader(dataset, batch_size=16)
|
||||
for batch_data, batch_labels in dataloader:
|
||||
print(f"Real batch shape: {batch_data.shape}") # (16, 3, 32, 32)
|
||||
print(f"Real labels: {batch_labels}") # Actual CIFAR-10 classes
|
||||
break
|
||||
```
|
||||
|
||||
### 🎨 Development Visual Feedback
|
||||
|
||||
The development notebook (`dataloader_dev.py`) includes **visual feedback** for learning:
|
||||
|
||||
```python
|
||||
# 👁️ SEE your data - Available in development notebook only
|
||||
show_cifar10_samples(dataset, num_samples=8, title="My CIFAR-10 Data")
|
||||
```
|
||||
|
||||
### 🎨 Visual Feedback Features (Development Only)
|
||||
|
||||
The development notebook includes **visual feedback** for learning and debugging:
|
||||
|
||||
- **Download progress bar**: Visual progress indicator during CIFAR-10 download (~170MB)
|
||||
- **`show_cifar10_samples()`**: Display a grid of CIFAR-10 images with class labels
|
||||
- **Real image visualization**: See actual airplanes, cars, birds, cats, etc.
|
||||
- **Batch visualization**: View what your DataLoader is producing
|
||||
- **Pipeline visualization**: See the complete data flow in action
|
||||
|
||||
**Why Visual Feedback Matters:**
|
||||
- **Build confidence**: See that your data pipeline is working correctly
|
||||
- **Debug issues**: Spot problems like incorrect normalization or corrupted images
|
||||
- **Understand data**: Build intuition about what your model will be learning from
|
||||
- **Immediate feedback**: Visual confirmation follows the "Build → Use → Understand" pattern
|
||||
|
||||
**Note**: Visual feedback is available in the development notebook (`data_dev.py`) for learning purposes. The core package exports only the essential data loading components.
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Your data module is complete when:
|
||||
|
||||
1. **All tests pass**: `tito test --module dataloader`
|
||||
2. **Data classes import correctly**: `from tinytorch.core.dataloader import Dataset, DataLoader`
|
||||
3. **Dataset loading works**: Can create datasets and access samples
|
||||
4. **Batching works**: DataLoader produces correct batch shapes
|
||||
5. **Preprocessing works**: Normalizer computes and applies statistics
|
||||
6. **Pipeline works**: Complete pipeline creates train/test loaders
|
||||
|
||||
## 💡 Implementation Tips
|
||||
|
||||
### Start with the Interface
|
||||
1. **Dataset base class** - Define the abstract interface
|
||||
2. **Simple test dataset** - Create mock data for testing
|
||||
3. **Basic DataLoader** - Implement batching without shuffling
|
||||
4. **Add shuffling** - Randomize sample order
|
||||
5. **Test frequently** - Verify each component works
|
||||
|
||||
### Design Patterns
|
||||
```python
|
||||
class Dataset:
|
||||
def __getitem__(self, index):
|
||||
# Return (data, label) tuple
|
||||
return data_tensor, label_tensor
|
||||
|
||||
def __len__(self):
|
||||
# Return total number of samples
|
||||
return self.num_samples
|
||||
|
||||
class DataLoader:
|
||||
def __iter__(self):
|
||||
# Yield batches of (batch_data, batch_labels)
|
||||
for batch in self._create_batches():
|
||||
yield batch_data, batch_labels
|
||||
```
|
||||
|
||||
### Systems Thinking
|
||||
- **Memory management**: Don't load entire dataset into RAM
|
||||
- **I/O efficiency**: Batch file operations when possible
|
||||
- **Preprocessing**: Compute statistics once, apply many times
|
||||
- **Interface design**: Make components easily swappable
|
||||
|
||||
### Common Challenges
|
||||
- **Binary file parsing** - CIFAR-10 uses custom format
|
||||
- **Batch size handling** - Last batch may be smaller
|
||||
- **Data type consistency** - Convert to consistent types
|
||||
- **Error handling** - Provide helpful debugging messages
|
||||
|
||||
## 🔧 Advanced Features (Optional)
|
||||
|
||||
If you finish early, try implementing:
|
||||
- **Data augmentation** - Random transformations for training
|
||||
- **Multi-worker loading** - Parallel data loading
|
||||
- **Caching** - Store processed data for faster access
|
||||
- **Different datasets** - MNIST, Fashion-MNIST, etc.
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once you complete the data module:
|
||||
|
||||
1. **Move to Autograd**: `cd modules/autograd/`
|
||||
2. **Build automatic differentiation**: Enable gradient computation
|
||||
3. **Combine with data**: Train models on real datasets
|
||||
4. **Prepare for training**: Ready for the training module
|
||||
|
||||
## 🔗 Why Data Engineering Matters
|
||||
|
||||
Data engineering is the foundation of all ML systems:
|
||||
- **Training loops** need efficient data loading
|
||||
- **Model performance** depends on data quality
|
||||
- **Production systems** require scalable data pipelines
|
||||
- **Research** needs flexible data interfaces
|
||||
|
||||
Your data implementation will power all TinyTorch training!
|
||||
|
||||
## 📊 Real-World Connection
|
||||
|
||||
The patterns you'll implement are used in:
|
||||
- **PyTorch DataLoader** - Same interface and concepts
|
||||
- **TensorFlow tf.data** - Similar pipeline architecture
|
||||
- **Production ML** - Scalable data processing systems
|
||||
- **Research** - Flexible experimentation frameworks
|
||||
|
||||
## 🎉 Ready to Build?
|
||||
|
||||
The data module is where TinyTorch becomes a real ML system. You're about to create the infrastructure that will feed neural networks, enable training loops, and power production ML pipelines.
|
||||
|
||||
Focus on clean interfaces, efficient implementation, and systems thinking! 🔥
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/06_cnn.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/08_autograd.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,321 @@
|
||||
# Autograd - Automatic Differentiation Engine
|
||||
---
|
||||
title: "Autograd"
|
||||
description: "Automatic differentiation engine for gradient computation"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Autograd module! This is where TinyTorch becomes truly powerful. You'll implement the automatic differentiation engine that makes neural network training possible.
|
||||
# 🧠 Module 7: Autograd - Automatic Differentiation Engine
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 8: 08 Autograd](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐ | <strong>Time:</strong> 6-8 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 6-8 hours
|
||||
- **Prerequisites**: Tensor, Activations, Layers modules
|
||||
- **Next Steps**: Training, Optimizers modules
|
||||
|
||||
**Build the automatic differentiation engine that makes neural network training possible**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand how automatic differentiation works through computational graphs
|
||||
- Implement the Variable class that tracks gradients and operations
|
||||
- Build backward propagation for gradient computation
|
||||
- Create the foundation for neural network training
|
||||
- Master the mathematical concepts behind backpropagation
|
||||
```
|
||||
- Create differentiable versions of all mathematical operations
|
||||
- Master the mathematical foundations of backpropagation
|
||||
|
||||
## 🧠 Build → Use → Analyze
|
||||
|
||||
This module follows the TinyTorch pedagogical framework:
|
||||
|
||||
## Build → Use → Analyze
|
||||
1. **Build**: Create the Variable class and gradient computation system
|
||||
2. **Use**: Perform automatic differentiation on complex expressions
|
||||
3. **Analyze**: Understand how gradients flow through computational graphs
|
||||
## 🚀 Interactive Learning
|
||||
3. **Analyze**: Understand how gradients flow through computational graphs and optimize performance
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
## 📚 What You'll Build
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/08_autograd/autograd_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### **Variable Class**
|
||||
```python
|
||||
# Gradient-tracking wrapper around Tensors
|
||||
x = Variable(5.0, requires_grad=True)
|
||||
y = Variable(3.0, requires_grad=True)
|
||||
z = x * y + x**2
|
||||
z.backward()
|
||||
print(x.grad) # Gradient of z with respect to x
|
||||
print(y.grad) # Gradient of z with respect to y
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/08_autograd/autograd_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### **Differentiable Operations**
|
||||
```python
|
||||
# All operations track gradients automatically
|
||||
def f(x, y):
|
||||
return x**2 + 2*x*y + y**2
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
x = Variable(2.0, requires_grad=True)
|
||||
y = Variable(3.0, requires_grad=True)
|
||||
result = f(x, y)
|
||||
result.backward()
|
||||
print(f"df/dx = {x.grad}") # Should be 2x + 2y = 10
|
||||
print(f"df/dy = {y.grad}") # Should be 2x + 2y = 10
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/08_autograd/autograd_dev.py
|
||||
:class-header: bg-light
|
||||
### **Neural Network Integration**
|
||||
```python
|
||||
# Works seamlessly with existing TinyTorch components
|
||||
from tinytorch.core.activations import ReLU
|
||||
from tinytorch.core.layers import Dense
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
# Create differentiable network
|
||||
x = Variable([[1.0, 2.0, 3.0]])
|
||||
layer = Dense(3, 2)
|
||||
relu = ReLU()
|
||||
|
||||
# Forward pass with gradient tracking
|
||||
output = relu(layer(x))
|
||||
loss = output.sum()
|
||||
loss.backward()
|
||||
|
||||
# Gradients available for all parameters
|
||||
print(layer.weights.grad) # Weight gradients
|
||||
print(layer.bias.grad) # Bias gradients
|
||||
```
|
||||
|
||||
````
|
||||
## 🚀 Getting Started
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
### Prerequisites
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
1. **Activate the virtual environment**:
|
||||
```bash
|
||||
source bin/activate-tinytorch.sh
|
||||
```
|
||||
|
||||
2. **Start development environment**:
|
||||
```bash
|
||||
tito jupyter
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
|
||||
1. **Open the development file**:
|
||||
```bash
|
||||
# Then open modules/source/07_autograd/autograd_dev.py
|
||||
```
|
||||
|
||||
2. **Implement the core components**:
|
||||
- Start with Variable class (gradient tracking)
|
||||
- Add basic operations (add, multiply, etc.)
|
||||
- Implement backward propagation
|
||||
- Add activation function gradients
|
||||
|
||||
3. **Test your implementation**:
|
||||
```bash
|
||||
tito test --module 07_autograd
|
||||
```
|
||||
|
||||
## 📊 Understanding Automatic Differentiation
|
||||
|
||||
### The Chain Rule in Action
|
||||
|
||||
Automatic differentiation is based on the chain rule:
|
||||
```
|
||||
If z = f(g(x)), then dz/dx = (dz/df) * (df/dx)
|
||||
```
|
||||
|
||||
### Computational Graph Example
|
||||
```
|
||||
Expression: f(x, y) = (x + y) * (x - y)
|
||||
|
||||
Forward Pass:
|
||||
x = 2, y = 3
|
||||
a = x + y = 5
|
||||
b = x - y = -1
|
||||
f = a * b = -5
|
||||
|
||||
Backward Pass:
|
||||
df/df = 1
|
||||
df/da = b = -1, df/db = a = 5
|
||||
da/dx = 1, da/dy = 1
|
||||
db/dx = 1, db/dy = -1
|
||||
df/dx = df/da * da/dx + df/db * db/dx = (-1)(1) + (5)(1) = 4
|
||||
df/dy = df/da * da/dy + df/db * db/dy = (-1)(1) + (5)(-1) = -6
|
||||
```
|
||||
|
||||
### Key Concepts
|
||||
|
||||
| Concept | Description | Example |
|
||||
|---------|-------------|---------|
|
||||
| **Variable** | Tensor wrapper with gradient tracking | `Variable(5.0, requires_grad=True)` |
|
||||
| **Computational Graph** | DAG representing operations | `z = x * y` creates graph |
|
||||
| **Forward Pass** | Computing function values | `z.data` contains result |
|
||||
| **Backward Pass** | Computing gradients | `z.backward()` fills gradients |
|
||||
| **Leaf Node** | Variable created by user | `x = Variable(5.0)` |
|
||||
| **Gradient Function** | How to compute gradients | `grad_fn` for each operation |
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Unit Tests
|
||||
```bash
|
||||
tito test --module 07_autograd
|
||||
```
|
||||
|
||||
**Test Coverage**:
|
||||
- ✅ Variable creation and properties
|
||||
- ✅ Basic arithmetic operations
|
||||
- ✅ Gradient computation correctness
|
||||
- ✅ Chain rule implementation
|
||||
- ✅ Integration with existing modules
|
||||
|
||||
### Manual Testing
|
||||
```python
|
||||
# Test basic gradients
|
||||
x = Variable(2.0, requires_grad=True)
|
||||
y = x**2 + 3*x + 1
|
||||
y.backward()
|
||||
print(x.grad) # Should be 2*2 + 3 = 7
|
||||
|
||||
# Test chain rule
|
||||
x = Variable(2.0, requires_grad=True)
|
||||
y = Variable(3.0, requires_grad=True)
|
||||
z = x * y
|
||||
w = z + x
|
||||
w.backward()
|
||||
print(x.grad) # Should be y + 1 = 4
|
||||
print(y.grad) # Should be x = 2
|
||||
```
|
||||
|
||||
## 📊 Mathematical Foundations
|
||||
|
||||
### Gradient Computation Rules
|
||||
|
||||
| Operation | Forward | Backward (Gradient) |
|
||||
|-----------|---------|-------------------|
|
||||
| Addition | `z = x + y` | `dx = dz, dy = dz` |
|
||||
| Multiplication | `z = x * y` | `dx = y * dz, dy = x * dz` |
|
||||
| Power | `z = x^n` | `dx = n * x^(n-1) * dz` |
|
||||
| Exp | `z = exp(x)` | `dx = exp(x) * dz` |
|
||||
| Log | `z = log(x)` | `dx = (1/x) * dz` |
|
||||
| ReLU | `z = max(0, x)` | `dx = (x > 0) * dz` |
|
||||
| Sigmoid | `z = 1/(1+exp(-x))` | `dx = z * (1-z) * dz` |
|
||||
|
||||
### Advanced Concepts
|
||||
- **Higher-order gradients**: Gradients of gradients
|
||||
- **Jacobian matrices**: Gradients for vector functions
|
||||
- **Hessian matrices**: Second-order derivatives
|
||||
- **Gradient checkpointing**: Memory optimization
|
||||
|
||||
## 🔧 Integration with TinyTorch
|
||||
|
||||
After implementation, your autograd system will enable:
|
||||
|
||||
```python
|
||||
from tinytorch.core.autograd import Variable
|
||||
from tinytorch.core.layers import Dense
|
||||
from tinytorch.core.activations import ReLU
|
||||
|
||||
# Create a simple neural network
|
||||
x = Variable([[1.0, 2.0, 3.0]])
|
||||
layer1 = Dense(3, 4)
|
||||
layer2 = Dense(4, 1)
|
||||
relu = ReLU()
|
||||
|
||||
# Forward pass
|
||||
h = relu(layer1(x))
|
||||
output = layer2(h)
|
||||
loss = output.sum()
|
||||
|
||||
# Backward pass
|
||||
loss.backward()
|
||||
|
||||
# All gradients computed automatically!
|
||||
print(layer1.weights.grad)
|
||||
print(layer2.weights.grad)
|
||||
```
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Your autograd module is complete when:
|
||||
|
||||
1. **All tests pass**: `tito test --module 07_autograd`
|
||||
2. **Variable imports correctly**: `from tinytorch.core.autograd import Variable`
|
||||
3. **Basic operations work**: Can create Variables and do arithmetic
|
||||
4. **Gradients compute correctly**: Backward pass produces correct gradients
|
||||
5. **Integration works**: Seamlessly works with existing TinyTorch modules
|
||||
|
||||
## 💡 Implementation Tips
|
||||
|
||||
### Start with the Basics
|
||||
1. **Variable class** - Wrap Tensors with gradient tracking
|
||||
2. **Simple operations** - Start with addition and multiplication
|
||||
3. **Backward method** - Implement gradient computation
|
||||
4. **Test frequently** - Verify gradients match analytical solutions
|
||||
|
||||
### Design Patterns
|
||||
```python
|
||||
class Variable:
|
||||
def __init__(self, data, requires_grad=True, grad_fn=None):
|
||||
# Store data, gradient state, and computation history
|
||||
|
||||
def backward(self, gradient=None):
|
||||
# Implement backpropagation using chain rule
|
||||
|
||||
def add(a, b):
|
||||
# Create new Variable with grad_fn that knows how to backprop
|
||||
def backward_fn(grad):
|
||||
# Distribute gradient to inputs
|
||||
return Variable(result, grad_fn=backward_fn)
|
||||
```
|
||||
|
||||
### Common Challenges
|
||||
- **Gradient accumulation** - Handle multiple paths to same Variable
|
||||
- **Memory management** - Store intermediate values efficiently
|
||||
- **Numerical stability** - Handle edge cases in gradient computation
|
||||
- **Graph construction** - Build computation graph correctly
|
||||
|
||||
## 🔧 Advanced Features (Optional)
|
||||
|
||||
If you finish early, try implementing:
|
||||
- **Higher-order gradients** - Gradients of gradients
|
||||
- **Gradient checkpointing** - Memory optimization
|
||||
- **Custom operations** - Define your own differentiable functions
|
||||
- **Gradient clipping** - Prevent exploding gradients
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once you complete the autograd module:
|
||||
|
||||
1. **Move to Training**: `cd modules/source/08_training/`
|
||||
2. **Build optimization algorithms**: Implement SGD, Adam, etc.
|
||||
3. **Create training loops**: Put it all together
|
||||
4. **Train real models**: Use your autograd system for actual ML!
|
||||
|
||||
## 🔗 Why Autograd Matters
|
||||
|
||||
Automatic differentiation is the foundation of modern ML:
|
||||
- **Neural networks** require gradients for backpropagation
|
||||
- **Optimization** needs gradients for parameter updates
|
||||
- **Research** benefits from easy gradient computation
|
||||
- **Production** systems rely on efficient autodiff
|
||||
|
||||
This module transforms TinyTorch from a static computation library into a dynamic, trainable ML framework!
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/07_dataloader.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/09_optimizers.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,236 @@
|
||||
# Optimizers - Gradient-Based Parameter Updates
|
||||
---
|
||||
title: "Optimizers"
|
||||
description: "Gradient-based parameter optimization algorithms"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Optimizers module! This is where neural networks learn to improve through intelligent parameter updates.
|
||||
# 🚀 Module 8: Optimizers - Gradient-Based Parameter Updates
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 9: 09 Optimizers](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐ | <strong>Time:</strong> 5-6 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐ Expert
|
||||
- **Time Estimate**: 6-8 hours
|
||||
- **Prerequisites**: Tensor, Autograd modules
|
||||
- **Next Steps**: Training, MLOps modules
|
||||
|
||||
**Build intelligent optimization algorithms that enable effective neural network training**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand gradient descent and how optimizers use gradients to update parameters
|
||||
- Implement SGD with momentum for accelerated convergence
|
||||
- Build Adam optimizer with adaptive learning rates
|
||||
- Master learning rate scheduling strategies
|
||||
- See how optimizers enable effective neural network training
|
||||
```
|
||||
- Build Adam optimizer with adaptive learning rates for modern deep learning
|
||||
- Master learning rate scheduling strategies for training stability
|
||||
- See how optimizers enable complete neural network training workflows
|
||||
|
||||
## 🧠 Build → Use → Analyze
|
||||
|
||||
## Build → Use → Analyze
|
||||
1. **Build**: Core optimization algorithms (SGD, Adam)
|
||||
2. **Use**: Apply optimizers to train neural networks
|
||||
This module follows the TinyTorch pedagogical framework:
|
||||
|
||||
1. **Build**: Core optimization algorithms (SGD, Adam, scheduling)
|
||||
2. **Use**: Apply optimizers to train neural networks effectively
|
||||
3. **Analyze**: Compare optimizer behavior and convergence patterns
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
## 📚 What You'll Build
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/09_optimizers/optimizers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### **Gradient Descent Foundation**
|
||||
```python
|
||||
# Basic gradient descent step
|
||||
def gradient_descent_step(parameter, learning_rate):
|
||||
parameter.data = parameter.data - learning_rate * parameter.grad.data
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/09_optimizers/optimizers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
### **SGD with Momentum**
|
||||
```python
|
||||
# Accelerated convergence
|
||||
sgd = SGD([w1, w2, bias], learning_rate=0.01, momentum=0.9)
|
||||
sgd.zero_grad()
|
||||
loss.backward()
|
||||
sgd.step()
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/09_optimizers/optimizers_dev.py
|
||||
:class-header: bg-light
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
### **Adam Optimizer**
|
||||
```python
|
||||
# Adaptive learning rates
|
||||
adam = Adam([w1, w2, bias], learning_rate=0.001, beta1=0.9, beta2=0.999)
|
||||
adam.zero_grad()
|
||||
loss.backward()
|
||||
adam.step()
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
### **Learning Rate Scheduling**
|
||||
```python
|
||||
# Strategic learning rate adjustment
|
||||
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
|
||||
scheduler.step() # Reduce learning rate every 10 epochs
|
||||
```
|
||||
|
||||
### **Complete Training Integration**
|
||||
```python
|
||||
# Modern training loop
|
||||
optimizer = Adam(model.parameters(), learning_rate=0.001)
|
||||
scheduler = StepLR(optimizer, step_size=20, gamma=0.5)
|
||||
|
||||
for epoch in range(num_epochs):
|
||||
for batch in dataloader:
|
||||
optimizer.zero_grad()
|
||||
loss = criterion(model(batch.inputs), batch.targets)
|
||||
loss.backward()
|
||||
optimizer.step()
|
||||
scheduler.step()
|
||||
```
|
||||
|
||||
## 🔬 Core Concepts
|
||||
|
||||
### **Gradient Descent Theory**
|
||||
- **Mathematical foundation**: θ = θ - α∇L(θ)
|
||||
- **Learning rate**: Balance between convergence speed and stability
|
||||
- **Convergence**: How optimizers reach optimal parameters
|
||||
|
||||
### **Momentum Acceleration**
|
||||
- **Velocity accumulation**: v_t = βv_{t-1} + ∇L(θ)
|
||||
- **Oscillation dampening**: Smooth progress in consistent directions
|
||||
- **Acceleration**: Build up speed toward minimum
|
||||
|
||||
### **Adaptive Learning Rates**
|
||||
- **First moment**: Exponential moving average of gradients
|
||||
- **Second moment**: Exponential moving average of squared gradients
|
||||
- **Bias correction**: Handle initialization bias in moment estimates
|
||||
|
||||
### **Learning Rate Scheduling**
|
||||
- **Step decay**: Reduce learning rate at fixed intervals
|
||||
- **Convergence strategy**: Start fast, then refine with smaller steps
|
||||
- **Training stability**: Prevent overshooting near optimum
|
||||
|
||||
## 🎮 What You'll Experience
|
||||
|
||||
### **Immediate Feedback**
|
||||
- **Test each optimizer**: See parameter updates in real-time
|
||||
- **Compare convergence**: SGD vs Adam on same problem
|
||||
- **Visualize learning**: Watch parameters converge to optimal values
|
||||
|
||||
### **Real Training Workflow**
|
||||
- **Complete training loop**: From gradients to parameter updates
|
||||
- **Learning rate scheduling**: Strategic adjustment during training
|
||||
- **Modern best practices**: Industry-standard optimization patterns
|
||||
|
||||
### **Mathematical Insights**
|
||||
- **Gradient interpretation**: How gradients guide parameter updates
|
||||
- **Momentum physics**: Velocity and acceleration in optimization
|
||||
- **Adaptive scaling**: Different learning rates for different parameters
|
||||
|
||||
## 🔧 Technical Implementation
|
||||
|
||||
### **State Management**
|
||||
- **Momentum buffers**: Track velocity for each parameter
|
||||
- **Moment estimates**: First and second moments for Adam
|
||||
- **Step counting**: Track iterations for bias correction
|
||||
|
||||
### **Numerical Stability**
|
||||
- **Epsilon handling**: Prevent division by zero
|
||||
- **Overflow protection**: Handle large gradients gracefully
|
||||
- **Precision**: Balance between float32 and numerical accuracy
|
||||
|
||||
### **Memory Efficiency**
|
||||
- **Lazy initialization**: Create buffers only when needed
|
||||
- **Parameter tracking**: Use object IDs for state management
|
||||
- **Gradient management**: Proper gradient zeroing and accumulation
|
||||
|
||||
## 📈 Performance Characteristics
|
||||
|
||||
### **SGD with Momentum**
|
||||
- **Memory**: O(P) for momentum buffers (P = number of parameters)
|
||||
- **Computation**: O(P) per step
|
||||
- **Convergence**: Linear in convex case, good for large batch training
|
||||
|
||||
### **Adam Optimizer**
|
||||
- **Memory**: O(2P) for first and second moment buffers
|
||||
- **Computation**: O(P) per step with additional operations
|
||||
- **Convergence**: Fast initial progress, good for most deep learning
|
||||
|
||||
### **Learning Rate Scheduling**
|
||||
- **Overhead**: Minimal computational cost
|
||||
- **Impact**: Significant improvement in final performance
|
||||
- **Flexibility**: Adaptable to different training scenarios
|
||||
|
||||
## 🔗 Integration with TinyTorch
|
||||
|
||||
### **Dependencies**
|
||||
- **Tensor**: Core data structure for parameters
|
||||
- **Autograd**: Gradient computation for parameter updates
|
||||
- **Variables**: Parameter containers with gradient tracking
|
||||
|
||||
### **Enables**
|
||||
- **Training Module**: Complete training loops with loss functions
|
||||
- **Advanced Training**: Distributed training, mixed precision
|
||||
- **Research**: Novel optimization algorithms and strategies
|
||||
|
||||
## 🎯 Real-World Applications
|
||||
|
||||
### **Computer Vision**
|
||||
- **ImageNet training**: ResNet, VGG, Vision Transformers
|
||||
- **Object detection**: YOLO, R-CNN optimization
|
||||
- **Segmentation**: U-Net, Mask R-CNN training
|
||||
|
||||
### **Natural Language Processing**
|
||||
- **Language models**: GPT, BERT, T5 training
|
||||
- **Machine translation**: Transformer optimization
|
||||
- **Text generation**: Large language model training
|
||||
|
||||
### **Scientific Computing**
|
||||
- **Physics simulations**: Neural ODE optimization
|
||||
- **Reinforcement learning**: Policy gradient methods
|
||||
- **Generative models**: GAN, VAE training
|
||||
|
||||
## 🚀 What's Next
|
||||
|
||||
After mastering optimizers, you'll be ready for:
|
||||
|
||||
1. **Training Module**: Complete training loops with loss functions and metrics
|
||||
2. **Advanced Optimizers**: RMSprop, AdaGrad, learning rate warm-up
|
||||
3. **Distributed Training**: Multi-GPU optimization strategies
|
||||
4. **MLOps**: Production optimization monitoring and tuning
|
||||
|
||||
## 💡 Key Insights
|
||||
|
||||
### **Optimization is Critical**
|
||||
- **Make or break**: Good optimizer choice determines training success
|
||||
- **Hyperparameter sensitivity**: Learning rate is the most important hyperparameter
|
||||
- **Architecture dependent**: Different models prefer different optimizers
|
||||
|
||||
### **Modern Defaults**
|
||||
- **Adam**: Default choice for most deep learning applications
|
||||
- **SGD with momentum**: Still preferred for some computer vision tasks
|
||||
- **Learning rate scheduling**: Almost always improves final performance
|
||||
|
||||
### **Systems Thinking**
|
||||
- **Memory trade-offs**: Adam uses more memory but often trains faster
|
||||
- **Convergence patterns**: Understanding when and why optimizers work
|
||||
- **Debugging**: Optimizer issues are common in training failures
|
||||
|
||||
**Ready to build the intelligent algorithms that power modern AI training?**
|
||||
|
||||
Your optimizers will be the engine that transforms gradients into intelligence!
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/08_autograd.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/10_training.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,309 @@
|
||||
# Training - Complete Neural Network Training Pipeline
|
||||
---
|
||||
title: "Training"
|
||||
description: "Neural network training loops, loss functions, and metrics"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Training module! This is where we bring everything together to train neural networks on real data.
|
||||
# 🏋️ Module 9: Training - Complete Neural Network Training Pipeline
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 10: 10 Training](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand loss functions and how they measure model performance
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐ | <strong>Time:</strong> 6-8 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐⭐ Expert
|
||||
- **Time Estimate**: 8-10 hours
|
||||
- **Prerequisites**: Tensor, Activations, Layers, Networks, DataLoader, Autograd, Optimizers modules
|
||||
- **Next Steps**: Compression, Kernels, Benchmarking, MLOps modules
|
||||
|
||||
**Build the complete training pipeline that brings all TinyTorch components together**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand loss functions and how they guide neural network training
|
||||
- Implement essential loss functions: MSE, CrossEntropy, and BinaryCrossEntropy
|
||||
- Build evaluation metrics for classification and regression
|
||||
- Create a complete training loop that orchestrates the entire process
|
||||
- Master checkpointing and model persistence for real-world deployment
|
||||
```
|
||||
- Build evaluation metrics for classification and regression tasks
|
||||
- Create a complete training loop that orchestrates the entire training process
|
||||
- Master training workflows with validation, logging, and progress tracking
|
||||
|
||||
## 🧠 Build → Use → Optimize
|
||||
|
||||
## Build → Use → Optimize
|
||||
1. **Build**: Loss functions, metrics, and training orchestration
|
||||
2. **Use**: Train complete models on real datasets
|
||||
This module follows the TinyTorch pedagogical framework:
|
||||
|
||||
1. **Build**: Loss functions, metrics, and training orchestration components
|
||||
2. **Use**: Train complete neural networks on real datasets
|
||||
3. **Optimize**: Analyze training dynamics and improve performance
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
## 📚 What You'll Build
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
### **Loss Functions**
|
||||
```python
|
||||
# Regression loss
|
||||
mse = MeanSquaredError()
|
||||
loss = mse(predictions, targets)
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/10_training/training_dev.ipynb
|
||||
:class-header: bg-light
|
||||
# Multi-class classification loss
|
||||
ce = CrossEntropyLoss()
|
||||
loss = ce(logits, class_indices)
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
# Binary classification loss
|
||||
bce = BinaryCrossEntropyLoss()
|
||||
loss = bce(logits, binary_labels)
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/10_training/training_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### **Evaluation Metrics**
|
||||
```python
|
||||
# Classification accuracy
|
||||
accuracy = Accuracy()
|
||||
acc = accuracy(predictions, true_labels) # Returns 0.0 to 1.0
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Regression metrics
|
||||
mae = MeanAbsoluteError()
|
||||
error = mae(predictions, targets)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/10_training/training_dev.py
|
||||
:class-header: bg-light
|
||||
### **Complete Training Pipeline**
|
||||
```python
|
||||
# Set up training components
|
||||
model = Sequential([
|
||||
Dense(784, 128), ReLU(),
|
||||
Dense(128, 64), ReLU(),
|
||||
Dense(64, 10), Softmax()
|
||||
])
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
optimizer = Adam(model.parameters, learning_rate=0.001)
|
||||
loss_fn = CrossEntropyLoss()
|
||||
metrics = [Accuracy()]
|
||||
|
||||
# Create trainer
|
||||
trainer = Trainer(model, optimizer, loss_fn, metrics)
|
||||
|
||||
# Train the model
|
||||
history = trainer.fit(
|
||||
train_dataloader,
|
||||
val_dataloader,
|
||||
epochs=10,
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
````
|
||||
### **Training with Real Data**
|
||||
```python
|
||||
# Load dataset
|
||||
from tinytorch.core.dataloader import SimpleDataset, DataLoader
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
# Create dataset
|
||||
train_dataset = SimpleDataset(size=1000, num_features=784, num_classes=10)
|
||||
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
# Train on real data
|
||||
history = trainer.fit(train_loader, epochs=50)
|
||||
|
||||
# Analyze training
|
||||
print(f"Final training loss: {history['train_loss'][-1]:.4f}")
|
||||
print(f"Final training accuracy: {history['train_accuracy'][-1]:.4f}")
|
||||
```
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- Complete Modules 1-8: Setup through Optimizers ✅
|
||||
- Understand backpropagation and gradient descent
|
||||
- Familiar with classification and regression tasks
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
# Navigate to the training module
|
||||
cd modules/source/09_training
|
||||
|
||||
# Open the development notebook
|
||||
jupyter lab training_dev.py
|
||||
|
||||
# Or use the TinyTorch CLI
|
||||
tito module info training
|
||||
tito module test training
|
||||
```
|
||||
|
||||
## 📖 Core Concepts
|
||||
|
||||
### **Loss Functions: The Training Signal**
|
||||
Loss functions measure how far our predictions are from the true values:
|
||||
|
||||
- **MSE**: For regression tasks, penalizes large errors heavily
|
||||
- **CrossEntropy**: For classification, works with softmax outputs
|
||||
- **BinaryCrossEntropy**: For binary classification, works with sigmoid outputs
|
||||
|
||||
### **Metrics: Human-Interpretable Performance**
|
||||
Metrics provide understandable measures of model performance:
|
||||
|
||||
- **Accuracy**: Fraction of correct predictions
|
||||
- **Precision**: Of positive predictions, how many were correct?
|
||||
- **Recall**: Of actual positives, how many were found?
|
||||
|
||||
### **Training Loop: Orchestrating Learning**
|
||||
The training loop coordinates all components:
|
||||
|
||||
1. **Forward Pass**: Model makes predictions
|
||||
2. **Loss Computation**: Measure prediction quality
|
||||
3. **Backward Pass**: Compute gradients
|
||||
4. **Parameter Update**: Improve model weights
|
||||
5. **Validation**: Monitor generalization performance
|
||||
|
||||
### **Training Dynamics**
|
||||
Understanding how training behaves:
|
||||
|
||||
- **Overfitting**: Model memorizes training data
|
||||
- **Underfitting**: Model too simple to learn patterns
|
||||
- **Convergence**: Loss stops decreasing
|
||||
- **Validation**: Monitoring generalization
|
||||
|
||||
## 🔬 Advanced Features
|
||||
|
||||
### **Training Monitoring**
|
||||
```python
|
||||
# Track training progress
|
||||
history = trainer.fit(train_loader, val_loader, epochs=100)
|
||||
|
||||
# Plot training curves
|
||||
import matplotlib.pyplot as plt
|
||||
plt.plot(history['train_loss'], label='Training Loss')
|
||||
plt.plot(history['val_loss'], label='Validation Loss')
|
||||
plt.legend()
|
||||
plt.show()
|
||||
```
|
||||
|
||||
### **Custom Metrics**
|
||||
```python
|
||||
# Create custom metrics
|
||||
class F1Score:
|
||||
def __call__(self, y_pred, y_true):
|
||||
# Implement F1 score calculation
|
||||
pass
|
||||
|
||||
# Use in training
|
||||
trainer = Trainer(model, optimizer, loss_fn, metrics=[Accuracy(), F1Score()])
|
||||
```
|
||||
|
||||
### **Training Strategies**
|
||||
```python
|
||||
# Learning rate scheduling
|
||||
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
|
||||
|
||||
# Early stopping
|
||||
class EarlyStopping:
|
||||
def __init__(self, patience=10):
|
||||
self.patience = patience
|
||||
self.best_loss = float('inf')
|
||||
self.counter = 0
|
||||
|
||||
def __call__(self, val_loss):
|
||||
if val_loss < self.best_loss:
|
||||
self.best_loss = val_loss
|
||||
self.counter = 0
|
||||
else:
|
||||
self.counter += 1
|
||||
return self.counter >= self.patience
|
||||
```
|
||||
|
||||
## 🛠️ Real-World Applications
|
||||
|
||||
### **Computer Vision**
|
||||
```python
|
||||
# Image classification pipeline
|
||||
model = Sequential([
|
||||
Conv2D((3, 3)), ReLU(),
|
||||
flatten,
|
||||
Dense(128), ReLU(),
|
||||
Dense(10), Softmax()
|
||||
])
|
||||
|
||||
trainer = Trainer(model, Adam(model.parameters), CrossEntropyLoss(), [Accuracy()])
|
||||
history = trainer.fit(cifar10_loader, epochs=50)
|
||||
```
|
||||
|
||||
### **Natural Language Processing**
|
||||
```python
|
||||
# Text classification
|
||||
model = Sequential([
|
||||
Dense(vocab_size, 128), ReLU(),
|
||||
Dense(128, 64), ReLU(),
|
||||
Dense(64, num_classes), Softmax()
|
||||
])
|
||||
|
||||
trainer = Trainer(model, SGD(model.parameters), CrossEntropyLoss(), [Accuracy()])
|
||||
history = trainer.fit(text_loader, epochs=20)
|
||||
```
|
||||
|
||||
### **Regression Tasks**
|
||||
```python
|
||||
# House price prediction
|
||||
model = Sequential([
|
||||
Dense(features, 64), ReLU(),
|
||||
Dense(64, 32), ReLU(),
|
||||
Dense(32, 1) # Single output for regression
|
||||
])
|
||||
|
||||
trainer = Trainer(model, Adam(model.parameters), MeanSquaredError(), [])
|
||||
history = trainer.fit(housing_loader, epochs=100)
|
||||
```
|
||||
|
||||
## 📈 Performance Optimization
|
||||
|
||||
### **Batch Size Selection**
|
||||
- **Small batches**: More updates, noisier gradients
|
||||
- **Large batches**: Fewer updates, smoother gradients
|
||||
- **Sweet spot**: Usually 32-256 depending on dataset
|
||||
|
||||
### **Learning Rate Tuning**
|
||||
- **Too high**: Training diverges or oscillates
|
||||
- **Too low**: Training is slow or gets stuck
|
||||
- **Adaptive methods**: Adam often works well out of the box
|
||||
|
||||
### **Regularization**
|
||||
- **Dropout**: Randomly disable neurons during training
|
||||
- **Weight decay**: L2 regularization on parameters
|
||||
- **Early stopping**: Stop when validation performance plateaus
|
||||
|
||||
## 🎯 Module Completion
|
||||
|
||||
### **What You've Built**
|
||||
✅ **Complete loss function library**: MSE, CrossEntropy, BinaryCrossEntropy
|
||||
✅ **Evaluation metrics**: Accuracy and extensible metric framework
|
||||
✅ **Training orchestration**: Full-featured Trainer class
|
||||
✅ **Real-world pipeline**: Train models on actual datasets
|
||||
✅ **Monitoring tools**: Track training progress and performance
|
||||
|
||||
### **Skills Developed**
|
||||
✅ **Training loop design**: Coordinate all training components
|
||||
✅ **Loss function implementation**: Measure prediction quality
|
||||
✅ **Metric computation**: Evaluate model performance
|
||||
✅ **Training dynamics**: Understand convergence and optimization
|
||||
✅ **Production workflows**: Build scalable training pipelines
|
||||
|
||||
### **Next Steps**
|
||||
1. **Export your training module**: `tito export training`
|
||||
2. **Train a complete model**: Use all TinyTorch components together
|
||||
3. **Explore advanced topics**: Regularization, scheduling, ensembles
|
||||
4. **Build production pipelines**: Scale training to larger datasets
|
||||
|
||||
**Ready for the final stretch?** Your training module completes the core TinyTorch framework. Next up: compression, kernels, and MLOps! 🚀
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/09_optimizers.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/11_compression.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,55 +1,207 @@
|
||||
# Compression & Optimization - Making AI Models Efficient
|
||||
---
|
||||
title: "Compression & Optimization"
|
||||
description: "Making AI models efficient for real-world deployment"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: ['Module 11: Kernels - Hardware-aware optimization', 'Module 12: Benchmarking - Performance measurement', 'Module 13: MLOps - Production deployment']
|
||||
learning_objectives: ['Understand model size and deployment constraints in real systems', 'Implement magnitude-based pruning to remove unimportant weights', 'Master quantization for 75% memory reduction (FP32 → INT8)', 'Build knowledge distillation for training compact models', 'Create structured pruning to optimize network architectures', 'Compare compression techniques and their trade-offs']
|
||||
---
|
||||
|
||||
Welcome to the Compression module! This is where you'll learn to make neural networks smaller, faster, and more efficient for real-world deployment.
|
||||
# Module 10: Compression & Optimization
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 11: 11 Compression](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand how model size affects deployment and why compression matters
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐⭐ | <strong>Time:</strong> 4-5 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## Overview
|
||||
This module teaches students to make neural networks smaller, faster, and more efficient for real-world deployment. Students implement four core compression techniques and learn to balance accuracy with efficiency.
|
||||
|
||||
## Learning Goals
|
||||
- Understand model size and deployment constraints in real systems
|
||||
- Implement magnitude-based pruning to remove unimportant weights
|
||||
- Master quantization to reduce memory usage by 75%
|
||||
- Master quantization for 75% memory reduction (FP32 → INT8)
|
||||
- Build knowledge distillation for training compact models
|
||||
- Create structured pruning to optimize network architectures
|
||||
- Compare compression techniques and their trade-offs
|
||||
|
||||
## Educational Flow
|
||||
|
||||
### Step 1: Understanding Model Size
|
||||
- **Concept**: Parameter counting and memory footprint analysis
|
||||
- **Implementation**: `CompressionMetrics` class for model analysis
|
||||
- **Learning**: Foundation for compression decision-making
|
||||
|
||||
### Step 2: Magnitude-Based Pruning
|
||||
- **Concept**: Remove weights with smallest absolute values
|
||||
- **Implementation**: `prune_weights_by_magnitude()` and sparsity calculation
|
||||
- **Learning**: Sparsity patterns and accuracy vs compression trade-offs
|
||||
|
||||
### Step 3: Quantization Experiments
|
||||
- **Concept**: Reduce precision from FP32 to INT8 for memory efficiency
|
||||
- **Implementation**: `quantize_layer_weights()` with scale/offset mapping
|
||||
- **Learning**: Numerical precision impact on model performance
|
||||
|
||||
### Step 4: Knowledge Distillation
|
||||
- **Concept**: Large models teach small models through soft targets
|
||||
- **Implementation**: `DistillationLoss` with temperature scaling
|
||||
- **Learning**: Advanced training techniques for compact models
|
||||
|
||||
### Step 5: Structured Pruning
|
||||
- **Concept**: Remove entire neurons/channels, not just weights
|
||||
- **Implementation**: `prune_layer_neurons()` with importance scoring
|
||||
- **Learning**: Architecture optimization and cascade effects
|
||||
|
||||
### Step 6: Comprehensive Comparison
|
||||
- **Concept**: Combine techniques for maximum efficiency
|
||||
- **Implementation**: Integrated compression pipeline
|
||||
- **Learning**: Systems thinking for production deployment
|
||||
|
||||
## Key Components
|
||||
|
||||
### CompressionMetrics
|
||||
- **Purpose**: Analyze model size and parameter distribution
|
||||
- **Methods**: `count_parameters()`, `calculate_model_size()`, `analyze_weight_distribution()`
|
||||
- **Usage**: Foundation for compression target selection
|
||||
|
||||
### Pruning Functions
|
||||
- **Purpose**: Remove unimportant weights and neurons
|
||||
- **Methods**: `prune_weights_by_magnitude()`, `prune_model_by_magnitude()`, `calculate_sparsity()`
|
||||
- **Usage**: Reduce model size while maintaining performance
|
||||
|
||||
### Quantization Functions
|
||||
- **Purpose**: Reduce memory usage through lower precision
|
||||
- **Methods**: `quantize_layer_weights()`, `dequantize_layer_weights()`
|
||||
- **Usage**: 75% memory reduction for mobile deployment
|
||||
|
||||
### Knowledge Distillation
|
||||
- **Purpose**: Train compact models with teacher guidance
|
||||
- **Methods**: `DistillationLoss`, `train_with_distillation()`
|
||||
- **Usage**: Achieve better small model performance
|
||||
|
||||
### Structured Pruning
|
||||
- **Purpose**: Remove entire neurons for actual speedup
|
||||
- **Methods**: `prune_layer_neurons()`, `compute_neuron_importance()`
|
||||
- **Usage**: Architecture optimization and hardware efficiency
|
||||
|
||||
## Real-World Applications
|
||||
|
||||
### Mobile AI Deployment
|
||||
- **Constraint**: Models must be < 10MB for smartphone apps
|
||||
- **Solution**: Combine pruning and quantization for 90% size reduction
|
||||
- **Examples**: Google Translate offline, mobile camera AI
|
||||
|
||||
### Edge Computing
|
||||
- **Constraint**: Severe memory and compute limitations
|
||||
- **Solution**: Structured pruning for actual inference speedup
|
||||
- **Examples**: IoT sensors, smart cameras, voice assistants
|
||||
|
||||
### Cost Optimization
|
||||
- **Constraint**: Expensive cloud inference at scale
|
||||
- **Solution**: Reduce model size for lower compute costs
|
||||
- **Examples**: Production recommendation systems, search engines
|
||||
|
||||
### Battery Efficiency
|
||||
- **Constraint**: Wearable devices need long battery life
|
||||
- **Solution**: Quantization and pruning for energy savings
|
||||
- **Examples**: Smartwatches, fitness trackers, AR glasses
|
||||
|
||||
## Industry Connections
|
||||
|
||||
### MobileNet Architecture
|
||||
- **Concept**: Depthwise separable convolutions for efficiency
|
||||
- **Connection**: Structured optimization for mobile deployment
|
||||
- **Learning**: Architecture design affects compression potential
|
||||
|
||||
### DistilBERT
|
||||
- **Concept**: 60% smaller than BERT with 97% performance
|
||||
- **Connection**: Knowledge distillation for language models
|
||||
- **Learning**: Teacher-student training for different domains
|
||||
|
||||
### TinyML Movement
|
||||
- **Concept**: ML on microcontrollers (< 1MB models)
|
||||
- **Connection**: Extreme compression for embedded systems
|
||||
- **Learning**: Efficiency requirements for edge deployment
|
||||
|
||||
### Neural Architecture Search
|
||||
- **Concept**: Automated model design for efficiency
|
||||
- **Connection**: Structured pruning as architecture optimization
|
||||
- **Learning**: Automated techniques for compression
|
||||
|
||||
## Assessment Criteria
|
||||
|
||||
### Technical Implementation (40%)
|
||||
- Correctly implement 4 compression techniques
|
||||
- Handle edge cases and error conditions
|
||||
- Provide comprehensive statistics and analysis
|
||||
|
||||
### Understanding Trade-offs (30%)
|
||||
- Explain accuracy vs efficiency spectrum
|
||||
- Identify appropriate techniques for different constraints
|
||||
- Analyze compression effectiveness quantitatively
|
||||
|
||||
### Real-World Application (30%)
|
||||
- Connect compression to deployment scenarios
|
||||
- Understand hardware and system constraints
|
||||
- Design compression strategies for specific use cases
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Module 11: Kernels
|
||||
- **Connection**: Hardware-aware optimization builds on compression
|
||||
- **Skills**: GPU kernels, SIMD operations, memory optimization
|
||||
- **Application**: Implement efficient compressed model inference
|
||||
|
||||
### Module 12: Benchmarking
|
||||
- **Connection**: Measure compression effectiveness systematically
|
||||
- **Skills**: Performance profiling, accuracy measurement, A/B testing
|
||||
- **Application**: Evaluate compression trade-offs in production
|
||||
|
||||
### Module 13: MLOps
|
||||
- **Connection**: Deploy compressed models in production systems
|
||||
- **Skills**: Model versioning, monitoring, continuous optimization
|
||||
- **Application**: Production-ready compressed model deployment
|
||||
|
||||
## File Structure
|
||||
```
|
||||
10_compression/
|
||||
├── compression_dev.py # Main development notebook
|
||||
├── module.yaml # Module configuration
|
||||
├── README.md # This file
|
||||
└── tests/ # Additional test files (if needed)
|
||||
```
|
||||
|
||||
## Getting Started
|
||||
|
||||
## Build → Use → Optimize
|
||||
1. **Build**: Four compression techniques from scratch
|
||||
2. **Use**: Apply compression to real neural networks
|
||||
3. **Optimize**: Combine techniques for maximum efficiency gains
|
||||
## 🚀 Interactive Learning
|
||||
1. **Review Dependencies**: Ensure modules 01, 02, 04, 05, 10 are complete
|
||||
2. **Open Development File**: `compression_dev.py`
|
||||
3. **Follow Educational Flow**: Work through Steps 1-6 sequentially
|
||||
4. **Test Thoroughly**: Run all inline tests as you progress
|
||||
5. **Export to Package**: Use `tito export 10_compression` when complete
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
## Key Takeaways
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
Students completing this module will:
|
||||
- **Understand** the efficiency requirements of production AI systems
|
||||
- **Implement** four essential compression techniques from scratch
|
||||
- **Analyze** accuracy vs efficiency trade-offs quantitatively
|
||||
- **Apply** compression strategies to real neural networks
|
||||
- **Connect** compression to mobile, edge, and production deployment
|
||||
- **Prepare** for advanced optimization and production deployment modules
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/11_compression/compression_dev.ipynb
|
||||
:class-header: bg-light
|
||||
This module bridges the gap between research-quality models and production-ready AI systems, teaching the essential skills for deploying AI in resource-constrained environments.
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/11_compression/compression_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/11_compression/compression_dev.py
|
||||
:class-header: bg-light
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/10_training.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/12_kernels.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,55 +1,177 @@
|
||||
# Kernels - Hardware-Optimized ML Operations
|
||||
---
|
||||
title: "Kernels - Hardware-Aware Optimization"
|
||||
description: "Custom operations, performance optimization, and hardware-aware computing for ML systems"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Kernels module! This is where we move beyond NumPy to understand how ML operations are optimized for modern hardware. You'll implement custom kernels that run faster than standard library functions.
|
||||
# 🚀 Module 11: Kernels - Hardware-Aware Optimization
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 12: 12 Kernels](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand why custom kernels matter for ML performance
|
||||
- Implement vectorized operations using SIMD principles
|
||||
- Master memory-efficient algorithms for better cache utilization
|
||||
- Build parallel processing patterns for CPU and GPU-style computing
|
||||
- Create performance profiling tools to measure and optimize code
|
||||
- Apply kernel optimizations to compressed model operations
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐⭐ | <strong>Time:</strong> 5-6 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐⭐ Expert
|
||||
- **Time Estimate**: 8-10 hours
|
||||
- **Prerequisites**: All previous modules (01-11), especially Compression
|
||||
- **Next Steps**: Benchmarking, MLOps modules
|
||||
|
||||
**Bridge the gap between algorithmic optimization and hardware-level performance engineering**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Understand how to implement custom ML operations beyond NumPy
|
||||
- Apply SIMD vectorization and CPU optimization techniques
|
||||
- Optimize memory layout and access patterns for cache efficiency
|
||||
- Implement GPU-style parallel computing concepts
|
||||
- Build comprehensive performance profiling and benchmarking tools
|
||||
- Create hardware-optimized operations for quantized and pruned models
|
||||
|
||||
## 🔗 Connection to Previous Modules
|
||||
|
||||
### What You Already Know
|
||||
- **Compression (Module 10)**: *What* to optimize (model size, computation)
|
||||
- **Layers (Module 03)**: Basic matrix multiplication with `matmul()`
|
||||
- **Training (Module 09)**: High-level optimization workflows
|
||||
- **Networks (Module 04)**: How operations compose into architectures
|
||||
|
||||
### The Performance Gap
|
||||
Students understand **algorithmic optimization** but not **hardware optimization**:
|
||||
- ✅ **Algorithmic**: Pruning, quantization, knowledge distillation
|
||||
- ❌ **Hardware**: Memory layout, vectorization, parallel processing
|
||||
|
||||
## 🧠 Build → Use → Optimize
|
||||
|
||||
This module follows the **"Build → Use → Optimize"** pedagogical framework:
|
||||
|
||||
### 1. **Build**: Custom Operations
|
||||
- Move beyond NumPy's black box implementations
|
||||
- Implement specialized matrix multiplication and activations
|
||||
- Understand the computational patterns underlying ML
|
||||
|
||||
### 2. **Use**: Performance Optimization
|
||||
- Apply SIMD vectorization for CPU optimization
|
||||
- Implement cache-friendly memory layouts
|
||||
- Build GPU-style parallel computing concepts
|
||||
|
||||
### 3. **Optimize**: Real-World Integration
|
||||
- Profile and benchmark performance improvements
|
||||
- Integrate with compressed models from Module 10
|
||||
- Apply systematic evaluation to validate optimizations
|
||||
|
||||
## 📚 What You'll Build
|
||||
|
||||
### Core Operations
|
||||
- **Matrix Multiplication**: Custom `matmul_baseline()` and `cache_friendly_matmul()`
|
||||
- **Activation Functions**: Vectorized `vectorized_relu()` and `parallel_relu()`
|
||||
- **Batch Processing**: `parallel_batch_processing()` with multiprocessing
|
||||
- **Quantized Operations**: `quantized_matmul()` and `quantized_relu()`
|
||||
|
||||
### Performance Tools
|
||||
- **Profiling**: `profile_operation()` for detailed timing analysis
|
||||
- **Benchmarking**: `benchmark_operation()` for statistical validation
|
||||
- **Memory Analysis**: Cache-friendly data layout optimization
|
||||
- **Parallel Computing**: Multi-core processing patterns
|
||||
|
||||
## 🛠️ Key Components
|
||||
|
||||
### Hardware-Optimized Operations
|
||||
- **Purpose**: Implement custom ML operations with hardware awareness
|
||||
- **Methods**: `matmul_baseline()`, `vectorized_relu()`, `cache_friendly_matmul()`
|
||||
- **Learning**: Understanding computational patterns beyond NumPy
|
||||
|
||||
### Parallel Processing Framework
|
||||
- **Purpose**: Multi-core optimization for batch operations
|
||||
- **Methods**: `parallel_batch_processing()`, `parallel_relu()`
|
||||
- **Learning**: GPU-style parallel computing concepts
|
||||
|
||||
### Quantization Integration
|
||||
- **Purpose**: Hardware-optimized operations for compressed models
|
||||
- **Methods**: `quantized_matmul()`, `quantized_relu()`
|
||||
- **Learning**: Bridging compression and performance optimization
|
||||
|
||||
### Performance Profiling
|
||||
- **Purpose**: Systematic measurement and validation of optimizations
|
||||
- **Methods**: `profile_operation()`, `benchmark_operation()`
|
||||
- **Learning**: Evidence-based performance engineering
|
||||
|
||||
## 🌟 Real-World Applications
|
||||
|
||||
### Industry Examples
|
||||
- **Google TPUs**: Custom hardware for ML operations
|
||||
- **Intel MKL**: Optimized math libraries for CPU performance
|
||||
- **NVIDIA cuDNN**: GPU-accelerated neural network operations
|
||||
- **Apple Neural Engine**: Hardware-specific ML acceleration
|
||||
|
||||
### Performance Patterns
|
||||
- **Memory Layout**: Row-major vs column-major access patterns
|
||||
- **Vectorization**: SIMD instructions for parallel computation
|
||||
- **Cache Optimization**: Data locality for memory hierarchy
|
||||
- **Parallel Processing**: Multi-core utilization strategies
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Prerequisites Check
|
||||
```bash
|
||||
tito test --module compression # Should pass
|
||||
tito status --module 11_kernels # Check module status
|
||||
```
|
||||
|
||||
|
||||
## Build → Use → Optimize
|
||||
1. **Build**: Custom operations, vectorization, and memory optimization
|
||||
2. **Use**: Apply optimized kernels to real ML workloads
|
||||
3. **Optimize**: Profile, measure, and improve performance systematically
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/12_kernels/kernels_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### Development Workflow
|
||||
```bash
|
||||
cd modules/source/11_kernels
|
||||
jupyter notebook kernels_dev.py # or edit directly
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/12_kernels/kernels_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
### Testing Your Implementation
|
||||
```bash
|
||||
# Test inline (within notebook)
|
||||
# Run comprehensive tests
|
||||
tito test --module kernels
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/12_kernels/kernels_dev.py
|
||||
:class-header: bg-light
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
## 📖 Module Structure
|
||||
```
|
||||
modules/source/11_kernels/
|
||||
├── kernels_dev.py # Main development file
|
||||
├── README.md # This overview
|
||||
└── module.yaml # Module configuration
|
||||
```
|
||||
|
||||
````
|
||||
## 🔗 Integration Points
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
### Input from Previous Modules
|
||||
- **Tensor operations** → Custom implementations
|
||||
- **Compressed models** → Hardware-optimized execution
|
||||
- **Training workflows** → Performance-critical operations
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
### Output to Next Modules
|
||||
- **Benchmarking** → Operations to evaluate systematically
|
||||
- **MLOps** → Production-ready optimized operations
|
||||
- **Complete system** → End-to-end performance optimization
|
||||
|
||||
## 🎓 Educational Philosophy
|
||||
|
||||
This module bridges the gap between **algorithmic understanding** and **systems performance**. Students learn that optimization isn't just about better algorithms—it's about understanding how algorithms interact with hardware to achieve real-world performance gains.
|
||||
|
||||
By the end, you'll think like a **performance engineer**, not just a machine learning practitioner.
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/11_compression.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/13_benchmarking.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,186 @@
|
||||
# Benchmarking - Systematic ML Performance Evaluation
|
||||
---
|
||||
title: "Benchmarking - Systematic ML Performance Evaluation"
|
||||
description: "Industry-standard benchmarking methodology for ML systems, inspired by MLPerf patterns"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the Benchmarking module! This is where we learn to systematically evaluate ML systems using industry-standard methodology inspired by MLPerf.
|
||||
# 📊 Module 12: Benchmarking - Systematic ML Performance Evaluation
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 13: 13 Benchmarking](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand the four-component MLPerf benchmarking architecture
|
||||
- Implement different benchmark scenarios (latency, throughput, offline)
|
||||
- Apply statistical validation for meaningful results
|
||||
- Create professional performance reports for ML projects
|
||||
- Learn to avoid common benchmarking pitfalls
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐⭐ | <strong>Time:</strong> 4-5 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐ Advanced
|
||||
- **Time Estimate**: 6-8 hours
|
||||
- **Prerequisites**: All previous modules (01-12), especially Kernels
|
||||
- **Next Steps**: MLOps module (13)
|
||||
|
||||
**Learn to systematically evaluate ML systems using industry-standard benchmarking methodology**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Design systematic benchmarking experiments for ML systems
|
||||
- Apply MLPerf-inspired patterns to evaluate model performance
|
||||
- Implement statistical validation for benchmark results
|
||||
- Create professional performance reports and comparisons
|
||||
- Apply systematic evaluation to real ML projects
|
||||
|
||||
## 🔗 Connection to Previous Modules
|
||||
|
||||
### What You Already Know
|
||||
- **Kernels (Module 11)**: *How* to optimize individual operations
|
||||
- **Training (Module 09)**: End-to-end model training workflows
|
||||
- **Compression (Module 10)**: Model optimization techniques
|
||||
- **Networks (Module 04)**: Model architectures and complexity
|
||||
|
||||
### The Evaluation Gap
|
||||
Students understand **how to build** ML systems but not **how to evaluate** them systematically:
|
||||
- ✅ **Implementation**: Can build tensors, layers, networks, optimizers
|
||||
- ❌ **Evaluation**: Don't know how to measure performance reliably
|
||||
- ✅ **Optimization**: Can implement kernels and compression
|
||||
- ❌ **Validation**: Can't prove optimizations actually work
|
||||
|
||||
## 🧠 Build → Use → Analyze
|
||||
|
||||
This module follows the **"Build → Use → Analyze"** pedagogical framework:
|
||||
|
||||
### 1. **Build**: Benchmarking Framework
|
||||
- Understand the four-component MLPerf architecture
|
||||
- Learn different benchmark scenarios (latency, throughput, server)
|
||||
- Implement statistical validation for meaningful results
|
||||
|
||||
### 2. **Use**: Systematic Evaluation
|
||||
- Apply benchmarking to your TinyTorch models
|
||||
- Compare different approaches systematically
|
||||
- Validate optimization claims with proper methodology
|
||||
|
||||
### 3. **Analyze**: Professional Reporting
|
||||
- Generate industry-standard performance reports
|
||||
- Present results with statistical confidence
|
||||
- Prepare for capstone project presentations
|
||||
|
||||
## 🎓 Why This Matters
|
||||
|
||||
### **Industry Reality**
|
||||
Real ML engineers spend significant time on:
|
||||
- **A/B testing**: Comparing model variants in production
|
||||
- **Performance optimization**: Proving optimizations actually work
|
||||
- **Research validation**: Demonstrating improvements over baselines
|
||||
- **System design**: Choosing between architectural alternatives
|
||||
|
||||
### **Professional Applications**
|
||||
This module prepares you for:
|
||||
- **ML project evaluation**: Systematic comparison against baselines
|
||||
- **Performance presentations**: Professional reporting of results
|
||||
- **Statistical validation**: Proving your improvements are significant
|
||||
- **Research methodology**: Reproducible evaluation practices
|
||||
|
||||
## 🚀 Key Concepts
|
||||
|
||||
### **MLPerf-Inspired Architecture**
|
||||
- **System Under Test (SUT)**: Your ML model/system
|
||||
- **Dataset**: Standardized evaluation data
|
||||
- **Model**: The specific architecture being tested
|
||||
- **Load Generator**: Controls how evaluation queries are sent
|
||||
|
||||
### **Benchmark Scenarios**
|
||||
- **Single-Stream**: Measures latency (mobile/edge use cases)
|
||||
- **Server**: Measures throughput (production server use cases)
|
||||
- **Offline**: Measures batch processing (data center use cases)
|
||||
|
||||
### **Statistical Validation**
|
||||
- **Confidence intervals**: Ensuring results are meaningful
|
||||
- **Multiple runs**: Accounting for variability
|
||||
- **Significance testing**: Proving improvements are real
|
||||
- **Pitfall detection**: Avoiding common benchmarking mistakes
|
||||
|
||||
## 🔧 What You'll Build
|
||||
|
||||
### **1. TinyTorchPerf Framework**
|
||||
```python
|
||||
from tinytorch.benchmarking import TinyTorchPerf
|
||||
|
||||
# Professional ML benchmarking
|
||||
benchmark = TinyTorchPerf()
|
||||
benchmark.set_model(your_model)
|
||||
benchmark.set_dataset('cifar10')
|
||||
|
||||
# Run different scenarios
|
||||
results = benchmark.run_all_scenarios()
|
||||
```
|
||||
|
||||
|
||||
## Build → Use → Analyze
|
||||
1. **Build**: Benchmarking framework with proper statistical validation
|
||||
2. **Use**: Apply systematic evaluation to your TinyTorch models
|
||||
3. **Analyze**: Generate professional reports with statistical confidence
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/13_benchmarking/benchmarking_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
### **2. Statistical Validator**
|
||||
```python
|
||||
# Ensure statistically valid results
|
||||
validator = StatisticalValidator()
|
||||
validation = validator.validate_results(results)
|
||||
if validation.significant:
|
||||
print("✅ Improvement is statistically significant")
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/13_benchmarking/benchmarking_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
### **3. Performance Reporter**
|
||||
```python
|
||||
# Generate professional reports
|
||||
reporter = PerformanceReporter()
|
||||
report = reporter.generate_report(results)
|
||||
report.save_as_html("my_capstone_results.html")
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/13_benchmarking/benchmarking_dev.py
|
||||
:class-header: bg-light
|
||||
## 📈 Real-World Applications
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
### **Immediate Use Cases**
|
||||
- **ML projects**: Systematic evaluation of your model implementations
|
||||
- **Module integration**: Validate that your TinyTorch components work together
|
||||
- **Performance optimization**: Prove your kernels actually improve performance
|
||||
|
||||
````
|
||||
### **Career Applications**
|
||||
- **Research**: Proper experimental methodology for papers
|
||||
- **Industry**: A/B testing and performance optimization
|
||||
- **Open source**: Contributing benchmarks to ML libraries
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
## 🎯 Success Metrics
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
```
|
||||
By the end of this module, you should be able to:
|
||||
- [ ] Design a systematic benchmark for any ML system
|
||||
- [ ] Apply MLPerf principles to your own evaluations
|
||||
- [ ] Generate statistically valid performance comparisons
|
||||
- [ ] Create professional reports suitable for presentations
|
||||
- [ ] Identify and avoid common benchmarking pitfalls
|
||||
|
||||
## 🔄 Connection to Module 13 (MLOps)
|
||||
|
||||
**Benchmarking** → **Production Monitoring**
|
||||
- Benchmarking establishes baselines for production systems
|
||||
- Monitoring detects when production deviates from benchmarks
|
||||
- Both use similar metrics and statistical validation
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [MLPerf Inference Rules](https://github.com/mlcommons/inference_policies)
|
||||
- [Statistical Methods for ML Evaluation](https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/)
|
||||
- [A/B Testing for ML Systems](https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15)
|
||||
|
||||
---
|
||||
|
||||
**🎉 Ready to become a systematic ML evaluator? Let's build professional benchmarking skills!**
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/12_kernels.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/14_mlops.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
|
||||
@@ -1,54 +1,355 @@
|
||||
# MLOps - Production ML Systems
|
||||
---
|
||||
title: "MLOps - Production ML Systems"
|
||||
description: "Complete MLOps pipeline for production deployment, monitoring, and continuous learning"
|
||||
difficulty: "Intermediate"
|
||||
time_estimate: "2-4 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
---
|
||||
|
||||
Welcome to the MLOps module! This is where we close the loop on the complete ML system lifecycle.
|
||||
# 🚀 Module 13: MLOps - Production ML Systems
|
||||
---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module 14: 14 Mlops](#)
|
||||
|
||||
```{admonition} 🎯 Learning Goals
|
||||
:class: tip
|
||||
- Understand why ML models degrade over time without maintenance
|
||||
- Implement performance monitoring and drift detection systems
|
||||
- Build automated retraining triggers that use your training pipeline
|
||||
- Create model comparison and deployment workflows
|
||||
- See how all TinyTorch components work together in production
|
||||
---
|
||||
|
||||
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ ⭐⭐⭐⭐⭐ | <strong>Time:</strong> 6-8 hours</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐⭐⭐⭐ Expert
|
||||
- **Time Estimate**: 10-12 hours
|
||||
- **Prerequisites**: All previous modules (01-13) - Complete TinyTorch ecosystem
|
||||
- **Next Steps**: **Final capstone module** - Deploy your complete ML system!
|
||||
|
||||
**Build production-ready ML systems with deployment, monitoring, and continuous learning**
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
|
||||
After completing this module, you will:
|
||||
- Build complete MLOps pipelines from model development to production
|
||||
- Implement model versioning and registry systems for lifecycle management
|
||||
- Create production-ready model serving and inference endpoints
|
||||
- Design monitoring systems for model performance and data drift detection
|
||||
- Apply A/B testing methodology for safe model deployment
|
||||
- Implement continuous learning systems for model improvement
|
||||
- Integrate all TinyTorch components into production-ready systems
|
||||
|
||||
## 🧠 Build → Use → Deploy
|
||||
|
||||
This module follows the TinyTorch **"Build → Use → Deploy"** pedagogical framework:
|
||||
|
||||
1. **Build**: Complete MLOps infrastructure and production systems
|
||||
2. **Use**: Deploy and operate ML systems in production environments
|
||||
3. **Deploy**: Create end-to-end ML pipelines ready for real-world deployment
|
||||
|
||||
## 🔗 Connection to Previous Modules
|
||||
|
||||
### The Complete TinyTorch Ecosystem
|
||||
MLOps is the **capstone module** that brings together everything you've built:
|
||||
|
||||
- **00_setup**: System configuration and development environment
|
||||
- **01_tensor**: Data structures and operations
|
||||
- **02_activations**: Nonlinear functions for neural networks
|
||||
- **03_layers**: Building blocks of neural networks
|
||||
- **04_networks**: Complete neural network architectures
|
||||
- **05_cnn**: Convolutional networks for image processing
|
||||
- **06_dataloader**: Data loading and preprocessing pipelines
|
||||
- **07_autograd**: Automatic differentiation for training
|
||||
- **08_optimizers**: Training algorithms and optimization
|
||||
- **09_training**: Complete training pipelines and workflows
|
||||
- **10_compression**: Model optimization for deployment
|
||||
- **11_kernels**: Hardware-optimized operations
|
||||
- **12_benchmarking**: Performance measurement and evaluation
|
||||
|
||||
### The Production Gap
|
||||
Students understand **how to build** and **how to optimize** ML systems but not **how to deploy** them:
|
||||
- ✅ **Development**: Can build complete ML systems from scratch
|
||||
- ✅ **Optimization**: Can compress, accelerate, and benchmark models
|
||||
- ❌ **Production**: Don't know how to deploy, monitor, and maintain systems
|
||||
- ❌ **Operations**: Can't handle model versioning, A/B testing, or continuous learning
|
||||
|
||||
## 📚 What You'll Build
|
||||
|
||||
### **Model Management System**
|
||||
```python
|
||||
# Model versioning and registry
|
||||
registry = ModelRegistry("production")
|
||||
model_v1 = registry.register_model(trained_model, version="1.0.0")
|
||||
model_v2 = registry.register_model(compressed_model, version="2.0.0")
|
||||
|
||||
# Version comparison
|
||||
comparison = registry.compare_models("1.0.0", "2.0.0")
|
||||
```
|
||||
|
||||
### **Production Serving System**
|
||||
```python
|
||||
# Model serving endpoint
|
||||
server = ModelServer(model_v2, port=8080)
|
||||
server.start()
|
||||
|
||||
## Build → Use → Deploy
|
||||
1. **Build**: Complete MLOps infrastructure for model lifecycle management
|
||||
2. **Use**: Deploy and monitor ML systems that automatically respond to issues
|
||||
3. **Deploy**: Create production-ready systems that maintain themselves over time
|
||||
## 🚀 Interactive Learning
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/14_mlops/mlops_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
# Inference endpoint
|
||||
endpoint = InferenceEndpoint(server)
|
||||
prediction = endpoint.predict(input_data)
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/14_mlops/mlops_dev.ipynb
|
||||
:class-header: bg-light
|
||||
### **Monitoring & Observability**
|
||||
```python
|
||||
# Model performance monitoring
|
||||
monitor = ModelMonitor(model_v2)
|
||||
monitor.track_latency(prediction_time)
|
||||
monitor.track_accuracy(predictions, true_labels)
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
# Data drift detection
|
||||
drift_detector = DriftDetector(reference_data)
|
||||
drift_detected = drift_detector.detect_drift(new_data)
|
||||
```
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/14_mlops/mlops_dev.py
|
||||
:class-header: bg-light
|
||||
### **A/B Testing Framework**
|
||||
```python
|
||||
# Safe model deployment
|
||||
ab_test = ABTestManager()
|
||||
ab_test.add_variant("control", model_v1, traffic_split=0.8)
|
||||
ab_test.add_variant("treatment", model_v2, traffic_split=0.2)
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
# Experiment tracking
|
||||
results = ab_test.run_experiment(test_data)
|
||||
```
|
||||
|
||||
````
|
||||
### **Continuous Learning System**
|
||||
```python
|
||||
# Automated retraining
|
||||
learner = ContinuousLearner(model_v2)
|
||||
learner.add_training_data(new_data)
|
||||
improved_model = learner.retrain_if_needed()
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
|
||||
Ready for serious development? → [🏗️ Local Setup Guide](../usage-paths/serious-development.md)
|
||||
# Automated deployment
|
||||
pipeline = MLOpsPipeline()
|
||||
pipeline.train_model(new_data)
|
||||
pipeline.validate_model(validation_data)
|
||||
pipeline.deploy_model(improved_model)
|
||||
```
|
||||
|
||||
## 🎓 Educational Structure
|
||||
|
||||
### **Step 1: Model Management & Versioning**
|
||||
- **Concept**: Model lifecycle management and version control
|
||||
- **Implementation**: ModelRegistry, ModelVersioning, ModelSerializer
|
||||
- **Learning**: Track model evolution and manage production deployments
|
||||
|
||||
### **Step 2: Production Serving & Deployment**
|
||||
- **Concept**: Scalable model serving and inference endpoints
|
||||
- **Implementation**: ModelServer, InferenceEndpoint, BatchInference
|
||||
- **Learning**: Deploy models for real-time and batch inference
|
||||
|
||||
### **Step 3: Monitoring & Observability**
|
||||
- **Concept**: Production model monitoring and performance tracking
|
||||
- **Implementation**: ModelMonitor, PerformanceTracker, DriftDetector
|
||||
- **Learning**: Detect issues and maintain model quality in production
|
||||
|
||||
### **Step 4: A/B Testing & Experimentation**
|
||||
- **Concept**: Safe deployment through controlled experiments
|
||||
- **Implementation**: ABTestManager, ExperimentTracker, ModelComparator
|
||||
- **Learning**: Validate model improvements with statistical rigor
|
||||
|
||||
### **Step 5: Continuous Learning & Automation**
|
||||
- **Concept**: Automated model improvement and retraining
|
||||
- **Implementation**: ContinuousLearner, AutoRetrainer, DataPipeline
|
||||
- **Learning**: Build self-improving ML systems
|
||||
|
||||
### **Step 6: Complete MLOps Pipeline**
|
||||
- **Concept**: End-to-end production ML system orchestration
|
||||
- **Implementation**: MLOpsPipeline, DeploymentManager, ProductionValidator
|
||||
- **Learning**: Integrate all components into production-ready systems
|
||||
|
||||
## 🌍 Real-World Applications
|
||||
|
||||
### **Production ML Systems**
|
||||
- **Netflix**: Recommendation system deployment and A/B testing
|
||||
- **Uber**: Real-time demand prediction and dynamic pricing
|
||||
- **Spotify**: Music recommendation and playlist generation
|
||||
- **Google**: Search ranking and ad serving systems
|
||||
|
||||
### **Model Lifecycle Management**
|
||||
- **Airbnb**: Price prediction model versioning and deployment
|
||||
- **Facebook**: News feed algorithm updates and rollbacks
|
||||
- **Amazon**: Product recommendation system evolution
|
||||
- **Tesla**: Autonomous driving model deployment and monitoring
|
||||
|
||||
### **Monitoring & Observability**
|
||||
- **Stripe**: Fraud detection system monitoring
|
||||
- **Zillow**: Home price prediction accuracy tracking
|
||||
- **LinkedIn**: Job recommendation performance monitoring
|
||||
- **Twitter**: Content moderation model drift detection
|
||||
|
||||
### **Continuous Learning**
|
||||
- **YouTube**: Video recommendation system adaptation
|
||||
- **Instagram**: Content filtering continuous improvement
|
||||
- **Snapchat**: Face filter quality enhancement
|
||||
- **TikTok**: Content discovery algorithm evolution
|
||||
|
||||
## 🔧 Technical Architecture
|
||||
|
||||
### **Production Requirements**
|
||||
```python
|
||||
# Performance requirements
|
||||
- Latency: < 100ms inference time
|
||||
- Throughput: > 1000 requests/second
|
||||
- Availability: 99.9% uptime
|
||||
- Scalability: Handle traffic spikes
|
||||
|
||||
# Reliability requirements
|
||||
- Model versioning: Track all model changes
|
||||
- Rollback capability: Revert to previous versions
|
||||
- Monitoring: Real-time performance tracking
|
||||
- Alerting: Automated issue detection
|
||||
```
|
||||
|
||||
### **Integration with TinyTorch Components**
|
||||
```python
|
||||
# Complete system integration
|
||||
from tinytorch.core.training import Trainer
|
||||
from tinytorch.core.compression import quantize_model
|
||||
from tinytorch.core.kernels import optimize_inference
|
||||
from tinytorch.core.benchmarking import benchmark_model
|
||||
from tinytorch.core.mlops import MLOpsPipeline
|
||||
|
||||
# End-to-end pipeline
|
||||
pipeline = MLOpsPipeline()
|
||||
trained_model = pipeline.train_with_trainer(Trainer, data)
|
||||
compressed_model = pipeline.compress_model(quantize_model, trained_model)
|
||||
optimized_model = pipeline.optimize_inference(optimize_inference, compressed_model)
|
||||
benchmark_results = pipeline.benchmark_model(benchmark_model, optimized_model)
|
||||
deployed_model = pipeline.deploy_model(optimized_model)
|
||||
```
|
||||
|
||||
## 🎯 Key Skills Developed
|
||||
|
||||
### **Systems Engineering**
|
||||
- **Architecture design**: Scalable, reliable ML system design
|
||||
- **Performance optimization**: Low-latency, high-throughput systems
|
||||
- **Reliability engineering**: Fault-tolerant and self-healing systems
|
||||
- **Monitoring & observability**: Comprehensive system health tracking
|
||||
|
||||
### **ML Engineering**
|
||||
- **Model lifecycle management**: Version control and deployment strategies
|
||||
- **Production deployment**: Safe, scalable model serving
|
||||
- **Continuous learning**: Automated model improvement workflows
|
||||
- **Experiment design**: A/B testing and statistical validation
|
||||
|
||||
### **DevOps & Platform Engineering**
|
||||
- **CI/CD pipelines**: Automated testing and deployment
|
||||
- **Infrastructure as code**: Reproducible deployment environments
|
||||
- **Container orchestration**: Scalable model serving infrastructure
|
||||
- **Monitoring & alerting**: Proactive issue detection and resolution
|
||||
|
||||
## 🏆 Capstone Project: Complete ML System
|
||||
|
||||
### **Project Overview**
|
||||
Build a complete, production-ready ML system that demonstrates mastery of the entire TinyTorch ecosystem.
|
||||
|
||||
### **Project Components**
|
||||
1. **Data Pipeline**: Automated data ingestion and preprocessing
|
||||
2. **Model Training**: Automated training with hyperparameter optimization
|
||||
3. **Model Optimization**: Compression and kernel optimization
|
||||
4. **Benchmarking**: Performance evaluation and comparison
|
||||
5. **Deployment**: Production serving with monitoring
|
||||
6. **Continuous Learning**: Automated retraining and improvement
|
||||
|
||||
### **Deliverables**
|
||||
- **Trained Model**: High-quality model trained on real data
|
||||
- **Compressed Model**: Optimized for production deployment
|
||||
- **Serving Endpoint**: Production-ready inference API
|
||||
- **Monitoring Dashboard**: Real-time performance tracking
|
||||
- **A/B Testing Framework**: Safe deployment validation
|
||||
- **Continuous Learning Pipeline**: Automated improvement system
|
||||
|
||||
## 🔮 Industry Connections
|
||||
|
||||
### **MLOps Platforms**
|
||||
- **MLflow**: Model lifecycle management and experiment tracking
|
||||
- **Kubeflow**: Kubernetes-based ML workflows and pipelines
|
||||
- **TensorFlow Extended (TFX)**: End-to-end ML platform
|
||||
- **Amazon SageMaker**: AWS managed ML platform
|
||||
- **Google AI Platform**: Google Cloud ML services
|
||||
- **Azure ML**: Microsoft's comprehensive ML platform
|
||||
|
||||
### **Production ML Systems**
|
||||
- **TensorFlow Serving**: High-performance model serving
|
||||
- **PyTorch Serve**: PyTorch model deployment
|
||||
- **ONNX Runtime**: Cross-platform inference optimization
|
||||
- **Apache Kafka**: Real-time data streaming
|
||||
- **Prometheus**: Monitoring and alerting
|
||||
- **Grafana**: Visualization and dashboards
|
||||
|
||||
### **Career Preparation**
|
||||
- **ML Engineer**: Production ML system development
|
||||
- **MLOps Engineer**: ML infrastructure and operations
|
||||
- **Data Engineer**: ML data pipeline development
|
||||
- **Platform Engineer**: ML platform and tooling
|
||||
- **Site Reliability Engineer**: Production system reliability
|
||||
- **ML Researcher**: Advanced ML system research
|
||||
|
||||
## 🚀 What's Next
|
||||
|
||||
### **Beyond TinyTorch**
|
||||
Your MLOps skills prepare you for:
|
||||
- **Production ML roles**: Industry-ready deployment expertise
|
||||
- **Advanced ML systems**: Distributed training, federated learning
|
||||
- **ML platform development**: Building ML infrastructure and tools
|
||||
- **Research applications**: Reproducible, scalable research systems
|
||||
|
||||
### **Continuous Learning**
|
||||
- **Advanced MLOps**: Multi-model systems, federated learning
|
||||
- **ML Security**: Model privacy, security, and governance
|
||||
- **AutoML**: Automated machine learning systems
|
||||
- **Edge ML**: Deployment on edge devices and IoT systems
|
||||
|
||||
## 📁 File Structure
|
||||
```
|
||||
13_mlops/
|
||||
├── mlops_dev.py # Main development notebook
|
||||
├── module.yaml # Module configuration
|
||||
├── README.md # This file
|
||||
├── deployments/ # Deployment configurations
|
||||
│ ├── docker/ # Container configurations
|
||||
│ ├── kubernetes/ # K8s deployment configs
|
||||
│ └── monitoring/ # Monitoring configurations
|
||||
└── tests/ # Additional test files
|
||||
└── test_mlops.py # External tests
|
||||
```
|
||||
|
||||
## 🎯 Getting Started
|
||||
|
||||
1. **Review Prerequisites**: Ensure all modules 01-13 are complete
|
||||
2. **Open Development File**: `mlops_dev.py`
|
||||
3. **Follow Educational Flow**: Work through Steps 1-6 sequentially
|
||||
4. **Build Capstone Project**: Complete end-to-end ML system
|
||||
5. **Test Production System**: Validate deployment and monitoring
|
||||
6. **Export to Package**: Use `tito export 13_mlops` when complete
|
||||
|
||||
## 🎉 Final Achievement
|
||||
|
||||
Students completing this module will:
|
||||
- **Master production ML systems**: End-to-end deployment expertise
|
||||
- **Understand ML operations**: Complete MLOps lifecycle management
|
||||
- **Build scalable systems**: Production-ready ML infrastructure
|
||||
- **Apply best practices**: Industry-standard deployment and monitoring
|
||||
- **Demonstrate expertise**: Complete TinyTorch ecosystem mastery
|
||||
- **Prepare for careers**: Industry-ready ML engineering skills
|
||||
|
||||
**Congratulations!** You've built a complete ML framework from scratch and learned to deploy it in production. You're now ready to tackle real-world ML systems with confidence and expertise!
|
||||
|
||||
This module represents the culmination of your TinyTorch journey - from basic tensors to production-ready ML systems. You've gained the skills to build, optimize, and deploy ML systems that can handle real-world challenges and scale to production requirements.
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/13_benchmarking.html" title="previous page">← Previous Module</a>
|
||||
</div>
|
||||
|
||||
253
book/convert_readmes.py
Normal file
253
book/convert_readmes.py
Normal file
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Convert module READMEs to Jupyter Book chapters.
|
||||
|
||||
This script takes README files from modules/source/*/README.md and converts them
|
||||
to Jupyter Book chapters in book/chapters/ with proper frontmatter and web optimization.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
def get_module_info(module_path: Path) -> Dict[str, str]:
|
||||
"""Extract module information from module.yaml file."""
|
||||
yaml_path = module_path / "module.yaml"
|
||||
if yaml_path.exists():
|
||||
with open(yaml_path, 'r') as f:
|
||||
module_data = yaml.safe_load(f)
|
||||
return {
|
||||
'title': module_data.get('title', module_path.name.replace('_', ' ').title()),
|
||||
'description': module_data.get('description', ''),
|
||||
'difficulty': module_data.get('difficulty', 'Intermediate'),
|
||||
'time_estimate': module_data.get('time_estimate', '2-4 hours'),
|
||||
'prerequisites': module_data.get('prerequisites', []),
|
||||
'next_steps': module_data.get('next_steps', [])
|
||||
}
|
||||
return {}
|
||||
|
||||
def extract_learning_objectives(content: str) -> List[str]:
|
||||
"""Extract learning objectives from README content."""
|
||||
objectives = []
|
||||
# Look for common patterns in READMEs
|
||||
patterns = [
|
||||
r'By the end of this module, you will:?\s*\n((?:- [^\n]+\n?)+)',
|
||||
r'Learning Goals?:?\s*\n((?:- [^\n]+\n?)+)',
|
||||
r'Learning Objectives?:?\s*\n((?:- [^\n]+\n?)+)'
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
match = re.search(pattern, content, re.IGNORECASE | re.MULTILINE)
|
||||
if match:
|
||||
objectives_text = match.group(1)
|
||||
objectives = [line.strip('- ').strip() for line in objectives_text.split('\n') if line.strip().startswith('-')]
|
||||
break
|
||||
|
||||
return objectives
|
||||
|
||||
def create_frontmatter(module_name: str, module_info: Dict[str, str], objectives: List[str]) -> str:
|
||||
"""Create Jupyter Book frontmatter for the chapter."""
|
||||
# Clean up module name for title
|
||||
title = module_info.get('title', module_name.replace('_', ' ').title())
|
||||
|
||||
frontmatter = f"""---
|
||||
title: "{title}"
|
||||
description: "{module_info.get('description', '')}"
|
||||
difficulty: "{module_info.get('difficulty', 'Intermediate')}"
|
||||
time_estimate: "{module_info.get('time_estimate', '2-4 hours')}"
|
||||
prerequisites: {module_info.get('prerequisites', [])}
|
||||
next_steps: {module_info.get('next_steps', [])}
|
||||
learning_objectives: {objectives}
|
||||
---
|
||||
|
||||
"""
|
||||
return frontmatter
|
||||
|
||||
def enhance_content_for_web(content: str, module_name: str, module_num: int) -> str:
|
||||
"""Enhance README content for web display."""
|
||||
# Add navigation breadcrumbs
|
||||
breadcrumb = f"""---
|
||||
**Course Navigation:** [Home](../intro.html) → [Module {module_num}: {module_name.replace('_', ' ').title()}](#)
|
||||
|
||||
---
|
||||
"""
|
||||
|
||||
# Add difficulty and time badges
|
||||
badges = f"""
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">📊 Module Info</p>
|
||||
<p><strong>Difficulty:</strong> ⭐ {get_difficulty_stars(module_name)} | <strong>Time:</strong> {get_time_estimate(module_name)}</p>
|
||||
</div>
|
||||
|
||||
"""
|
||||
|
||||
# Add navigation links at the end
|
||||
nav_links = f"""
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
"""
|
||||
|
||||
if module_num > 1:
|
||||
prev_module = f"{module_num-1:02d}_{get_prev_module_name(module_num)}"
|
||||
nav_links += f'<a class="left-prev" href="../chapters/{prev_module}.html" title="previous page">← Previous Module</a>\n'
|
||||
|
||||
if module_num < 14: # Assuming 14 modules total
|
||||
next_module = f"{module_num+1:02d}_{get_next_module_name(module_num)}"
|
||||
nav_links += f'<a class="right-next" href="../chapters/{next_module}.html" title="next page">Next Module →</a>\n'
|
||||
|
||||
nav_links += "</div>\n"
|
||||
|
||||
# Insert breadcrumb and badges after the first heading
|
||||
lines = content.split('\n')
|
||||
enhanced_lines = []
|
||||
added_breadcrumb = False
|
||||
added_badges = False
|
||||
|
||||
for i, line in enumerate(lines):
|
||||
enhanced_lines.append(line)
|
||||
|
||||
# Add breadcrumb after first heading
|
||||
if not added_breadcrumb and line.startswith('# '):
|
||||
enhanced_lines.append(breadcrumb)
|
||||
added_breadcrumb = True
|
||||
|
||||
# Add badges after breadcrumb
|
||||
if added_breadcrumb and not added_badges:
|
||||
enhanced_lines.append(badges)
|
||||
added_badges = True
|
||||
|
||||
# Add navigation at the end
|
||||
enhanced_lines.append(nav_links)
|
||||
|
||||
return '\n'.join(enhanced_lines)
|
||||
|
||||
def get_difficulty_stars(module_name: str) -> str:
|
||||
"""Get difficulty stars based on module name."""
|
||||
difficulty_map = {
|
||||
'01_setup': '⭐',
|
||||
'02_tensor': '⭐⭐',
|
||||
'03_activations': '⭐⭐',
|
||||
'04_layers': '⭐⭐⭐',
|
||||
'05_networks': '⭐⭐⭐',
|
||||
'06_cnn': '⭐⭐⭐⭐',
|
||||
'07_dataloader': '⭐⭐⭐',
|
||||
'08_autograd': '⭐⭐⭐⭐',
|
||||
'09_optimizers': '⭐⭐⭐⭐',
|
||||
'10_training': '⭐⭐⭐⭐',
|
||||
'11_compression': '⭐⭐⭐⭐⭐',
|
||||
'12_kernels': '⭐⭐⭐⭐⭐',
|
||||
'13_benchmarking': '⭐⭐⭐⭐⭐',
|
||||
'14_mlops': '⭐⭐⭐⭐⭐'
|
||||
}
|
||||
return difficulty_map.get(module_name, '⭐⭐')
|
||||
|
||||
def get_time_estimate(module_name: str) -> str:
|
||||
"""Get time estimate based on module name."""
|
||||
time_map = {
|
||||
'01_setup': '1-2 hours',
|
||||
'02_tensor': '4-6 hours',
|
||||
'03_activations': '3-4 hours',
|
||||
'04_layers': '4-5 hours',
|
||||
'05_networks': '5-6 hours',
|
||||
'06_cnn': '6-8 hours',
|
||||
'07_dataloader': '4-5 hours',
|
||||
'08_autograd': '6-8 hours',
|
||||
'09_optimizers': '5-6 hours',
|
||||
'10_training': '6-8 hours',
|
||||
'11_compression': '4-5 hours',
|
||||
'12_kernels': '5-6 hours',
|
||||
'13_benchmarking': '4-5 hours',
|
||||
'14_mlops': '6-8 hours'
|
||||
}
|
||||
return time_map.get(module_name, '3-4 hours')
|
||||
|
||||
def get_prev_module_name(module_num: int) -> str:
|
||||
"""Get previous module name."""
|
||||
module_names = [
|
||||
'setup', 'tensor', 'activations', 'layers', 'networks', 'cnn',
|
||||
'dataloader', 'autograd', 'optimizers', 'training', 'compression',
|
||||
'kernels', 'benchmarking', 'mlops'
|
||||
]
|
||||
return module_names[module_num - 2] if module_num > 1 else 'setup'
|
||||
|
||||
def get_next_module_name(module_num: int) -> str:
|
||||
"""Get next module name."""
|
||||
module_names = [
|
||||
'setup', 'tensor', 'activations', 'layers', 'networks', 'cnn',
|
||||
'dataloader', 'autograd', 'optimizers', 'training', 'compression',
|
||||
'kernels', 'benchmarking', 'mlops'
|
||||
]
|
||||
return module_names[module_num] if module_num < len(module_names) else 'mlops'
|
||||
|
||||
def convert_readme_to_chapter(readme_path: Path, chapter_path: Path, module_num: int):
|
||||
"""Convert a single README to a Jupyter Book chapter."""
|
||||
print(f"Converting {readme_path} to {chapter_path}")
|
||||
|
||||
# Read README content
|
||||
with open(readme_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
# Get module information
|
||||
module_path = readme_path.parent
|
||||
module_name = module_path.name
|
||||
module_info = get_module_info(module_path)
|
||||
|
||||
# Extract learning objectives
|
||||
objectives = extract_learning_objectives(content)
|
||||
|
||||
# Create frontmatter
|
||||
frontmatter = create_frontmatter(module_name, module_info, objectives)
|
||||
|
||||
# Enhance content for web
|
||||
enhanced_content = enhance_content_for_web(content, module_name, module_num)
|
||||
|
||||
# Write chapter file
|
||||
with open(chapter_path, 'w', encoding='utf-8') as f:
|
||||
f.write(frontmatter)
|
||||
f.write(enhanced_content)
|
||||
|
||||
print(f"✅ Created {chapter_path}")
|
||||
|
||||
def main():
|
||||
"""Convert all module READMEs to Jupyter Book chapters."""
|
||||
# Setup paths
|
||||
modules_dir = Path("../modules/source")
|
||||
chapters_dir = Path("chapters")
|
||||
|
||||
# Ensure chapters directory exists
|
||||
chapters_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Get all module directories (sorted by number)
|
||||
module_dirs = []
|
||||
for item in modules_dir.iterdir():
|
||||
if item.is_dir() and item.name != 'utils':
|
||||
# Extract module number from directory name
|
||||
match = re.match(r'(\d+)_(.+)', item.name)
|
||||
if match:
|
||||
module_num = int(match.group(1))
|
||||
module_dirs.append((module_num, item))
|
||||
|
||||
# Sort by module number
|
||||
module_dirs.sort(key=lambda x: x[0])
|
||||
|
||||
print(f"Found {len(module_dirs)} modules to convert")
|
||||
|
||||
# Convert each README
|
||||
for module_num, module_dir in module_dirs:
|
||||
readme_path = module_dir / "README.md"
|
||||
if readme_path.exists():
|
||||
# Create chapter filename (just module number and name, no duplicate)
|
||||
chapter_filename = f"{module_num:02d}-{module_dir.name.split('_', 1)[1]}.md"
|
||||
chapter_path = chapters_dir / chapter_filename
|
||||
|
||||
convert_readme_to_chapter(readme_path, chapter_path, module_num)
|
||||
else:
|
||||
print(f"⚠️ No README.md found in {module_dir}")
|
||||
|
||||
print(f"\n🎉 Converted {len(module_dirs)} modules to chapters in {chapters_dir}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -104,7 +104,7 @@ class BookCommand(BaseCommand):
|
||||
try:
|
||||
os.chdir("book")
|
||||
result = subprocess.run(
|
||||
["python3", "convert_modules.py", "--overview"],
|
||||
["python3", "convert_readmes.py"],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
@@ -112,17 +112,16 @@ class BookCommand(BaseCommand):
|
||||
if result.returncode == 0:
|
||||
console.print("✅ Overview pages generated successfully")
|
||||
# Show summary from the output
|
||||
if "Success:" in result.stdout:
|
||||
for line in result.stdout.split('\n'):
|
||||
if "Success:" in line or "Output:" in line:
|
||||
console.print(f" {line.strip()}")
|
||||
for line in result.stdout.split('\n'):
|
||||
if "✅ Created" in line or "🎉 Converted" in line:
|
||||
console.print(f" {line.strip()}")
|
||||
return 0
|
||||
else:
|
||||
console.print(f"[red]❌ Failed to generate overview pages: {result.stderr}[/red]")
|
||||
return 1
|
||||
|
||||
except FileNotFoundError:
|
||||
console.print("[red]❌ Python3 not found or convert_modules.py missing[/red]")
|
||||
console.print("[red]❌ Python3 not found or convert_readmes.py missing[/red]")
|
||||
return 1
|
||||
except Exception as e:
|
||||
console.print(f"[red]❌ Error generating overview pages: {e}[/red]")
|
||||
@@ -131,48 +130,35 @@ class BookCommand(BaseCommand):
|
||||
os.chdir("..")
|
||||
|
||||
def _generate_all(self) -> int:
|
||||
"""Generate notebooks in book/chapters/ and update book references."""
|
||||
"""Convert READMEs to Jupyter Book chapters."""
|
||||
console = self.console
|
||||
console.print("🔄 Generating notebooks for Jupyter Book...")
|
||||
console.print("🔄 Converting module READMEs to Jupyter Book chapters...")
|
||||
|
||||
# Step 1: Generate notebooks in book/chapters/ (single source of truth for the book)
|
||||
console.print("📝 Step 1: Generating notebooks in book/chapters/...")
|
||||
# Step 1: Convert READMEs to chapters
|
||||
console.print("📝 Step 1: Converting READMEs to chapters...")
|
||||
try:
|
||||
os.chdir("book")
|
||||
result = subprocess.run([
|
||||
"python3", "convert_modules.py", "--all"
|
||||
"python3", "convert_readmes.py"
|
||||
], capture_output=True, text=True)
|
||||
|
||||
if result.returncode == 0:
|
||||
console.print(" ✅ All notebooks generated in book/chapters/")
|
||||
console.print(" ✅ All READMEs converted to chapters")
|
||||
# Show summary from the output
|
||||
for line in result.stdout.split('\n'):
|
||||
if "✅ Created" in line or "🎉 Converted" in line:
|
||||
console.print(f" {line.strip()}")
|
||||
else:
|
||||
console.print(f" ❌ Failed to generate notebooks: {result.stderr}")
|
||||
console.print(f" ❌ Failed to convert READMEs: {result.stderr}")
|
||||
return 1
|
||||
except Exception as e:
|
||||
console.print(f"[red]❌ Error generating notebooks: {e}[/red]")
|
||||
console.print(f"[red]❌ Error converting READMEs: {e}[/red]")
|
||||
return 1
|
||||
finally:
|
||||
os.chdir("..")
|
||||
|
||||
# Step 2: Generate overview pages (if needed)
|
||||
console.print("📚 Step 2: Generating overview pages...")
|
||||
try:
|
||||
os.chdir("book")
|
||||
result = subprocess.run([
|
||||
"python3", "convert_modules.py", "--overview"
|
||||
], capture_output=True, text=True)
|
||||
|
||||
if result.returncode == 0:
|
||||
console.print("✅ Overview pages generated in book/chapters/")
|
||||
return 0
|
||||
else:
|
||||
console.print(f"[red]❌ Failed to generate overview pages: {result.stderr}[/red]")
|
||||
return 1
|
||||
except Exception as e:
|
||||
console.print(f"[red]❌ Error generating overview pages: {e}[/red]")
|
||||
return 1
|
||||
finally:
|
||||
os.chdir("..")
|
||||
console.print("✅ Chapters generated successfully")
|
||||
return 0
|
||||
|
||||
def _build_book(self, args: Namespace) -> int:
|
||||
"""Build the Jupyter Book locally."""
|
||||
|
||||
Reference in New Issue
Block a user