mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-05 16:17:42 -05:00
6491a7512e
- Removed temporary test files and audit reports - Deleted backup and temp_holding directories - Reorganized module structure (07->09 spatial, 09->07 dataloader) - Added new modules: 11-14 (tokenization, embeddings, attention, transformers) - Updated examples with historical ML milestones - Cleaned up documentation structure
770 lines
33 KiB
Python
770 lines
33 KiB
Python
# ---
|
|
# jupyter:
|
|
# jupytext:
|
|
# text_representation:
|
|
# extension: .py
|
|
# format_name: percent
|
|
# format_version: '1.3'
|
|
# jupytext_version: 1.17.1
|
|
# ---
|
|
|
|
# %% [markdown]
|
|
"""
|
|
# Activations - Essential Nonlinearity Functions
|
|
|
|
Welcome to the streamlined Activations module! You'll implement the two most important activation functions in modern deep learning: ReLU and Softmax.
|
|
|
|
## Learning Goals
|
|
- Systems understanding: Why ReLU became the dominant activation and how Softmax enables classification
|
|
- Core implementation skill: Build the two activation functions that power 90%+ of modern architectures
|
|
- Pattern recognition: Understand when to use ReLU (hidden layers) vs Softmax (output layers)
|
|
- Framework connection: See how your implementations match PyTorch's essential activations
|
|
- Performance insight: Learn why ReLU is computationally efficient and Softmax requires careful numerical stability
|
|
|
|
## Build → Use → Reflect
|
|
1. **Build**: ReLU and Softmax activation functions with proper numerical stability
|
|
2. **Use**: Apply these activations in realistic neural network scenarios
|
|
3. **Reflect**: Why did ReLU revolutionize deep learning, and why is Softmax essential for classification?
|
|
|
|
## What You'll Achieve
|
|
By the end of this module, you'll understand:
|
|
- Deep technical understanding of the two activation functions that enable modern deep learning
|
|
- Practical capability to implement numerically stable activations used in production systems
|
|
- Systems insight into why activation choice determines training success and computational efficiency
|
|
- Performance consideration of how ReLU's simplicity and Softmax's complexity affect system design
|
|
- Connection to production ML systems and the design decisions behind activation function choice
|
|
|
|
## Why Only ReLU and Softmax?
|
|
|
|
In this educational framework, we focus on the two most important activation functions:
|
|
|
|
### ReLU (Rectified Linear Unit)
|
|
- **Most widely used** in hidden layers (90%+ of architectures)
|
|
- **Computationally efficient**: Just max(0, x)
|
|
- **Solves vanishing gradients**: Doesn't saturate for positive values
|
|
- **Enables deep networks**: Critical breakthrough for training very deep networks
|
|
|
|
### Softmax
|
|
- **Essential for classification**: Converts logits to probabilities
|
|
- **Attention mechanisms**: Used in transformers and attention-based models
|
|
- **Output layer standard**: Multi-class classification standard
|
|
|
|
### Educational Focus
|
|
- **Master the fundamentals**: Deep understanding of essential functions
|
|
- **Real-world relevance**: These two handle the majority of practical use cases
|
|
- **System insight**: Understand why these became dominant
|
|
- **Foundation building**: Understanding these gives you the foundation for any activation
|
|
|
|
## Systems Reality Check
|
|
💡 **Production Context**: PyTorch implements ReLU with highly optimized CUDA kernels, while Softmax requires careful numerical stability - your implementation reveals these design decisions
|
|
⚡ **Performance Note**: ReLU is popular partly because it's computationally cheap (just max(0,x)), while Softmax requires expensive exponentials and normalization
|
|
"""
|
|
|
|
# %% nbgrader={"grade": false, "grade_id": "activations-imports", "locked": false, "schema_version": 3, "solution": false, "task": false}
|
|
#| default_exp core.activations
|
|
|
|
#| export
|
|
import math
|
|
import numpy as np
|
|
import os
|
|
import sys
|
|
from typing import Union, List
|
|
|
|
# Import our tensor foundation
|
|
try:
|
|
from tinytorch.core.tensor import Tensor
|
|
except ImportError:
|
|
# For development - import from local modules
|
|
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '02_tensor'))
|
|
from tensor_dev import Tensor
|
|
|
|
# %% nbgrader={"grade": false, "grade_id": "activations-setup", "locked": false, "schema_version": 3, "solution": false, "task": false}
|
|
print("🔥 TinyTorch Activations Module")
|
|
print(f"NumPy version: {np.__version__}")
|
|
print(f"Python version: {sys.version_info.major}.{sys.version_info.minor}")
|
|
print("Ready to build essential activation functions!")
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## ReLU - The Breakthrough Activation
|
|
|
|
### What is ReLU?
|
|
**ReLU (Rectified Linear Unit)** is the simplest possible nonlinear activation:
|
|
|
|
```
|
|
f(x) = max(0, x)
|
|
```
|
|
|
|
### Why ReLU Revolutionized Deep Learning
|
|
1. **Computationally efficient**: No expensive exponentials or divisions
|
|
2. **Solves vanishing gradients**: Gradient is 1 for positive inputs, 0 for negative
|
|
3. **Sparse activation**: Naturally creates sparse representations (many zeros)
|
|
4. **Deep network enabler**: Made training networks with 100+ layers possible
|
|
|
|
### Visual Understanding
|
|
```
|
|
Input: [-2, -1, 0, 1, 2]
|
|
ReLU: [0, 0, 0, 1, 2]
|
|
```
|
|
|
|
### Real-World Impact
|
|
- **Computer Vision**: Enabled deep CNNs (AlexNet, ResNet, etc.)
|
|
- **NLP**: Powers transformer hidden layers
|
|
- **Training Speed**: 6x faster than sigmoid in many cases
|
|
- **Hardware**: Optimized in every GPU and AI accelerator
|
|
|
|
### Mathematical Properties
|
|
- **Range**: [0, ∞)
|
|
- **Derivative**: f'(x) = 1 if x > 0, else 0
|
|
- **Dead neurons**: Neurons can "die" if they always output 0
|
|
- **Sparsity**: Naturally creates sparse activations
|
|
"""
|
|
|
|
# %% nbgrader={"grade": false, "grade_id": "relu-class", "locked": false, "schema_version": 3, "solution": true, "task": false}
|
|
#| export
|
|
class ReLU:
|
|
"""
|
|
ReLU Activation Function: f(x) = max(0, x)
|
|
|
|
The most important activation function in modern deep learning.
|
|
Computationally efficient and enables training very deep networks.
|
|
"""
|
|
|
|
def forward(self, x):
|
|
"""
|
|
Apply ReLU activation: f(x) = max(0, x)
|
|
|
|
STEP-BY-STEP IMPLEMENTATION:
|
|
1. Use numpy maximum function to compute max(0, x)
|
|
2. Return new Tensor with ReLU applied
|
|
|
|
MATHEMATICAL FOUNDATION:
|
|
- Forward: f(x) = max(0, x)
|
|
- Sets all negative values to 0, keeps positive values unchanged
|
|
|
|
EXAMPLE USAGE:
|
|
```python
|
|
relu = ReLU()
|
|
tensor_input = Tensor([[-1.0, 0.0, 1.0]])
|
|
tensor_output = relu(tensor_input) # [[0.0, 0.0, 1.0]]
|
|
```
|
|
|
|
IMPLEMENTATION HINTS:
|
|
- Use np.maximum(0, x.data) for element-wise max
|
|
- Create new Tensor from result
|
|
|
|
LEARNING CONNECTIONS:
|
|
- This is the core of torch.nn.ReLU() in PyTorch
|
|
- Used in 90%+ of hidden layers in modern architectures
|
|
- Enables training very deep networks
|
|
- Computationally efficient: just a comparison and selection
|
|
"""
|
|
### BEGIN SOLUTION
|
|
result = np.maximum(0, x.data)
|
|
return Tensor(result)
|
|
### END SOLUTION
|
|
|
|
def forward_(self, x):
|
|
"""
|
|
Apply ReLU activation in-place: modifies input tensor directly
|
|
|
|
In-place ReLU saves memory by reusing existing tensor buffer.
|
|
|
|
STEP-BY-STEP IMPLEMENTATION:
|
|
1. Apply ReLU directly to tensor._data
|
|
2. Return the same tensor object (modified in-place)
|
|
|
|
MEMORY BENEFITS:
|
|
- No new tensor allocation
|
|
- Critical for large networks and limited memory
|
|
- Used in PyTorch with relu_() syntax
|
|
|
|
IMPLEMENTATION HINTS:
|
|
- Use np.maximum(0, x._data, out=x._data) for in-place operation
|
|
"""
|
|
### BEGIN SOLUTION
|
|
np.maximum(0, x._data, out=x._data)
|
|
return x
|
|
### END SOLUTION
|
|
|
|
def __call__(self, x):
|
|
"""Make the class callable: relu(x) instead of relu.forward(x)"""
|
|
return self.forward(x)
|
|
|
|
# %% [markdown]
|
|
"""
|
|
### 🧪 Test Your ReLU Implementation
|
|
|
|
Let's test your ReLU implementation immediately:
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "test-relu-immediate", "locked": true, "points": 10, "schema_version": 3, "solution": false, "task": false}
|
|
def test_unit_relu_activation():
|
|
"""Unit test for the ReLU activation function."""
|
|
print("🔬 Unit Test: ReLU Activation...")
|
|
|
|
# Create ReLU instance
|
|
relu = ReLU()
|
|
|
|
# Test with mixed positive/negative values
|
|
test_input = Tensor([[-2, -1, 0, 1, 2]])
|
|
result = relu(test_input)
|
|
expected = np.array([[0, 0, 0, 1, 2]])
|
|
|
|
assert np.array_equal(result.data, expected), f"ReLU failed: expected {expected}, got {result.data}"
|
|
|
|
# Test with all negative values
|
|
negative_input = Tensor([[-5, -3, -1]])
|
|
negative_result = relu(negative_input)
|
|
expected_negative = np.array([[0, 0, 0]])
|
|
|
|
assert np.array_equal(negative_result.data, expected_negative), "ReLU should zero out negative values"
|
|
|
|
# Test with all positive values (should be unchanged)
|
|
positive_input = Tensor([[1, 3, 5]])
|
|
positive_result = relu(positive_input)
|
|
|
|
assert np.array_equal(positive_result.data, positive_input.data), "ReLU should preserve positive values"
|
|
|
|
# Test with 2D tensor
|
|
matrix_input = Tensor([[-1, 2], [3, -4]])
|
|
matrix_result = relu(matrix_input)
|
|
expected_matrix = np.array([[0, 2], [3, 0]])
|
|
|
|
assert np.array_equal(matrix_result.data, expected_matrix), "ReLU should work with 2D tensors"
|
|
assert matrix_result.shape == matrix_input.shape, "ReLU should preserve shape"
|
|
|
|
# Test in-place operation
|
|
inplace_input = Tensor([[-1, 0, 1]])
|
|
original_data = inplace_input.data.copy()
|
|
relu.forward_(inplace_input)
|
|
expected_inplace = np.array([[0, 0, 1]])
|
|
|
|
assert np.array_equal(inplace_input.data, expected_inplace), "In-place ReLU should modify original tensor"
|
|
|
|
print("✅ ReLU activation tests passed!")
|
|
print(f"✅ Correctly zeros out negative values")
|
|
print(f"✅ Preserves positive values")
|
|
print(f"✅ Shape preservation working")
|
|
print(f"✅ In-place operation working")
|
|
|
|
# Test function defined (called in main block)
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## Softmax - Probability Distribution Creator
|
|
|
|
### What is Softmax?
|
|
**Softmax** converts any real-valued vector into a probability distribution:
|
|
|
|
```
|
|
f(x_i) = e^(x_i) / Σ(e^(x_j))
|
|
```
|
|
|
|
### Why Softmax is Essential
|
|
1. **Probability interpretation**: Outputs sum to 1 and are all positive
|
|
2. **Classification**: Standard for multi-class classification output layers
|
|
3. **Attention mechanisms**: Core component of transformer attention
|
|
4. **Differentiable**: Smooth gradients for optimization
|
|
|
|
### Visual Understanding
|
|
```
|
|
Input: [1.0, 2.0, 3.0]
|
|
Softmax: [0.09, 0.24, 0.67] # Probabilities that sum to 1
|
|
```
|
|
|
|
### Real-World Applications
|
|
- **Classification**: Convert logits to class probabilities
|
|
- **Attention**: Transformer attention weights
|
|
- **Language modeling**: Next token prediction probabilities
|
|
- **Reinforcement learning**: Action probability distributions
|
|
|
|
### Numerical Stability Challenge
|
|
Raw softmax can overflow with large inputs. The solution:
|
|
```
|
|
f(x_i) = e^(x_i - max(x)) / Σ(e^(x_j - max(x)))
|
|
```
|
|
"""
|
|
|
|
# %% nbgrader={"grade": false, "grade_id": "softmax-class", "locked": false, "schema_version": 3, "solution": true, "task": false}
|
|
#| export
|
|
class Softmax:
|
|
"""
|
|
Softmax Activation Function: f(x_i) = e^(x_i) / Σ(e^(x_j))
|
|
|
|
Converts logits to probability distributions.
|
|
Essential for classification and attention mechanisms.
|
|
"""
|
|
|
|
def __init__(self, dim=-1):
|
|
"""
|
|
Initialize Softmax with specified dimension.
|
|
|
|
Args:
|
|
dim: Dimension along which to apply softmax (default: -1, last dimension)
|
|
"""
|
|
self.dim = dim
|
|
|
|
def forward(self, x):
|
|
"""
|
|
Apply Softmax activation with numerical stability.
|
|
|
|
STEP-BY-STEP IMPLEMENTATION:
|
|
1. Subtract max value for numerical stability: x_stable = x - max(x)
|
|
2. Compute exponentials: exp_vals = exp(x_stable)
|
|
3. Compute sum of exponentials: sum_exp = sum(exp_vals)
|
|
4. Divide: softmax = exp_vals / sum_exp
|
|
|
|
MATHEMATICAL FOUNDATION:
|
|
- Forward: f(x_i) = e^(x_i - max(x)) / Σ(e^(x_j - max(x)))
|
|
- Numerically stable version prevents overflow
|
|
- Output is a probability distribution (sums to 1)
|
|
|
|
EXAMPLE USAGE:
|
|
```python
|
|
softmax = Softmax()
|
|
tensor_input = Tensor([[1.0, 2.0, 3.0]])
|
|
tensor_output = softmax(tensor_input) # [[0.09, 0.24, 0.67]]
|
|
```
|
|
|
|
IMPLEMENTATION HINTS:
|
|
- Use np.max(x.data, axis=self.dim, keepdims=True) for stability
|
|
- Use np.exp() for exponentials
|
|
- Use np.sum() with same axis for normalization
|
|
|
|
LEARNING CONNECTIONS:
|
|
- This is the core of torch.nn.Softmax() in PyTorch
|
|
- Used in classification output layers
|
|
- Critical component of attention mechanisms
|
|
- Requires careful numerical implementation
|
|
"""
|
|
### BEGIN SOLUTION
|
|
# Numerical stability: subtract max value
|
|
max_vals = np.max(x.data, axis=self.dim, keepdims=True)
|
|
x_stable = x.data - max_vals
|
|
|
|
# Compute exponentials
|
|
exp_vals = np.exp(x_stable)
|
|
|
|
# Compute softmax
|
|
sum_exp = np.sum(exp_vals, axis=self.dim, keepdims=True)
|
|
result = exp_vals / sum_exp
|
|
|
|
return Tensor(result)
|
|
### END SOLUTION
|
|
|
|
def __call__(self, x):
|
|
"""Make the class callable: softmax(x) instead of softmax.forward(x)"""
|
|
return self.forward(x)
|
|
|
|
# %% [markdown]
|
|
"""
|
|
### 🧪 Test Your Softmax Implementation
|
|
|
|
Let's test your Softmax implementation immediately:
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "test-softmax-immediate", "locked": true, "points": 10, "schema_version": 3, "solution": false, "task": false}
|
|
def test_unit_softmax_activation():
|
|
"""Unit test for the Softmax activation function."""
|
|
print("🔬 Unit Test: Softmax Activation...")
|
|
|
|
# Create Softmax instance
|
|
softmax = Softmax()
|
|
|
|
# Test with simple values
|
|
test_input = Tensor([[1.0, 2.0, 3.0]])
|
|
result = softmax(test_input)
|
|
|
|
# Check that outputs sum to 1 (probability distribution)
|
|
sum_result = np.sum(result.data, axis=-1)
|
|
assert np.allclose(sum_result, 1.0), f"Softmax should sum to 1, got {sum_result}"
|
|
|
|
# Check that all values are positive
|
|
assert np.all(result.data >= 0), "Softmax outputs should be non-negative"
|
|
|
|
# Test with zero input
|
|
zero_input = Tensor([[0.0, 0.0, 0.0]])
|
|
zero_result = softmax(zero_input)
|
|
expected_uniform = np.array([[1/3, 1/3, 1/3]])
|
|
|
|
assert np.allclose(zero_result.data, expected_uniform, atol=1e-6), "Equal inputs should give uniform distribution"
|
|
|
|
# Test numerical stability with large values
|
|
large_input = Tensor([[1000.0, 1001.0, 1002.0]])
|
|
large_result = softmax(large_input)
|
|
|
|
# Should not produce NaN or Inf
|
|
assert not np.any(np.isnan(large_result.data)), "Softmax should handle large values without NaN"
|
|
assert not np.any(np.isinf(large_result.data)), "Softmax should handle large values without Inf"
|
|
assert np.allclose(np.sum(large_result.data, axis=-1), 1.0), "Large value softmax should still sum to 1"
|
|
|
|
# Test with 2D tensor (batch processing)
|
|
batch_input = Tensor([[1.0, 2.0], [3.0, 4.0]])
|
|
batch_result = softmax(batch_input)
|
|
|
|
# Each row should sum to 1
|
|
row_sums = np.sum(batch_result.data, axis=-1)
|
|
assert np.allclose(row_sums, [1.0, 1.0]), "Each batch item should sum to 1"
|
|
|
|
# Test shape preservation
|
|
assert batch_result.shape == batch_input.shape, "Softmax should preserve shape"
|
|
|
|
print("✅ Softmax activation tests passed!")
|
|
print(f"✅ Outputs form valid probability distributions (sum to 1)")
|
|
print(f"✅ All outputs are non-negative")
|
|
print(f"✅ Numerically stable with large inputs")
|
|
print(f"✅ Batch processing works correctly")
|
|
print(f"✅ Shape preservation working")
|
|
|
|
# Test function defined (called in main block)
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## Comprehensive Testing
|
|
|
|
Let's run comprehensive tests that validate both activations working together:
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "test-activations-comprehensive", "locked": true, "points": 15, "schema_version": 3, "solution": false, "task": false}
|
|
def test_unit_activations_comprehensive():
|
|
"""Comprehensive test of both activation functions."""
|
|
print("🔬 Comprehensive Test: ReLU + Softmax Pipeline...")
|
|
|
|
# Create activation instances
|
|
relu = ReLU()
|
|
softmax = Softmax()
|
|
|
|
# Test realistic neural network scenario
|
|
# Simulate a network layer output (could be negative)
|
|
layer_output = Tensor([[-2.0, -1.0, 0.0, 1.0, 2.0]])
|
|
|
|
# Apply ReLU (hidden layer activation)
|
|
hidden_activation = relu(layer_output)
|
|
expected_relu = np.array([[0.0, 0.0, 0.0, 1.0, 2.0]])
|
|
|
|
assert np.array_equal(hidden_activation.data, expected_relu), "ReLU should zero negatives"
|
|
|
|
# Apply Softmax to different tensor (classification output)
|
|
logits = Tensor([[2.0, 1.0, 0.1]])
|
|
class_probabilities = softmax(logits)
|
|
|
|
# Verify probability properties
|
|
assert np.allclose(np.sum(class_probabilities.data, axis=-1), 1.0), "Softmax should create probability distribution"
|
|
assert np.all(class_probabilities.data >= 0), "Probabilities should be non-negative"
|
|
|
|
# Test that highest logit gets highest probability
|
|
max_logit_idx = np.argmax(logits.data)
|
|
max_prob_idx = np.argmax(class_probabilities.data)
|
|
assert max_logit_idx == max_prob_idx, "Highest logit should get highest probability"
|
|
|
|
# Test with batch data (realistic scenario)
|
|
batch_logits = Tensor([
|
|
[1.0, 2.0, 0.5], # Batch item 1
|
|
[0.1, 0.2, 0.9], # Batch item 2
|
|
[2.0, 1.0, 1.5] # Batch item 3
|
|
])
|
|
|
|
batch_probs = softmax(batch_logits)
|
|
|
|
# Each row should sum to 1
|
|
row_sums = np.sum(batch_probs.data, axis=1)
|
|
assert np.allclose(row_sums, [1.0, 1.0, 1.0]), "Each batch item should form probability distribution"
|
|
|
|
print("✅ Comprehensive activation tests passed!")
|
|
print(f"✅ ReLU correctly processes hidden layer outputs")
|
|
print(f"✅ Softmax correctly creates probability distributions")
|
|
print(f"✅ Batch processing works for realistic scenarios")
|
|
print(f"✅ Activations preserve expected mathematical properties")
|
|
|
|
# Test function defined (called in main block)
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## Integration Test: Real Neural Network Scenario
|
|
|
|
Let's test these activations in a realistic neural network context:
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "test-activations-integration", "locked": true, "points": 10, "schema_version": 3, "solution": false, "task": false}
|
|
def test_module_activation_integration():
|
|
"""Integration test: activations in a realistic neural network pipeline."""
|
|
print("🔬 Integration Test: Neural Network Pipeline...")
|
|
|
|
# Simulate a complete forward pass through a small network
|
|
relu = ReLU()
|
|
softmax = Softmax()
|
|
|
|
# Step 1: Input data (batch of 3 samples, 4 features each)
|
|
input_data = Tensor([
|
|
[0.5, -0.3, 1.2, -0.8], # Sample 1
|
|
[-1.0, 0.8, 0.0, 1.5], # Sample 2
|
|
[0.2, -0.5, -0.9, 0.3] # Sample 3
|
|
])
|
|
|
|
# Step 2: Simulate hidden layer output (after linear transformation)
|
|
# In real network this would be: input @ weights + bias
|
|
hidden_output = Tensor([
|
|
[-1.5, 0.8, 2.1], # Sample 1 hidden activations
|
|
[0.3, -0.6, 1.2], # Sample 2 hidden activations
|
|
[-0.8, 1.5, -0.3] # Sample 3 hidden activations
|
|
])
|
|
|
|
# Step 3: Apply ReLU to hidden layer
|
|
hidden_activated = relu(hidden_output)
|
|
|
|
# Verify ReLU behavior
|
|
expected_relu = np.array([
|
|
[0.0, 0.8, 2.1],
|
|
[0.3, 0.0, 1.2],
|
|
[0.0, 1.5, 0.0]
|
|
])
|
|
assert np.allclose(hidden_activated.data, expected_relu), "ReLU should zero negatives in hidden layer"
|
|
|
|
# Step 4: Simulate final layer output (logits for 3 classes)
|
|
final_logits = Tensor([
|
|
[2.1, 0.5, 1.2], # Sample 1 class scores
|
|
[0.8, 1.5, 0.3], # Sample 2 class scores
|
|
[1.0, 2.0, 0.1] # Sample 3 class scores
|
|
])
|
|
|
|
# Step 5: Apply Softmax for classification
|
|
class_probabilities = softmax(final_logits)
|
|
|
|
# Verify softmax properties
|
|
batch_sums = np.sum(class_probabilities.data, axis=1)
|
|
assert np.allclose(batch_sums, [1.0, 1.0, 1.0]), "Each sample should have probabilities summing to 1"
|
|
|
|
# Verify predictions make sense (highest logit -> highest probability)
|
|
for i in range(3):
|
|
max_logit_class = np.argmax(final_logits.data[i])
|
|
max_prob_class = np.argmax(class_probabilities.data[i])
|
|
assert max_logit_class == max_prob_class, f"Sample {i}: highest logit should get highest probability"
|
|
|
|
# Test memory efficiency (shapes preserved)
|
|
assert hidden_activated.shape == hidden_output.shape, "ReLU should preserve tensor shape"
|
|
assert class_probabilities.shape == final_logits.shape, "Softmax should preserve tensor shape"
|
|
|
|
print("✅ Integration test passed!")
|
|
print(f"✅ Complete forward pass simulation successful")
|
|
print(f"✅ ReLU enables nonlinear hidden representations")
|
|
print(f"✅ Softmax provides interpretable classification outputs")
|
|
print(f"✅ Batch processing works throughout pipeline")
|
|
|
|
# Display sample predictions
|
|
print(f"\n📊 Sample Predictions:")
|
|
for i in range(3):
|
|
probs = class_probabilities.data[i]
|
|
predicted_class = np.argmax(probs)
|
|
confidence = probs[predicted_class]
|
|
print(f" Sample {i+1}: Class {predicted_class} (confidence: {confidence:.3f})")
|
|
|
|
# Test function defined (called in main block)
|
|
|
|
# Main execution block
|
|
if __name__ == "__main__":
|
|
# Run all activation tests
|
|
test_unit_relu_activation()
|
|
test_unit_softmax_activation()
|
|
test_unit_activations_comprehensive()
|
|
test_module_activation_integration()
|
|
|
|
print("\n🎉 All activation tests passed!")
|
|
print("✅ ReLU: The foundation of modern deep learning")
|
|
print("✅ Softmax: The key to interpretable classifications")
|
|
print("💡 Ready to build neural networks with essential nonlinearity!")
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## 🤔 ML Systems Thinking: Interactive Questions
|
|
|
|
Now that you've built the essential activation functions, let's connect this work to broader ML systems challenges. These questions help you think critically about how activation choices scale to production ML environments.
|
|
|
|
### Question 1: Performance and Hardware Optimization
|
|
|
|
**Context**: Your ReLU implementation uses a simple `np.maximum(0, x)` operation, while Softmax requires exponentials and division. In production ML systems, activation functions are called billions of times during training and inference.
|
|
|
|
**Reflection Question**: Design a performance optimization strategy for activation functions in a production ML framework. How would you optimize ReLU and Softmax differently for CPU vs GPU execution? Consider the trade-offs between memory bandwidth, computational complexity, and numerical precision. What specific optimizations would you implement for training vs inference scenarios?
|
|
|
|
Think about: SIMD vectorization, kernel fusion, memory layout optimization, and precision requirements across different hardware architectures.
|
|
|
|
*Target length: 150-300 words*
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "ml-systems-performance", "locked": false, "points": 10, "schema_version": 3, "solution": true, "task": false}
|
|
"""
|
|
YOUR REFLECTION ON PERFORMANCE AND HARDWARE OPTIMIZATION:
|
|
|
|
TODO: Replace this text with your thoughtful response about activation function optimization.
|
|
|
|
Consider addressing:
|
|
- How would you optimize ReLU vs Softmax differently for various hardware platforms?
|
|
- What role does memory bandwidth vs computational complexity play in optimization decisions?
|
|
- How would you handle precision trade-offs between training and inference?
|
|
- What specific CUDA kernel optimizations would benefit each activation?
|
|
- How would you design kernel fusion strategies to minimize memory traffic?
|
|
|
|
Write a technical analysis connecting your implementations to real performance optimization challenges.
|
|
|
|
GRADING RUBRIC (Instructor Use):
|
|
- Demonstrates understanding of hardware-specific optimization strategies (3 points)
|
|
- Addresses CPU vs GPU optimization differences appropriately (3 points)
|
|
- Shows practical knowledge of memory bandwidth and computational trade-offs (2 points)
|
|
- Demonstrates systems thinking about training vs inference requirements (2 points)
|
|
- Clear technical reasoning with performance insights (bonus points for innovative approaches)
|
|
"""
|
|
|
|
### BEGIN SOLUTION
|
|
# Student response area - instructor will replace this section during grading setup
|
|
# This is a manually graded question requiring technical analysis of hardware optimization
|
|
# Students should demonstrate understanding of performance optimization across different platforms
|
|
### END SOLUTION
|
|
|
|
# %% [markdown]
|
|
"""
|
|
### Question 2: Numerical Stability and Production Reliability
|
|
|
|
**Context**: Your Softmax implementation includes numerical stability measures (subtracting max values), but production systems face additional challenges: mixed precision training, gradient underflow, and distributed training synchronization.
|
|
|
|
**Reflection Question**: Architect a numerically stable activation system for a production ML framework that handles edge cases and maintains training stability across different scenarios. How would you handle extreme input values, gradient explosion/vanishing, and precision loss in distributed training? Consider the challenges of maintaining numerical consistency when the same model runs on different hardware with different floating-point behaviors.
|
|
|
|
Think about: numerical precision hierarchies, gradient clipping strategies, hardware-specific floating-point behaviors, and distributed synchronization requirements.
|
|
|
|
*Target length: 150-300 words*
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "ml-systems-stability", "locked": false, "points": 10, "schema_version": 3, "solution": true, "task": false}
|
|
"""
|
|
YOUR REFLECTION ON NUMERICAL STABILITY AND PRODUCTION RELIABILITY:
|
|
|
|
TODO: Replace this text with your thoughtful response about numerical stability design.
|
|
|
|
Consider addressing:
|
|
- How would you design activation functions to handle extreme input values gracefully?
|
|
- What strategies would you use for maintaining numerical consistency across different hardware?
|
|
- How would you integrate gradient clipping and stability measures into activation implementations?
|
|
- What role does mixed precision training play in activation function design?
|
|
- How would you ensure distributed training maintains numerical consistency?
|
|
|
|
Write an architectural analysis connecting your activation implementations to production stability challenges.
|
|
|
|
GRADING RUBRIC (Instructor Use):
|
|
- Shows understanding of numerical stability challenges in production systems (3 points)
|
|
- Addresses hardware-specific floating-point considerations (3 points)
|
|
- Designs practical stability measures for distributed training (2 points)
|
|
- Demonstrates systems thinking about gradient stability and precision (2 points)
|
|
- Clear architectural reasoning with stability insights (bonus points for comprehensive understanding)
|
|
"""
|
|
|
|
### BEGIN SOLUTION
|
|
# Student response area - instructor will replace this section during grading setup
|
|
# This is a manually graded question requiring understanding of numerical stability in production
|
|
# Students should demonstrate knowledge of floating-point challenges and distributed training
|
|
### END SOLUTION
|
|
|
|
# %% [markdown]
|
|
"""
|
|
### Question 3: Activation Function Evolution and System Design
|
|
|
|
**Context**: You implemented ReLU and Softmax, the current standards, but activation functions continue to evolve (GELU, Swish, etc.). Production ML systems must support both established and experimental activations while maintaining backward compatibility and performance.
|
|
|
|
**Reflection Question**: Design an extensible activation function system that can efficiently support both current standards (ReLU, Softmax) and future experimental activations. How would you balance the need for optimal performance of established functions with the flexibility to add new activations? Consider the challenges of maintaining API compatibility, performance benchmarking, and automatic differentiation support across diverse activation functions.
|
|
|
|
Think about: plugin architectures, performance profiling systems, automatic differentiation integration, and backward compatibility strategies.
|
|
|
|
*Target length: 150-300 words*
|
|
"""
|
|
|
|
# %% nbgrader={"grade": true, "grade_id": "ml-systems-evolution", "locked": false, "points": 10, "schema_version": 3, "solution": true, "task": false}
|
|
"""
|
|
YOUR REFLECTION ON ACTIVATION FUNCTION EVOLUTION AND SYSTEM DESIGN:
|
|
|
|
TODO: Replace this text with your thoughtful response about extensible activation system design.
|
|
|
|
Consider addressing:
|
|
- How would you design a plugin architecture for new activation functions?
|
|
- What strategies would you use to maintain performance for established activations while supporting experimentation?
|
|
- How would you handle automatic differentiation for diverse activation types?
|
|
- What role would performance benchmarking and profiling play in your system design?
|
|
- How would you ensure backward compatibility while enabling innovation?
|
|
|
|
Write a system design analysis connecting your activation foundation to framework evolution challenges.
|
|
|
|
GRADING RUBRIC (Instructor Use):
|
|
- Designs practical extensible architecture for activation functions (3 points)
|
|
- Addresses performance vs flexibility trade-offs appropriately (3 points)
|
|
- Shows understanding of automatic differentiation integration challenges (2 points)
|
|
- Demonstrates systems thinking about framework evolution and compatibility (2 points)
|
|
- Clear design reasoning with innovation insights (bonus points for forward-thinking approaches)
|
|
"""
|
|
|
|
### BEGIN SOLUTION
|
|
# Student response area - instructor will replace this section during grading setup
|
|
# This is a manually graded question requiring understanding of extensible system design
|
|
# Students should demonstrate knowledge of framework architecture and evolution challenges
|
|
### END SOLUTION
|
|
|
|
# %% [markdown]
|
|
"""
|
|
## 🎯 MODULE SUMMARY: Essential Activations
|
|
|
|
Congratulations! You've successfully implemented the two most important activation functions in modern deep learning:
|
|
|
|
## What You've Built
|
|
- **ReLU Activation**: The foundation of deep learning that enabled training very deep networks
|
|
- **Softmax Activation**: The probability distribution creator essential for classification
|
|
- **Numerical Stability**: Proper implementation techniques that prevent overflow and underflow
|
|
- **Performance Awareness**: Understanding of computational trade-offs between different activations
|
|
- **Production Insight**: Connection to real-world optimization and stability challenges
|
|
|
|
## Key Learning Outcomes
|
|
- **Understanding**: Why these two activations dominate modern architectures
|
|
- **Implementation**: Built numerically stable activation functions from scratch
|
|
- **Systems thinking**: Connecting computational efficiency to architecture design decisions
|
|
- **Real-world connection**: Understanding how activation choice affects system performance
|
|
- **Foundation building**: Prepared for implementing any activation function
|
|
|
|
## Mathematical Foundations Mastered
|
|
- **ReLU Mathematics**: f(x) = max(0, x) and its gradient properties
|
|
- **Softmax Mathematics**: Numerically stable probability distribution computation
|
|
- **Gradient Flow**: How different activations affect training dynamics
|
|
- **Numerical Stability**: Techniques for preventing overflow and maintaining precision
|
|
|
|
## Professional Skills Developed
|
|
- **Performance Analysis**: Understanding computational complexity of different activations
|
|
- **Numerical Programming**: Implementing mathematically stable algorithms
|
|
- **System Design**: Considering hardware and performance implications
|
|
- **Error Handling**: Graceful handling of edge cases and extreme values
|
|
|
|
## Ready for Advanced Applications
|
|
Your activation implementations now enable:
|
|
- **Hidden Layer Processing**: ReLU for nonlinear transformations
|
|
- **Classification**: Softmax for probability-based outputs
|
|
- **Attention Mechanisms**: Softmax for attention weight computation
|
|
- **Deep Networks**: ReLU enabling training of very deep architectures
|
|
|
|
## Connection to Real ML Systems
|
|
Your implementations mirror production systems:
|
|
- **PyTorch**: `torch.nn.ReLU()` and `torch.nn.Softmax()` implement identical mathematics
|
|
- **TensorFlow**: `tf.nn.relu()` and `tf.nn.softmax()` follow the same principles
|
|
- **Hardware Acceleration**: Modern GPUs have specialized kernels for these exact operations
|
|
- **Industry Standard**: Every major ML framework optimizes these specific activations
|
|
|
|
## The Power of Strategic Simplicity
|
|
You've learned that effective systems focus on essentials:
|
|
- **ReLU's Simplicity**: Revolutionary because it's computationally trivial yet mathematically powerful
|
|
- **Softmax's Precision**: Complex implementation required for mathematically correct probability distributions
|
|
- **Strategic Focus**: Understanding 2 essential functions deeply vs 10 functions superficially
|
|
- **Real-World Impact**: These functions power 90%+ of production deep learning systems
|
|
|
|
## What's Next
|
|
Your activation implementations are the foundation for:
|
|
- **Layers**: Building neural network components that use these activations
|
|
- **Networks**: Composing layers with appropriate activations for different tasks
|
|
- **Training**: Optimizing networks where activation choice determines success
|
|
- **Advanced Architectures**: Modern systems that depend on these fundamental building blocks
|
|
|
|
**Next Module**: Layers - building the neural network components that combine linear transformations with your activations!
|
|
|
|
You've built the nonlinear intelligence that makes neural networks powerful. Now let's combine these activations with linear transformations to create the building blocks of any neural architecture!
|
|
""" |