mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-08-02 22:07:52 -05:00
Update Module 03 Layers with professional template
- Add Foundation Tier badge and complete metadata - Reduce emoji usage for professional tone - Add Xavier initialization explanation - Add systems thinking questions - Add parameter management details - Link to next module (Losses)
This commit is contained in:
Vijay Janapa Reddi
+281
-182
@@ -1,235 +1,334 @@
|
||||
---
|
||||
title: "Layers"
|
||||
description: "Neural network layers (Linear, activation layers)"
|
||||
difficulty: "⭐⭐"
|
||||
description: "Neural network layers (Linear, Sequential)"
|
||||
module_number: 3
|
||||
tier: "foundation"
|
||||
difficulty: "intermediate"
|
||||
time_estimate: "4-5 hours"
|
||||
prerequisites: []
|
||||
next_steps: []
|
||||
learning_objectives: []
|
||||
prerequisites: ["01. Tensor", "02. Activations"]
|
||||
next_module: "04. Losses"
|
||||
learning_objectives:
|
||||
- "Understand layers as composable functions that transform tensor representations"
|
||||
- "Implement Linear layers with weight matrices and bias vectors (y = Wx + b)"
|
||||
- "Build Sequential containers for composing multiple layers into networks"
|
||||
- "Recognize parameter initialization strategies and their impact on training"
|
||||
- "Analyze memory usage and computational complexity of layer operations"
|
||||
---
|
||||
|
||||
# Module: Layers
|
||||
# 03. Layers
|
||||
|
||||
```{div} badges
|
||||
⭐⭐ | ⏱️ 4-5 hours
|
||||
```
|
||||
<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 0.5rem 1.5rem; border-radius: 0.5rem; display: inline-block; margin-bottom: 2rem; font-weight: 600;">
|
||||
Foundation Tier | Module 03 of 20
|
||||
</div>
|
||||
|
||||
**Build composable building blocks - Linear layers and Sequential containers.**
|
||||
|
||||
## 📊 Module Info
|
||||
- **Difficulty**: ⭐⭐ Intermediate
|
||||
- **Time Estimate**: 4-5 hours
|
||||
- **Prerequisites**: Tensor, Activations modules
|
||||
- **Next Steps**: Networks module
|
||||
Difficulty: Intermediate | Time: 4-5 hours | Prerequisites: Modules 01-02
|
||||
|
||||
Build the fundamental transformations that compose into neural networks. This module teaches you that layers are simply functions that transform tensors, and neural networks are just sophisticated function composition using these building blocks.
|
||||
---
|
||||
|
||||
## 🎯 Learning Objectives
|
||||
## What You'll Build
|
||||
|
||||
By the end of this module, you will be able to:
|
||||
Layers are the building blocks of neural networks. Each layer is a function that transforms tensors, and networks are compositions of these functions.
|
||||
|
||||
- **Understand layers as mathematical functions**: Recognize that layers transform tensors through well-defined mathematical operations
|
||||
- **Implement Dense layers**: Build linear transformations using matrix multiplication and bias addition (`y = Wx + b`)
|
||||
- **Integrate activation functions**: Combine linear layers with nonlinear activations to enable complex pattern learning
|
||||
- **Compose simple building blocks**: Chain layers together to create complete neural network architectures
|
||||
- **Debug layer implementations**: Use shape analysis and mathematical properties to verify correct implementation
|
||||
By the end of this module, you'll have implemented:
|
||||
|
||||
## 🧠 Build → Use → Reflect
|
||||
- **Linear Layer** - Affine transformation `y = Wx + b` (also called Dense or Fully Connected)
|
||||
- **Sequential Container** - Chains multiple layers into a network
|
||||
- **Parameter Management** - Weight initialization and storage
|
||||
|
||||
This module follows TinyTorch's **Build → Use → Reflect** framework:
|
||||
### Example Usage
|
||||
|
||||
1. **Build**: Implement Dense layers and activation functions from mathematical foundations
|
||||
2. **Use**: Transform tensors through layer operations and see immediate results in various scenarios
|
||||
3. **Reflect**: Understand how simple layers compose into complex neural networks and why architecture matters
|
||||
|
||||
## 📚 What You'll Build
|
||||
|
||||
### Core Layer Implementation
|
||||
```python
|
||||
# Dense layer: fundamental building block
|
||||
layer = Dense(input_size=3, output_size=2)
|
||||
x = Tensor([[1.0, 2.0, 3.0]])
|
||||
y = layer(x) # Shape transformation: (1, 3) → (1, 2)
|
||||
from tinytorch.core.layers import Linear, Sequential
|
||||
from tinytorch.core.activations import ReLU
|
||||
from tinytorch.core.tensor import Tensor
|
||||
|
||||
# With activation functions
|
||||
relu = ReLU()
|
||||
activated = relu(y) # Apply nonlinearity
|
||||
# Single linear layer
|
||||
layer = Linear(in_features=784, out_features=128)
|
||||
x = Tensor(np.random.randn(32, 784)) # Batch of 32 images
|
||||
output = layer.forward(x) # Shape: (32, 128)
|
||||
|
||||
# Chaining operations
|
||||
layer1 = Dense(784, 128) # Image → hidden
|
||||
layer2 = Dense(128, 10) # Hidden → classes
|
||||
activation = ReLU()
|
||||
# Multi-layer network
|
||||
model = Sequential([
|
||||
Linear(784, 128),
|
||||
ReLU(),
|
||||
Linear(128, 64),
|
||||
ReLU(),
|
||||
Linear(64, 10)
|
||||
])
|
||||
|
||||
# Forward pass composition
|
||||
x = Tensor([[1.0, 2.0, 3.0, ...]]) # Input data
|
||||
h1 = activation(layer1(x)) # First transformation
|
||||
output = layer2(h1) # Final prediction
|
||||
logits = model.forward(x) # Shape: (32, 10)
|
||||
```
|
||||
|
||||
### Dense Layer Implementation
|
||||
- **Mathematical foundation**: Linear transformation `y = Wx + b`
|
||||
- **Weight initialization**: Xavier/Glorot uniform initialization for stable gradients
|
||||
- **Bias handling**: Optional bias terms for translation invariance
|
||||
- **Shape management**: Automatic handling of batch dimensions and matrix operations
|
||||
---
|
||||
|
||||
### Activation Layer Integration
|
||||
- **ReLU integration**: Most common activation for hidden layers
|
||||
- **Sigmoid integration**: Probability outputs for binary classification
|
||||
- **Tanh integration**: Zero-centered outputs for better optimization
|
||||
- **Composition patterns**: Standard ways to combine layers and activations
|
||||
## Learning Pattern: Build → Use → Understand
|
||||
|
||||
## 🚀 Getting Started
|
||||
### 1. Build
|
||||
Implement the Linear layer with weight matrix initialization and forward pass computation. Build a Sequential container to chain layers.
|
||||
|
||||
### Prerequisites
|
||||
Ensure you have completed the foundational modules:
|
||||
### 2. Use
|
||||
Create multi-layer networks, transform data through layers, and observe how representations change across network depth.
|
||||
|
||||
### 3. Understand
|
||||
Grasp how simple layers compose into powerful networks, why parameter initialization matters, and how memory/computation scales with layer size.
|
||||
|
||||
---
|
||||
|
||||
## Learning Objectives
|
||||
|
||||
By completing this module, you will:
|
||||
|
||||
1. **Systems Understanding**: Recognize layers as modular transformations that compose into networks, mirroring Unix philosophy of composable tools
|
||||
|
||||
2. **Core Implementation**: Build Linear layers with proper weight initialization and Sequential containers for network composition
|
||||
|
||||
3. **Pattern Recognition**: Understand the affine transformation pattern (linear + bias) and how it appears everywhere in neural networks
|
||||
|
||||
4. **Framework Connection**: See how your Linear mirrors `torch.nn.Linear` and Sequential mirrors `torch.nn.Sequential`
|
||||
|
||||
5. **Performance Trade-offs**: Analyze computational cost (O(batch × in × out) for Linear) and memory usage (parameters + activations)
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
### Production Context
|
||||
|
||||
Linear layers are ubiquitous in production ML:
|
||||
|
||||
- **MLPs**: Stack of Linear + ReLU layers for tabular data (recommendation systems, fraud detection)
|
||||
- **Vision**: Linear layers in classifier heads (ResNet's final layer: 2048 → 1000 classes)
|
||||
- **NLP**: Linear projections in transformers (attention Q/K/V projections, feedforward layers)
|
||||
- **RL**: Policy/value networks use Linear layers to map states to actions
|
||||
|
||||
Every neural network contains Linear layers. Understanding their implementation is fundamental.
|
||||
|
||||
### Systems Reality Check
|
||||
|
||||
**Performance Note**: A single Linear layer with 1000 × 1000 weights performs 1M multiply-adds per sample. Batch size 32 = 32M operations. Matrix multiplication dominates training time.
|
||||
|
||||
**Memory Note**: Parameters are persistent (stored across batches), but activations are temporary (one batch at a time). A 1000 × 1000 layer uses 4MB for FP32 weights, but activations can be deallocated after backprop.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Guide
|
||||
|
||||
### Prerequisites Check
|
||||
|
||||
```bash
|
||||
# Activate TinyTorch environment
|
||||
source bin/activate-tinytorch.sh
|
||||
|
||||
# Verify prerequisite modules
|
||||
tito test --module tensor
|
||||
tito test --module activations
|
||||
tito test 01 02
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
1. **Open the development file**: `modules/source/04_layers/layers_dev.py`
|
||||
2. **Implement Dense layer class**: Start with `__init__` and `forward` methods
|
||||
3. **Test layer functionality**: Use inline tests for immediate feedback
|
||||
4. **Add activation integration**: Combine layers with activation functions
|
||||
5. **Build complete networks**: Chain multiple layers together
|
||||
6. **Export and verify**: `tito export --module layers && tito test --module layers`
|
||||
|
||||
## 🧪 Testing Your Implementation
|
||||
|
||||
### Comprehensive Test Suite
|
||||
Run the full test suite to verify mathematical correctness:
|
||||
|
||||
```bash
|
||||
# TinyTorch CLI (recommended)
|
||||
tito test --module layers
|
||||
|
||||
# Direct pytest execution
|
||||
python -m pytest tests/ -k layers -v
|
||||
cd modules/source/03_layers/
|
||||
jupyter lab layers_dev.py
|
||||
```
|
||||
|
||||
### Test Coverage Areas
|
||||
- ✅ **Layer Functionality**: Verify Dense layers perform correct linear transformations
|
||||
- ✅ **Weight Initialization**: Ensure proper weight initialization for training stability
|
||||
- ✅ **Shape Preservation**: Confirm layers handle batch dimensions correctly
|
||||
- ✅ **Activation Integration**: Test seamless combination with activation functions
|
||||
- ✅ **Network Composition**: Verify layers can be chained into complete networks
|
||||
### Step-by-Step Build
|
||||
|
||||
#### Step 1: Linear Layer Foundation
|
||||
|
||||
Implement the core affine transformation:
|
||||
|
||||
### Inline Testing & Development
|
||||
The module includes educational feedback during development:
|
||||
```python
|
||||
# Example inline test output
|
||||
🔬 Unit Test: Dense layer functionality...
|
||||
✅ Dense layer computes y = Wx + b correctly
|
||||
✅ Weight initialization within expected range
|
||||
✅ Output shape matches expected dimensions
|
||||
📈 Progress: Dense Layer ✓
|
||||
|
||||
# Integration testing
|
||||
🔬 Unit Test: Layer composition...
|
||||
✅ Multiple layers chain correctly
|
||||
✅ Activations integrate seamlessly
|
||||
📈 Progress: Layer Composition ✓
|
||||
class Linear:
|
||||
def __init__(self, in_features, out_features):
|
||||
"""Initialize linear layer: y = Wx + b"""
|
||||
# Xavier/Glorot initialization
|
||||
limit = np.sqrt(6 / (in_features + out_features))
|
||||
self.weight = Tensor(
|
||||
np.random.uniform(-limit, limit, (in_features, out_features))
|
||||
)
|
||||
self.bias = Tensor(np.zeros(out_features))
|
||||
|
||||
self.in_features = in_features
|
||||
self.out_features = out_features
|
||||
|
||||
def forward(self, x):
|
||||
"""Forward pass: y = xW + b"""
|
||||
return x.matmul(self.weight) + self.bias
|
||||
```
|
||||
|
||||
### Manual Testing Examples
|
||||
**Why Xavier initialization**: Random uniform initialization with variance scaled by layer size prevents vanishing/exploding gradients. Critical for deep network training.
|
||||
|
||||
**Design decision**: Weight shape is `(in, out)` so we compute `x @ W` (not `W @ x`). This matches PyTorch conventions and enables clean batched computation.
|
||||
|
||||
#### Step 2: Parameter Management
|
||||
|
||||
Track parameters for optimization:
|
||||
|
||||
```python
|
||||
from tinytorch.core.tensor import Tensor
|
||||
from layers_dev import Dense
|
||||
from activations_dev import ReLU
|
||||
|
||||
# Test basic layer functionality
|
||||
layer = Dense(input_size=3, output_size=2)
|
||||
x = Tensor([[1.0, 2.0, 3.0]])
|
||||
y = layer(x)
|
||||
print(f"Input shape: {x.shape}, Output shape: {y.shape}")
|
||||
|
||||
# Test layer composition
|
||||
layer1 = Dense(3, 4)
|
||||
layer2 = Dense(4, 2)
|
||||
relu = ReLU()
|
||||
|
||||
# Forward pass
|
||||
h1 = relu(layer1(x))
|
||||
output = layer2(h1)
|
||||
print(f"Final output: {output.data}")
|
||||
def parameters(self):
|
||||
"""Return list of trainable parameters"""
|
||||
return [self.weight, self.bias]
|
||||
```
|
||||
|
||||
## 🎯 Key Concepts
|
||||
**Why this matters**: Optimizers need access to all parameters to update them during training. This method provides that access.
|
||||
|
||||
### Real-World Applications
|
||||
- **Computer Vision**: Dense layers process flattened image features in CNNs (like VGG, ResNet final layers)
|
||||
- **Natural Language Processing**: Dense layers transform word embeddings in transformers and RNNs
|
||||
- **Recommendation Systems**: Dense layers combine user and item features for preference prediction
|
||||
- **Scientific Computing**: Dense layers approximate complex functions in physics simulations and engineering
|
||||
#### Step 3: Sequential Container
|
||||
|
||||
### Mathematical Foundations
|
||||
- **Linear Transformation**: `y = Wx + b` where W is the weight matrix and b is the bias vector
|
||||
- **Matrix Multiplication**: Efficient batch processing through vectorized operations
|
||||
- **Weight Initialization**: Xavier/Glorot initialization prevents vanishing/exploding gradients
|
||||
- **Function Composition**: Networks as nested function calls: `f3(f2(f1(x)))`
|
||||
Chain layers into networks:
|
||||
|
||||
### Neural Network Building Blocks
|
||||
- **Modularity**: Layers as reusable components that can be combined in different ways
|
||||
- **Standardized Interface**: All layers follow the same input/output pattern for easy composition
|
||||
- **Shape Consistency**: Automatic handling of batch dimensions and shape transformations
|
||||
- **Nonlinearity**: Activation functions between layers enable learning of complex patterns
|
||||
|
||||
### Implementation Patterns
|
||||
- **Class-based Design**: Layers as objects with state (weights) and behavior (forward pass)
|
||||
- **Initialization Strategy**: Proper weight initialization for stable training dynamics
|
||||
- **Error Handling**: Graceful handling of shape mismatches and invalid inputs
|
||||
- **Testing Philosophy**: Comprehensive testing of mathematical properties and edge cases
|
||||
|
||||
## 🎉 Ready to Build?
|
||||
|
||||
You're about to build the fundamental building blocks that power every neural network! Dense layers might seem simple, but they're the workhorses of deep learning—from the final layers of image classifiers to the core components of language models.
|
||||
|
||||
Understanding how these simple linear transformations compose into complex intelligence is one of the most beautiful insights in machine learning. Take your time, understand the mathematics, and enjoy building the foundation of artificial intelligence!
|
||||
|
||||
|
||||
|
||||
|
||||
Choose your preferred way to engage with this module:
|
||||
|
||||
````{grid} 1 2 3 3
|
||||
|
||||
```{grid-item-card} 🚀 Launch Binder
|
||||
:link: https://mybinder.org/v2/gh/mlsysbook/TinyTorch/main?filepath=modules/source/04_layers/layers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
|
||||
Run this module interactively in your browser. No installation required!
|
||||
```python
|
||||
class Sequential:
|
||||
def __init__(self, layers):
|
||||
"""Container for sequential layer execution"""
|
||||
self.layers = layers
|
||||
|
||||
def forward(self, x):
|
||||
"""Forward pass through all layers"""
|
||||
for layer in self.layers:
|
||||
x = layer.forward(x)
|
||||
return x
|
||||
|
||||
def parameters(self):
|
||||
"""Collect parameters from all layers"""
|
||||
params = []
|
||||
for layer in self.layers:
|
||||
if hasattr(layer, 'parameters'):
|
||||
params.extend(layer.parameters())
|
||||
return params
|
||||
```
|
||||
|
||||
```{grid-item-card} ⚡ Open in Colab
|
||||
:link: https://colab.research.google.com/github/mlsysbook/TinyTorch/blob/main/modules/source/04_layers/layers_dev.ipynb
|
||||
:class-header: bg-light
|
||||
**Pattern insight**: Sequential treats activation functions and layers uniformly—anything with a `forward()` method works. This composability is powerful.
|
||||
|
||||
Use Google Colab for GPU access and cloud compute power.
|
||||
```
|
||||
#### Step 4: Network Building
|
||||
|
||||
```{grid-item-card} 📖 View Source
|
||||
:link: https://github.com/mlsysbook/TinyTorch/blob/main/modules/source/04_layers/layers_dev.py
|
||||
:class-header: bg-light
|
||||
Combine components:
|
||||
|
||||
Browse the Python source code and understand the implementation.
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
```{admonition} 💾 Save Your Progress
|
||||
:class: tip
|
||||
**Binder sessions are temporary!** Download your completed notebook when done, or switch to local development for persistent work.
|
||||
```python
|
||||
# Two-layer MLP
|
||||
model = Sequential([
|
||||
Linear(784, 256),
|
||||
ReLU(),
|
||||
Linear(256, 10)
|
||||
])
|
||||
|
||||
# Access all parameters
|
||||
all_params = model.parameters() # [W1, b1, W2, b2]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev" href="../chapters/03_activations.html" title="previous page">← Previous Module</a>
|
||||
<a class="right-next" href="../chapters/05_dense.html" title="next page">Next Module →</a>
|
||||
</div>
|
||||
## Testing Your Implementation
|
||||
|
||||
### Inline Tests
|
||||
|
||||
```python
|
||||
# Test Linear layer shape
|
||||
layer = Linear(10, 5)
|
||||
x = Tensor(np.random.randn(3, 10)) # Batch of 3
|
||||
y = layer.forward(x)
|
||||
assert y.shape == (3, 5)
|
||||
print("✓ Linear shape correct")
|
||||
|
||||
# Test parameter count
|
||||
params = layer.parameters()
|
||||
assert len(params) == 2 # weight + bias
|
||||
assert params[0].shape == (10, 5)
|
||||
assert params[1].shape == (5,)
|
||||
print("✓ Parameters correct")
|
||||
|
||||
# Test Sequential
|
||||
model = Sequential([Linear(10, 5), ReLU(), Linear(5, 2)])
|
||||
output = model.forward(x)
|
||||
assert output.shape == (3, 2)
|
||||
print("✓ Sequential working")
|
||||
```
|
||||
|
||||
### Module Export & Validation
|
||||
|
||||
```bash
|
||||
tito export 03
|
||||
tito test 03
|
||||
```
|
||||
|
||||
**Expected output**:
|
||||
```
|
||||
✓ All tests passed! [18/18]
|
||||
✓ Module 03 complete!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Where This Code Lives
|
||||
|
||||
Your layers are used everywhere in TinyTorch:
|
||||
|
||||
```python
|
||||
# Networks use your Linear layers:
|
||||
from tinytorch.nn.networks import MLP
|
||||
from tinytorch.core.layers import Linear, Sequential
|
||||
|
||||
# Example: MNIST classifier
|
||||
mnist_model = Sequential([
|
||||
Linear(784, 128),
|
||||
ReLU(),
|
||||
Linear(128, 10)
|
||||
])
|
||||
|
||||
# This is how all neural networks are built!
|
||||
```
|
||||
|
||||
**Package structure**:
|
||||
```
|
||||
tinytorch/
|
||||
├── core/
|
||||
│ ├── layers.py ← YOUR implementations
|
||||
│ ├── tensor.py
|
||||
│ ├── activations.py
|
||||
├── nn/
|
||||
│ ├── networks.py (uses your layers)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Systems Thinking Questions
|
||||
|
||||
1. **Complexity Analysis**: A Linear(1000, 1000) layer performs 1M multiply-adds per sample. How does batch size affect total computation? How does this scale to GPT-3's Linear(12288, 49152) layers?
|
||||
|
||||
2. **Memory Trade-offs**: Why store both parameters (weights/biases) and activations? Can activations be deallocated after forward pass? What about during training?
|
||||
|
||||
3. **Initialization Strategies**: Why not initialize all weights to zero? What would happen? Why does Xavier initialization use `sqrt(6 / (in + out))`?
|
||||
|
||||
4. **Design Decisions**: Why is weight shape `(in, out)` instead of `(out, in)`? How does this affect batching and GPU performance?
|
||||
|
||||
5. **Framework Comparison**: PyTorch's Linear supports `bias=False` for layers without bias. When would this be useful? (Hint: LayerNorm, BatchNorm)
|
||||
|
||||
---
|
||||
|
||||
## Real-World Connections
|
||||
|
||||
### Industry Applications
|
||||
|
||||
- **ResNet Final Layer**: Linear(2048, 1000) maps features to ImageNet classes
|
||||
- **BERT Embeddings**: Linear(768, 30522) projects to vocabulary for masked LM
|
||||
- **GPT-3 Feedforward**: Linear(12288, 49152) in transformer MLP blocks
|
||||
- **Recommendation Systems**: Linear layers map user/item embeddings to scores
|
||||
|
||||
### Architecture Patterns
|
||||
|
||||
- **MLPs**: Stack of Linear + activation (used in tabular data, RL)
|
||||
- **Residual Connections**: `x + Linear(ReLU(Linear(x)))` (ResNet pattern)
|
||||
- **Attention Projections**: Q = Linear(x), K = Linear(x), V = Linear(x)
|
||||
|
||||
---
|
||||
|
||||
## What's Next?
|
||||
|
||||
**Great progress!** You can now build multi-layer networks. Next, you'll implement loss functions to measure how well networks perform.
|
||||
|
||||
**Module 04: Losses** - Implement MSE, Cross-Entropy, and Binary Cross-Entropy loss functions for training
|
||||
|
||||
[Continue to Module 04: Losses →](04-losses.html)
|
||||
|
||||
---
|
||||
|
||||
**Need Help?**
|
||||
- [Ask in GitHub Discussions](https://github.com/mlsysbook/TinyTorch/discussions)
|
||||
- [View Layers API Reference](../appendices/api-reference.html#layers)
|
||||
- [Report Issues](https://github.com/mlsysbook/TinyTorch/issues)
|
||||
|
||||
Reference in New Issue
Block a user