mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-07-25 13:33:13 -05:00
Vijay Janapa Reddi
8a101cf52d
Implement comprehensive grading workflow wrapped behind tito CLI: • tito grade setup - Initialize NBGrader course structure • tito grade generate - Create instructor version with solutions • tito grade release - Create student version without solutions • tito grade collect - Collect student submissions • tito grade autograde - Automatically grade submissions • tito grade manual - Open manual grading interface • tito grade feedback - Generate student feedback • tito grade export - Export grades to CSV This allows users to only learn tito commands without needing to understand NBGrader's complex interface. All grading functionality is accessible through simple, consistent tito commands.
120 KiB
120 KiB
In [ ]:
#| default_exp core.layers
#| export
import numpy as np
import os
import sys
# Import our dependencies - try from package first, then local modules
try:
from tinytorch.core.tensor import Tensor
from tinytorch.core.activations import ReLU, Sigmoid, Tanh, Softmax
except ImportError:
# For development, import from local modules
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '01_tensor'))
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '02_activations'))
try:
from tensor_dev import Tensor
from activations_dev import ReLU, Sigmoid, Tanh, Softmax
except ImportError:
# If the local modules are not available, use relative imports
from ..tensor.tensor_dev import Tensor
from ..activations.activations_dev import ReLU, Sigmoid, Tanh, SoftmaxIn [ ]:
print("🔥 TinyTorch Layers Module")
print(f"NumPy version: {np.__version__}")
print(f"Python version: {sys.version_info.major}.{sys.version_info.minor}")
print("Ready to build neural network layers!")In [ ]:
#| export
def matmul(A: np.ndarray, B: np.ndarray) -> np.ndarray:
"""
Matrix multiplication using explicit for-loops for deep understanding.
This implementation reveals the mathematical essence of neural networks!
Every time a neural network processes data, it's doing exactly this operation.
TODO: Implement matrix multiplication using three nested for-loops.
APPROACH:
1. Extract and validate matrix dimensions
2. Initialize result matrix with zeros
3. Implement the triple-nested loop structure
4. Accumulate dot products for each output element
MATHEMATICAL FOUNDATION:
For C = A @ B, each element C[i,j] is the dot product of:
- Row i from matrix A: [A[i,0], A[i,1], ..., A[i,n-1]]
- Column j from matrix B: [B[0,j], B[1,j], ..., B[n-1,j]]
VISUAL STEP-BY-STEP:
```
A = [[1, 2], B = [[5, 6], C = [[?, ?],
[3, 4]] [7, 8]] [?, ?]]
Computing C[0,0] (row 0 of A, column 0 of B):
A[0,:] = [1, 2] ←→ B[:,0] = [5, 7]
C[0,0] = 1*5 + 2*7 = 5 + 14 = 19
Computing C[0,1] (row 0 of A, column 1 of B):
A[0,:] = [1, 2] ←→ B[:,1] = [6, 8]
C[0,1] = 1*6 + 2*8 = 6 + 16 = 22
Computing C[1,0] (row 1 of A, column 0 of B):
A[1,:] = [3, 4] ←→ B[:,0] = [5, 7]
C[1,0] = 3*5 + 4*7 = 15 + 28 = 43
Computing C[1,1] (row 1 of A, column 1 of B):
A[1,:] = [3, 4] ←→ B[:,1] = [6, 8]
C[1,1] = 3*6 + 4*8 = 18 + 32 = 50
Final result: C = [[19, 22], [43, 50]]
```
IMPLEMENTATION ALGORITHM:
```python
# 1. Get dimensions and validate
m, n = A.shape # A is m×n
n2, p = B.shape # B is n×p (n2 must equal n)
assert n == n2 # Inner dimensions must match
# 2. Initialize result matrix
C = zeros(m, p) # Result is m×p
# 3. Triple nested loops
for i in range(m): # For each row of A
for j in range(p): # For each column of B
for k in range(n): # For each element in dot product
C[i,j] += A[i,k] * B[k,j] # Accumulate
```
NEURAL NETWORK CONNECTION:
In a neural network layer:
- A = input batch (batch_size × input_features)
- B = weight matrix (input_features × output_features)
- C = output batch (batch_size × output_features)
Each C[i,j] represents how much output feature j is activated for input sample i.
DEBUGGING HINTS:
- Check shapes: A.shape = (m,n), B.shape = (n,p) → C.shape = (m,p)
- Common error: Swapping B's dimensions (should be input_features × output_features)
- Accumulation: Start with C[i,j] = 0, then add all A[i,k] * B[k,j]
- Index bounds: i ∈ [0,m), j ∈ [0,p), k ∈ [0,n)
PERFORMANCE NOTE:
This implementation is O(mnp) time complexity and helps you understand:
- Why GPUs are essential for deep learning (parallelizable operations)
- Why NumPy/BLAS libraries are much faster (optimized C/Fortran)
- How memory access patterns affect performance
LEARNING CONNECTIONS:
- Foundation of ALL neural network computations
- Understanding enables debugging shape mismatches
- Basis for implementing custom layer types
- Essential for optimizing model performance
- Connects to linear algebra theory
"""
### BEGIN SOLUTION
# Get matrix dimensions
m, n = A.shape
n2, p = B.shape
# Check compatibility
if n != n2:
raise ValueError(f"Incompatible matrix dimensions: A is {m}x{n}, B is {n2}x{p}")
# Initialize result matrix
C = np.zeros((m, p))
# Triple nested loop for matrix multiplication
for i in range(m):
for j in range(p):
for k in range(n):
C[i, j] += A[i, k] * B[k, j]
return C
### END SOLUTIONIn [ ]:
def test_unit_matrix_multiplication():
"""Test matrix multiplication implementation"""
print("🔬 Unit Test: Matrix Multiplication...")
# Test simple 2x2 case
A = np.array([[1, 2], [3, 4]], dtype=np.float32)
B = np.array([[5, 6], [7, 8]], dtype=np.float32)
result = matmul(A, B)
expected = np.array([[19, 22], [43, 50]], dtype=np.float32)
assert np.allclose(result, expected), f"Matrix multiplication failed: expected {expected}, got {result}"
# Compare with NumPy
numpy_result = A @ B
assert np.allclose(result, numpy_result), f"Doesn't match NumPy: got {result}, expected {numpy_result}"
# Test different shapes
A2 = np.array([[1, 2, 3]], dtype=np.float32) # 1x3
B2 = np.array([[4], [5], [6]], dtype=np.float32) # 3x1
result2 = matmul(A2, B2)
expected2 = np.array([[32]], dtype=np.float32) # 1*4 + 2*5 + 3*6 = 32
assert np.allclose(result2, expected2), f"1x3 @ 3x1 failed: expected {expected2}, got {result2}"
# Test 3x3 case
A3 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32)
B3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.float32) # Identity
result3 = matmul(A3, B3)
assert np.allclose(result3, A3), "Multiplication by identity should preserve matrix"
# Test incompatible shapes
A4 = np.array([[1, 2]], dtype=np.float32) # 1x2
B4 = np.array([[3], [4], [5]], dtype=np.float32) # 3x1
try:
matmul(A4, B4)
assert False, "Should raise error for incompatible shapes"
except ValueError as e:
assert "Incompatible matrix dimensions" in str(e)
print("✅ Matrix multiplication tests passed!")
print(f"✅ 2x2 multiplication working correctly")
print(f"✅ Matches NumPy's implementation")
print(f"✅ Handles different shapes correctly")
print(f"✅ Proper error handling for incompatible shapes")
# Test function defined (called in main block)In [ ]:
#| export
class Dense:
"""
Dense (Linear/Fully Connected) Layer
Applies a linear transformation: y = xW + b
This is the fundamental building block of neural networks.
"""
def __init__(self, input_size: int, output_size: int, use_bias: bool = True):
"""
Initialize Dense layer with random weights and optional bias.
This initialization is CRITICAL for successful neural network training!
Poor initialization can cause vanishing/exploding gradients and training failure.
TODO: Implement Dense layer initialization with proper weight scaling.
APPROACH:
1. Store layer configuration parameters
2. Initialize weights using Xavier/Glorot strategy
3. Initialize bias terms (typically zeros)
4. Convert arrays to Tensor objects for compatibility
WEIGHT INITIALIZATION DEEP DIVE:
Why Random Initialization?
- Breaks symmetry: All neurons start different
- Enables learning: Gradients won't be identical
- Avoids dead neurons: Some neurons activate from start
Xavier/Glorot Initialization Strategy:
```
scale = sqrt(2 / (input_size + output_size))
weights ~ Normal(0, scale²)
```
Mathematical Justification:
- Maintains activation variance across layers
- Prevents vanishing/exploding gradients
- Empirically proven to improve training
VISUAL INITIALIZATION PATTERN:
```
Input Layer (3 neurons) Dense Layer (2 neurons)
┌─────┐ ┌─────┐
│ x₁ │ ──w₁₁──→ │ y₁ │
│ │ \\ │ │
│ x₂ │ ──w₂₁─w₂₂──→ │ y₂ │
│ │ / │ │
│ x₃ │ ──w₃₁──→ │ │
└─────┘ +b₁ +b₂ └─────┘
Weight Matrix W (3×2): Bias Vector b (2×1):
┌──────────────┐ ┌────┐
│ w₁₁ w₁₂ │ │ b₁ │
│ w₂₁ w₂₂ │ │ b₂ │
│ w₃₁ w₃₂ │ └────┘
└──────────────┘
```
EXAMPLE INITIALIZATION:
```python
layer = Dense(input_size=784, output_size=10) # MNIST classifier
# Weight shape: (784, 10) - each output connects to all inputs
# Bias shape: (10,) - one bias per output neuron
# Scale: sqrt(2/(784+10)) ≈ 0.05 - prevents gradients from exploding
```
IMPLEMENTATION STEPS:
```python
# 1. Store configuration
self.input_size = input_size # Number of input features
self.output_size = output_size # Number of output neurons
self.use_bias = use_bias # Whether to include bias terms
# 2. Calculate Xavier scale
scale = np.sqrt(2.0 / (input_size + output_size))
# 3. Initialize weights (shape matters!)
weight_data = np.random.randn(input_size, output_size) * scale
# 4. Initialize bias (usually zeros)
if use_bias:
bias_data = np.zeros(output_size)
# 5. Convert to Tensors
self.weights = Tensor(weight_data)
self.bias = Tensor(bias_data) if use_bias else None
```
ALTERNATIVE INITIALIZATION STRATEGIES:
He Initialization (better for ReLU):
```python
scale = np.sqrt(2.0 / input_size) # Only input size
```
Uniform Xavier:
```python
limit = np.sqrt(6.0 / (input_size + output_size))
weights = np.random.uniform(-limit, limit, (input_size, output_size))
```
COMMON INITIALIZATION MISTAKES:
1. **All zeros**: No learning (dead neurons)
2. **Too large**: Exploding gradients
3. **Too small**: Vanishing gradients
4. **Wrong shape**: Broadcasting errors
5. **Same values**: Symmetry problem
PRODUCTION SYSTEM COMPARISON:
```python
# Your implementation
layer = Dense(input_size, output_size)
# PyTorch equivalent
layer = torch.nn.Linear(input_size, output_size)
# Uses Kaiming uniform initialization by default
# TensorFlow equivalent
layer = tf.keras.layers.Dense(output_size, input_shape=(input_size,))
# Uses Glorot uniform initialization by default
```
DEBUGGING HINTS:
- Print weight statistics: mean ≈ 0, std ≈ scale
- Check shapes: weights (input_size, output_size), bias (output_size,)
- Verify Tensor conversion: isinstance(self.weights, Tensor)
- Test forward pass: no shape errors
LEARNING CONNECTIONS:
- Foundation for all layer types (Conv2D, LSTM, Attention)
- Understanding gradients and backpropagation
- Basis for transfer learning (loading pre-trained weights)
- Essential for model architecture design
"""
### BEGIN SOLUTION
# Store layer parameters
self.input_size = input_size
self.output_size = output_size
self.use_bias = use_bias
# Xavier/Glorot initialization
scale = np.sqrt(2.0 / (input_size + output_size))
# Initialize weights with random values
weight_data = np.random.randn(input_size, output_size) * scale
self.weights = Tensor(weight_data)
# Initialize bias
if use_bias:
bias_data = np.zeros(output_size)
self.bias = Tensor(bias_data)
else:
self.bias = None
### END SOLUTION
def forward(self, x):
"""
Forward pass through the Dense layer: the heart of neural computation.
This function implements y = xW + b, the fundamental equation that powers
all neural networks from simple perceptrons to massive transformers!
TODO: Implement the forward pass with proper shape handling.
APPROACH:
1. Apply matrix multiplication for feature combination
2. Add bias terms for output shifting
3. Return properly shaped Tensor result
4. Handle batch processing automatically
MATHEMATICAL FOUNDATION:
The Linear Transformation:
```
y = xW + b
Where:
x: Input features (batch_size × input_features)
W: Weight matrix (input_features × output_features)
b: Bias vector (output_features,)
y: Output features (batch_size × output_features)
```
VISUAL DATA FLOW:
```
Input Batch Weight Matrix Bias Vector Output Batch
┌─────────────┐ ┌─────────────┐ ┌─────────┐ ┌─────────────┐
│ [x₁₁ x₁₂] │ │ [w₁₁ w₁₂] │ │ [b₁ b₂] │ │ [y₁₁ y₁₂] │
│ [x₂₁ x₂₂] │ @ │ [w₂₁ w₂₂] │ + │ │ = │ [y₂₁ y₂₂] │
│ [x₃₁ x₃₂] │ └─────────────┘ └─────────┘ │ [y₃₁ y₃₂] │
└─────────────┘ └─────────────┘
(3×2) (2×2) (2,) (3×2)
```
STEP-BY-STEP COMPUTATION:
For each output element y[i,j]:
```
y[i,j] = Σₖ x[i,k] * W[k,j] + b[j]
Example:
x = [[1, 2]] # 1 sample, 2 features
W = [[0.5, 0.3], # 2 input → 2 output
[0.7, 0.4]]
b = [0.1, 0.2] # bias for each output
y[0,0] = x[0,0]*W[0,0] + x[0,1]*W[1,0] + b[0]
= 1*0.5 + 2*0.7 + 0.1 = 0.5 + 1.4 + 0.1 = 2.0
y[0,1] = x[0,0]*W[0,1] + x[0,1]*W[1,1] + b[1]
= 1*0.3 + 2*0.4 + 0.2 = 0.3 + 0.8 + 0.2 = 1.3
Result: y = [[2.0, 1.3]]
```
BATCH PROCESSING MAGIC:
The same operation works for ANY batch size:
```
Single sample: (1, features) @ (features, outputs) = (1, outputs)
Mini-batch: (32, features) @ (features, outputs) = (32, outputs)
Large batch: (1000, features) @ (features, outputs) = (1000, outputs)
```
IMPLEMENTATION DETAILS:
```python
# 1. Matrix multiplication (the core operation)
linear_output = matmul(x.data, self.weights.data)
# 2. Bias addition (broadcasting handles shape automatically)
if self.use_bias and self.bias is not None:
linear_output = linear_output + self.bias.data
# Broadcasting: (batch_size, output_features) + (output_features,)
# → (batch_size, output_features)
# 3. Return as proper Tensor type
return type(x)(linear_output) # Preserves Tensor class
```
BROADCASTING EXPLANATION:
NumPy automatically broadcasts the bias:
```
linear_output.shape = (batch_size, output_features) # e.g., (32, 10)
bias.shape = (output_features,) # e.g., (10,)
# Broadcasting adds bias to each sample:
result[i,j] = linear_output[i,j] + bias[j] # for all i
```
REAL-WORLD APPLICATIONS:
Image Classification:
```
# Flatten image: (28, 28) → (784,)
# Dense layer: (784,) → (10,) class scores
x = flattened_image # Shape: (batch, 784)
scores = dense_layer(x) # Shape: (batch, 10)
```
Language Model:
```
# Word embedding: word_id → dense vector
# Dense layer: hidden → vocabulary scores
x = hidden_state # Shape: (batch, hidden_size)
logits = output_layer(x) # Shape: (batch, vocab_size)
```
COMMON SHAPE ERRORS AND SOLUTIONS:
```
Error: "Cannot multiply (32, 784) and (10, 784)"
Solution: Weight shape should be (784, 10), not (10, 784)
Error: "Cannot add (32, 10) and (784,)"
Solution: Bias shape should be (10,), not (784,)
Error: "Expected 2D input, got 1D"
Solution: Reshape input from (features,) to (1, features)
```
DEBUGGING CHECKLIST:
- Input shape: (batch_size, input_features)
- Weight shape: (input_features, output_features)
- Bias shape: (output_features,) or None
- Output shape: (batch_size, output_features)
PERFORMANCE NOTES:
- Matrix multiplication is O(batch × input × output)
- Most computation time spent here in large models
- GPU acceleration crucial for large layers
- Memory usage: store input, weights, bias, output
LEARNING CONNECTIONS:
- Foundation of backpropagation (gradients flow through this operation)
- Basis for all advanced layer types (attention, convolution)
- Understanding enables custom layer development
- Critical for model optimization and deployment
"""
### BEGIN SOLUTION
# Perform matrix multiplication
linear_output = matmul(x.data, self.weights.data)
# Add bias if present
if self.use_bias and self.bias is not None:
linear_output = linear_output + self.bias.data
return type(x)(linear_output)
### END SOLUTION
def __call__(self, x):
"""Make the layer callable: layer(x) instead of layer.forward(x)"""
return self.forward(x)In [ ]:
def test_unit_dense_layer():
"""Test Dense layer implementation"""
print("🔬 Unit Test: Dense Layer...")
# Test layer creation
layer = Dense(input_size=3, output_size=2)
# Check weight and bias shapes
assert layer.weights.shape == (3, 2), f"Weight shape should be (3, 2), got {layer.weights.shape}"
assert layer.bias is not None, "Bias should not be None when use_bias=True"
assert layer.bias.shape == (2,), f"Bias shape should be (2,), got {layer.bias.shape}"
# Test forward pass
input_data = Tensor([[1, 2, 3]]) # Shape: (1, 3)
output = layer(input_data)
# Check output shape
assert output.shape == (1, 2), f"Output shape should be (1, 2), got {output.shape}"
# Test batch processing
batch_input = Tensor([[1, 2, 3], [4, 5, 6]]) # Shape: (2, 3)
batch_output = layer(batch_input)
assert batch_output.shape == (2, 2), f"Batch output shape should be (2, 2), got {batch_output.shape}"
# Test without bias
no_bias_layer = Dense(input_size=3, output_size=2, use_bias=False)
assert no_bias_layer.bias is None, "Layer without bias should have None bias"
no_bias_output = no_bias_layer(input_data)
assert no_bias_output.shape == (1, 2), "No-bias layer should still produce correct shape"
# Test that different inputs produce different outputs
input1 = Tensor([[1, 0, 0]])
input2 = Tensor([[0, 1, 0]])
output1 = layer(input1)
output2 = layer(input2)
# Should not be equal (with high probability due to random initialization)
assert not np.allclose(output1.data, output2.data), "Different inputs should produce different outputs"
# Test linearity property: layer(a*x) = a*layer(x)
scale = 2.0
scaled_input = Tensor([[2, 4, 6]]) # 2 * [1, 2, 3]
scaled_output = layer(scaled_input)
# Due to bias, this won't be exactly 2*output, but the linear part should scale
print("✅ Dense layer tests passed!")
print(f"✅ Correct weight and bias initialization")
print(f"✅ Forward pass produces correct shapes")
print(f"✅ Batch processing works correctly")
print(f"✅ Bias and no-bias variants work")
print(f"✅ Naive matrix multiplication option works")
# Test function defined (called in main block)In [ ]:
def test_unit_layer_activation():
"""Test Dense layer comprehensive testing with activation functions"""
print("🔬 Unit Test: Layer-Activation Comprehensive Test...")
# Create layer and activation functions
layer = Dense(input_size=4, output_size=3)
relu = ReLU()
sigmoid = Sigmoid()
tanh = Tanh()
softmax = Softmax()
# Test input
input_data = Tensor([[1, -2, 3, -4], [2, 1, -1, 3]]) # Shape: (2, 4)
# Test Dense + ReLU (common hidden layer pattern)
linear_output = layer(input_data)
relu_output = relu(linear_output)
assert relu_output.shape == (2, 3), "ReLU output should preserve shape"
assert np.all(relu_output.data >= 0), "ReLU output should be non-negative"
# Test Dense + Softmax (classification output pattern)
softmax_output = softmax(linear_output)
assert softmax_output.shape == (2, 3), "Softmax output should preserve shape"
# Each row should sum to 1 (probability distribution)
for i in range(2):
row_sum = np.sum(softmax_output.data[i])
assert abs(row_sum - 1.0) < 1e-6, f"Row {i} should sum to 1, got {row_sum}"
# Test Dense + Sigmoid (binary classification pattern)
sigmoid_output = sigmoid(linear_output)
assert sigmoid_output.shape == (2, 3), "Sigmoid output should preserve shape"
assert np.all(sigmoid_output.data > 0), "Sigmoid output should be positive"
assert np.all(sigmoid_output.data < 1), "Sigmoid output should be less than 1"
# Test Dense + Tanh (hidden layer with centered outputs)
tanh_output = tanh(linear_output)
assert tanh_output.shape == (2, 3), "Tanh output should preserve shape"
assert np.all(tanh_output.data > -1), "Tanh output should be > -1"
assert np.all(tanh_output.data < 1), "Tanh output should be < 1"
# Test chained layers (simple 2-layer network)
layer1 = Dense(input_size=4, output_size=5)
layer2 = Dense(input_size=5, output_size=3)
# Forward pass through 2-layer network
hidden = relu(layer1(input_data))
output = softmax(layer2(hidden))
assert output.shape == (2, 3), "2-layer network should produce correct output shape"
# Each output should be a valid probability distribution
for i in range(2):
row_sum = np.sum(output.data[i])
assert abs(row_sum - 1.0) < 1e-6, f"Network output row {i} should sum to 1"
# Test that layers are learning-ready (have parameters)
assert hasattr(layer1, 'weights'), "Layer should have weights"
assert hasattr(layer1, 'bias'), "Layer should have bias"
assert isinstance(layer1.weights, Tensor), "Weights should be Tensor"
assert isinstance(layer1.bias, Tensor), "Bias should be Tensor"
print("✅ Layer-activation comprehensive tests passed!")
print(f"✅ Dense + ReLU working correctly")
print(f"✅ Dense + Softmax producing valid probabilities")
print(f"✅ Dense + Sigmoid bounded correctly")
print(f"✅ Dense + Tanh centered correctly")
print(f"✅ Multi-layer networks working")
print(f"✅ All components ready for training!")
# Test function defined (called in main block)In [ ]:
def test_module_layer_tensor_integration():
"""
Tests that a Tensor can be passed through a Layer subclass
and that the output is of the correct type and shape.
"""
print("🔬 Running Integration Test: Layer with Tensor...")
# 1. Define a simple Layer that doubles the input
class DoubleLayer(Dense): # Inherit from Dense to get __call__
def forward(self, x: Tensor) -> Tensor:
return x * 2
# 2. Create an instance of the layer
double_layer = DoubleLayer(input_size=1, output_size=1) # Dummy sizes
# 3. Create a Tensor from the previous module
input_tensor = Tensor([1, 2, 3])
# 4. Perform the forward pass
output_tensor = double_layer(input_tensor)
# 5. Assert correctness
assert isinstance(output_tensor, Tensor), "Output should be a Tensor"
assert np.array_equal(output_tensor.data, np.array([2, 4, 6])), "Output data is incorrect"
print("✅ Integration Test Passed: Layer correctly processed Tensor.")
# Test function defined (called in main block)In [ ]:
import time
import psutil
import os
class LayerArchitectureProfiler:
"""
Architecture analysis toolkit for neural network layers.
Helps ML engineers understand memory scaling, parameter counts,
and computational complexity of different layer configurations.
"""
def __init__(self):
self.process = psutil.Process(os.getpid())
self.analysis_cache = {}
def analyze_layer_parameters(self, input_size, hidden_size, output_size):
"""
Analyze parameter count and memory usage for a layer configuration.
TODO: Implement parameter count analysis.
STEP-BY-STEP IMPLEMENTATION:
1. Calculate weight matrix parameters: input_size * hidden_size
2. Calculate bias parameters: hidden_size
3. Calculate total parameters: weights + bias
4. Calculate memory usage: parameters * 4 bytes (float32)
5. Return analysis dictionary with all metrics
EXAMPLE:
profiler = LayerArchitectureProfiler()
analysis = profiler.analyze_layer_parameters(784, 128, 10)
print(f"Parameters: {analysis['total_parameters']:,}")
print(f"Memory: {analysis['memory_mb']:.2f} MB")
HINTS:
- Weight matrix shape: (input_size, hidden_size)
- Bias vector shape: (hidden_size,)
- Float32 = 4 bytes per parameter
- Convert bytes to MB: bytes / (1024 * 1024)
"""
### BEGIN SOLUTION
# Calculate parameters
weight_params = input_size * hidden_size
bias_params = hidden_size
total_params = weight_params + bias_params
# Calculate memory (assuming float32 = 4 bytes)
memory_bytes = total_params * 4
memory_mb = memory_bytes / (1024 * 1024)
return {
'input_size': input_size,
'hidden_size': hidden_size,
'output_size': output_size,
'weight_parameters': weight_params,
'bias_parameters': bias_params,
'total_parameters': total_params,
'memory_bytes': memory_bytes,
'memory_mb': memory_mb
}
### END SOLUTION
def analyze_network_scaling(self, input_size, hidden_sizes, output_size):
"""
Analyze how network depth affects parameter count and memory.
TODO: Implement network scaling analysis.
STEP-BY-STEP IMPLEMENTATION:
1. Initialize total parameters counter
2. For each layer in the network:
a. Calculate layer parameters using analyze_layer_parameters
b. Add to total count
c. Update input_size for next layer
3. Calculate total memory usage
4. Return comprehensive analysis
EXAMPLE:
profiler = LayerArchitectureProfiler()
analysis = profiler.analyze_network_scaling(784, [512, 256, 128], 10)
print(f"Total parameters: {analysis['total_parameters']:,}")
print(f"Layers: {len(analysis['layer_details'])}")
HINTS:
- Loop through hidden_sizes for each layer
- Track input_size changes: input → hidden[0] → hidden[1] → ... → output
- Sum all layer parameters
- Store per-layer details for analysis
"""
### BEGIN SOLUTION
total_parameters = 0
layer_details = []
current_input = input_size
# Analyze each hidden layer
for i, hidden_size in enumerate(hidden_sizes):
layer_analysis = self.analyze_layer_parameters(current_input, hidden_size, 0)
layer_analysis['layer_name'] = f'Hidden_{i+1}'
layer_details.append(layer_analysis)
total_parameters += layer_analysis['total_parameters']
current_input = hidden_size
# Analyze output layer
output_analysis = self.analyze_layer_parameters(current_input, output_size, 0)
output_analysis['layer_name'] = 'Output'
layer_details.append(output_analysis)
total_parameters += output_analysis['total_parameters']
# Calculate total memory
total_memory_mb = total_parameters * 4 / (1024 * 1024)
return {
'network_architecture': f"{input_size} → {' → '.join(map(str, hidden_sizes))} → {output_size}",
'total_parameters': total_parameters,
'total_memory_mb': total_memory_mb,
'num_layers': len(hidden_sizes) + 1,
'layer_details': layer_details
}
### END SOLUTION
def compare_architectures(self, input_size, architecture_configs, output_size=10):
"""
Compare different network architectures for parameter efficiency.
This function is PROVIDED to demonstrate architecture analysis.
Students use it to understand architecture trade-offs.
"""
print(f"🏗️ ARCHITECTURE COMPARISON")
print(f"=" * 50)
print(f"Input size: {input_size}, Output size: {output_size}")
results = {}
for arch_name, hidden_sizes in architecture_configs.items():
analysis = self.analyze_network_scaling(input_size, hidden_sizes, output_size)
results[arch_name] = analysis
print(f"\n📊 {arch_name}:")
print(f" Architecture: {analysis['network_architecture']}")
print(f" Parameters: {analysis['total_parameters']:,}")
print(f" Memory: {analysis['total_memory_mb']:.2f} MB")
print(f" Layers: {analysis['num_layers']}")
# Find most/least parameter efficient
sorted_by_params = sorted(results.items(), key=lambda x: x[1]['total_parameters'])
most_efficient = sorted_by_params[0]
least_efficient = sorted_by_params[-1]
print(f"\n🎯 EFFICIENCY ANALYSIS:")
print(f" Most efficient: {most_efficient[0]} ({most_efficient[1]['total_parameters']:,} params)")
print(f" Least efficient: {least_efficient[0]} ({least_efficient[1]['total_parameters']:,} params)")
efficiency_ratio = least_efficient[1]['total_parameters'] / most_efficient[1]['total_parameters']
print(f" Parameter difference: {efficiency_ratio:.1f}x")
return results
def analyze_depth_vs_width_tradeoffs(self, input_size=784, output_size=10):
"""
Analyze the classic deep vs wide network trade-off.
This function is PROVIDED to show systems thinking.
Students run it to understand architecture decisions.
"""
print(f"🔍 DEPTH vs WIDTH ANALYSIS")
print(f"=" * 40)
# Test different depth vs width configurations
configurations = {
'Shallow Wide': [1024], # 1 huge layer
'Medium Wide': [512, 512], # 2 medium layers
'Medium Deep': [256, 256, 256], # 3 smaller layers
'Deep Narrow': [128, 128, 128, 128], # 4 narrow layers
'Very Deep': [64, 64, 64, 64, 64, 64] # 6 very narrow layers
}
results = {}
for config_name, hidden_sizes in configurations.items():
analysis = self.analyze_network_scaling(input_size, hidden_sizes, output_size)
results[config_name] = analysis
# Calculate depth and width metrics
depth = len(hidden_sizes)
avg_width = sum(hidden_sizes) / len(hidden_sizes)
max_width = max(hidden_sizes)
print(f"\n{config_name}:")
print(f" Depth: {depth} layers")
print(f" Avg width: {avg_width:.0f} neurons")
print(f" Max width: {max_width} neurons")
print(f" Parameters: {analysis['total_parameters']:,}")
print(f" Memory: {analysis['total_memory_mb']:.2f} MB")
print(f"\n💡 ARCHITECTURE INSIGHTS:")
print(f" - Deeper networks: Better representation learning, harder to train")
print(f" - Wider networks: More capacity per layer, more parameters")
print(f" - Modern trend: Very deep (100+ layers) with skip connections")
print(f" - Memory scales with total parameters regardless of arrangement")
return results
def analyze_famous_architectures():
"""
Analyze parameter counts of famous neural network architectures.
This function is PROVIDED to connect student work to real systems.
Shows how layer analysis applies to production models.
"""
profiler = LayerArchitectureProfiler()
print(f"🌟 FAMOUS ARCHITECTURE ANALYSIS")
print(f"=" * 50)
# Simplified versions of famous architectures
famous_models = {
'LeNet-5 (1998)': {
'description': 'First successful CNN',
'approx_params': 60_000,
'era': 'Early deep learning'
},
'AlexNet (2012)': {
'description': 'ImageNet breakthrough',
'approx_params': 60_000_000,
'era': 'Deep learning revolution'
},
'VGG-16 (2014)': {
'description': 'Very deep networks',
'approx_params': 138_000_000,
'era': 'Going deeper'
},
'ResNet-50 (2015)': {
'description': 'Skip connections enable very deep nets',
'approx_params': 25_600_000,
'era': 'Architecture innovation'
},
'GPT-3 (2020)': {
'description': 'Large language model',
'approx_params': 175_000_000_000,
'era': 'Scale revolution'
},
'GPT-4 (2023)': {
'description': 'Estimated multimodal model',
'approx_params': 1_800_000_000_000,
'era': 'Massive scale'
}
}
print(f"Model Evolution Over Time:")
for model_name, info in famous_models.items():
params = info['approx_params']
memory_gb = params * 4 / (1024**3) # Rough memory estimate
print(f"\n{model_name}:")
print(f" Parameters: {params:,}")
print(f" Est. Memory: {memory_gb:.1f} GB")
print(f" Description: {info['description']}")
print(f" Era: {info['era']}")
# Show scaling progression
print(f"\n📈 SCALING PROGRESSION:")
params_1998 = famous_models['LeNet-5 (1998)']['approx_params']
params_2023 = famous_models['GPT-4 (2023)']['approx_params']
scaling_factor = params_2023 / params_1998
print(f" 1998 → 2023: {scaling_factor:,.0f}x parameter increase")
print(f" That's about {scaling_factor/1000000:.1f} million times larger!")
print(f" Memory requirements grew from KB to TB")
print(f"\n🎯 SYSTEMS IMPLICATIONS:")
print(f" - Parameter count directly affects memory requirements")
print(f" - Larger models need distributed training across multiple GPUs")
print(f" - Model serving requires careful memory management")
print(f" - Architecture efficiency becomes crucial at scale")
return famous_modelsIn [ ]:
"""
YOUR REFLECTION ON PARAMETER MANAGEMENT AND MEMORY OPTIMIZATION:
TODO: Replace this text with your thoughtful response about parameter management system design.
Consider addressing:
- How would you optimize parameter storage and memory usage for billion-parameter models?
- What strategies would you use for parameter initialization in very deep networks?
- How would you implement parameter sharing and distributed storage efficiently?
- What role would quantization play in your parameter management system?
- How would you balance memory constraints with model capacity requirements?
Write a technical analysis connecting your layer implementations to real parameter management challenges.
GRADING RUBRIC (Instructor Use):
- Demonstrates understanding of large-scale parameter management challenges (3 points)
- Addresses memory optimization and distributed storage strategies (3 points)
- Shows practical knowledge of initialization and quantization techniques (2 points)
- Demonstrates systems thinking about memory vs capacity trade-offs (2 points)
- Clear technical reasoning and practical considerations (bonus points for innovative approaches)
"""
### BEGIN SOLUTION
# Student response area - instructor will replace this section during grading setup
# This is a manually graded question requiring technical analysis of parameter management
# Students should demonstrate understanding of memory optimization and distributed parameter storage
### END SOLUTIONIn [ ]:
"""
YOUR REFLECTION ON ABSTRACTION DESIGN AND FRAMEWORK INTEGRATION:
TODO: Replace this text with your thoughtful response about layer abstraction system design.
Consider addressing:
- How would you design layer abstractions that balance simplicity with optimization potential?
- What strategies would you use to integrate layers with automatic differentiation systems?
- How would you enable operation fusion while maintaining clean abstractions?
- What role would backend abstraction play in supporting multiple hardware platforms?
- How would you maintain API compatibility while enabling hardware-specific optimizations?
Write an architectural analysis connecting your layer abstractions to real framework design challenges.
GRADING RUBRIC (Instructor Use):
- Shows understanding of abstraction design principles for ML systems (3 points)
- Designs practical approaches to automatic differentiation integration (3 points)
- Addresses performance optimization and hardware abstraction (2 points)
- Demonstrates systems thinking about framework architecture (2 points)
- Clear architectural reasoning with framework insights (bonus points for comprehensive understanding)
"""
### BEGIN SOLUTION
# Student response area - instructor will replace this section during grading setup
# This is a manually graded question requiring understanding of framework abstraction design
# Students should demonstrate knowledge of balancing simplicity with optimization potential
### END SOLUTIONIn [ ]:
"""
YOUR REFLECTION ON INITIALIZATION STRATEGIES AND TRAINING STABILITY:
TODO: Replace this text with your thoughtful response about advanced initialization system design.
Consider addressing:
- How would you design initialization strategies for different types of layers and architectures?
- What approaches would you use to ensure stable gradient flow in ultra-deep networks?
- How would you handle initialization for complex architectures with skip connections and attention?
- What role would adaptive initialization play in improving training stability?
- How would you prevent initialization-related training failures in large-scale experiments?
Write a design analysis connecting your initialization implementations to real training stability challenges.
GRADING RUBRIC (Instructor Use):
- Understands initialization impact on training stability and gradient flow (3 points)
- Designs practical approaches to layer-specific and adaptive initialization (3 points)
- Addresses complex architecture initialization challenges (2 points)
- Shows systems thinking about training optimization and stability (2 points)
- Clear design reasoning with initialization optimization insights (bonus points for deep understanding)
"""
### BEGIN SOLUTION
# Student response area - instructor will replace this section during grading setup
# This is a manually graded question requiring understanding of initialization strategies and training stability
# Students should demonstrate knowledge of gradient flow and initialization optimization challenges
### END SOLUTION