github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-03-11 23:03:33 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity

Update testing design to inline-first approach

Browse Source 
- Prioritize student learning effectiveness over context switching
- Define three-tier architecture: Inline → Module → Integration
- Emphasize comprehensive inline testing with educational context
- Maintain professional module tests for grading
- Preserve flow state by keeping students in notebooks
- Provide immediate, encouraging feedback with visual indicators
This commit is contained in:
Vijay Janapa Reddi
2025-07-12 19:36:09 -04:00
parent fb4f92c35f
commit 1d3314add5
1 changed files with 194 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files

493
docs/development/testing-design.md
View File

@@ -2,46 +2,111 @@
## Overview
This document defines the four-tier testing architecture for TinyTorch, ensuring comprehensive validation while maintaining educational clarity and avoiding dependency cascades.
This document defines the **inline-first testing architecture** for TinyTorch, prioritizing student learning effectiveness and flow state over context switching overhead.
## Four-Tier Testing Architecture
## Core Philosophy: Learning-First Testing
### 1. Unit Tests (In Notebooks)
**Goal**: Immediate feedback on individual functions during development
**Primary Goal**: Student learning and confidence building
**Secondary Goal**: Comprehensive validation for grading
**Tertiary Goal**: Professional testing practices
### Key Insight
Students learning ML concepts are already at cognitive capacity. Every context switch is expensive and disrupts the learning flow. We prioritize immediate feedback and confidence building over professional tool complexity.
## Three-Tier Testing Architecture
### 1. Inline Tests (Primary - In Notebooks)
**Goal**: Immediate feedback and confidence building during development
**Location**: Embedded in `*_dev.py` files as NBGrader cells
**Dependencies**: None (or minimal, well-controlled)
**Scope**: Individual functions and methods
**Purpose**: Catch basic implementation errors immediately
**Scope**: Comprehensive testing with educational context
**Purpose**: Build confidence step-by-step while ensuring correctness
**Example**:
```python
# %% nbgrader={"grade": true, "grade_id": "test-relu-basic", "locked": true, "points": 5}
# Quick validation of ReLU function
def test_relu_basic():
# Test with simple inputs
result = relu([-1, 0, 1, 2])
expected = [0, 0, 1, 2]
assert result == expected, f"Expected {expected}, got {result}"
print("✅ ReLU function works!")
# %% [markdown]
"""
### 🧪 Test Your ReLU Implementation
test_relu_basic()
Let's verify your ReLU function works correctly with various inputs.
This tests the core functionality you just implemented.
"""
# %% nbgrader={"grade": true, "grade_id": "test-relu-comprehensive", "locked": true, "points": 15, "schema_version": 3, "solution": false, "task": false}
def test_relu_comprehensive():
"""Comprehensive test of ReLU function."""
print("🔬 Testing ReLU function...")
# Test 1: Basic positive numbers
try:
result = relu([1, 2, 3])
expected = [1, 2, 3]
assert result == expected, f"Positive numbers: expected {expected}, got {result}"
print("✅ Positive numbers work correctly")
except Exception as e:
print(f"❌ Positive numbers failed: {e}")
return False
# Test 2: Negative numbers (should become 0)
try:
result = relu([-1, -2, -3])
expected = [0, 0, 0]
assert result == expected, f"Negative numbers: expected {expected}, got {result}"
print("✅ Negative numbers correctly clipped to 0")
except Exception as e:
print(f"❌ Negative numbers failed: {e}")
return False
# Test 3: Mixed positive and negative
try:
result = relu([-2, -1, 0, 1, 2])
expected = [0, 0, 0, 1, 2]
assert result == expected, f"Mixed numbers: expected {expected}, got {result}"
print("✅ Mixed positive/negative numbers work correctly")
except Exception as e:
print(f"❌ Mixed numbers failed: {e}")
return False
# Test 4: Edge case - zero
try:
result = relu([0])
expected = [0]
assert result == expected, f"Zero: expected {expected}, got {result}"
print("✅ Zero handled correctly")
except Exception as e:
print(f"❌ Zero failed: {e}")
return False
print("🎉 All ReLU tests passed! Your implementation works correctly.")
print("📈 Progress: ReLU function ✓")
return True
# Run the test
success = test_relu_comprehensive()
if not success:
print("\n💡 Hints for fixing ReLU:")
print("- ReLU should return max(0, x) for each element")
print("- Negative numbers should become 0")
print("- Positive numbers should stay unchanged")
print("- Zero should remain zero")
```
**Characteristics**:
- **Fast**: Execute in seconds
- **Simple**: Easy to understand and debug
- **Focused**: Test one function at a time
- **Visual**: Clear pass/fail feedback with emojis
- **Educational**: Explain what's being tested
- **Comprehensive**: Test all functionality, edge cases, error conditions
- **Educational**: Explain what's being tested and why
- **Visual**: Clear pass/fail feedback with emojis and progress tracking
- **Immediate**: No context switching required
- **Encouraging**: Build confidence with positive reinforcement
- **Helpful**: Provide hints and guidance when tests fail
### 2. Module Tests (Separate Files with Mocks)
**Goal**: Comprehensive validation of module functionality using simple, visible mocks
### 2. Module Tests (For Grading - Separate Files with Mocks)
**Goal**: Comprehensive validation for instructor grading using simple, visible mocks
**Location**: `tests/test_{module}.py` files
**Dependencies**: Simple, visible mock objects (no cross-module dependencies)
**Scope**: Complete module functionality
**Purpose**: Verify module works correctly with well-defined interfaces
**Scope**: Complete module functionality with professional structure
**Purpose**: Verify module works correctly for grading and assessment
**Example**:
```python
@@ -50,18 +115,16 @@ test_relu_basic()
Comprehensive Layers Module Tests
Tests Dense layer functionality using simple mock objects.
No dependencies on other TinyTorch modules.
Used for instructor grading and comprehensive validation.
"""
class SimpleTensor:
"""
Simple mock of what Dense layer expects from Tensor.
Simple mock tensor for testing layers.
Your Dense layer should work with any object that has:
Shows exactly what interface the Dense layer expects:
- .data (numpy array): The actual numerical data
- .shape (tuple): Dimensions of the data
This mock shows exactly what interface your layer needs.
"""
def __init__(self, data):
self.data = np.array(data)
@@ -71,29 +134,25 @@ class SimpleTensor:
return f"SimpleTensor(shape={self.shape})"
class TestDenseLayer:
"""Comprehensive tests for Dense layer implementation."""
"""Comprehensive tests for Dense layer - used for grading."""
def test_initialization(self):
"""Test Dense layer creation and weight initialization."""
layer = Dense(input_size=3, output_size=2)
# Check weights and bias are created
assert hasattr(layer, 'weights'), "Dense layer should have weights"
assert hasattr(layer, 'bias'), "Dense layer should have bias"
assert layer.weights.shape == (3, 2), f"Expected weights shape (3, 2), got {layer.weights.shape}"
assert layer.bias.shape == (2,), f"Expected bias shape (2,), got {layer.bias.shape}"
def test_forward_pass(self):
"""Test Dense layer forward pass with mock tensor."""
def test_forward_pass_comprehensive(self):
"""Comprehensive test of Dense layer forward pass."""
layer = Dense(input_size=3, output_size=2)
# Create mock input
input_tensor = SimpleTensor([[1.0, 2.0, 3.0]]) # Batch size 1, 3 features
# Forward pass
# Test single sample
input_tensor = SimpleTensor([[1.0, 2.0, 3.0]])
output = layer(input_tensor)
# Verify output
assert hasattr(output, 'data'), "Layer should return tensor-like object with .data"
assert hasattr(output, 'shape'), "Layer should return tensor-like object with .shape"
assert output.shape == (1, 2), f"Expected output shape (1, 2), got {output.shape}"
@@ -101,331 +160,167 @@ class TestDenseLayer:
# Verify computation (y = Wx + b)
expected = np.dot(input_tensor.data, layer.weights) + layer.bias
np.testing.assert_array_almost_equal(output.data, expected)
def test_batch_processing(self):
"""Test Dense layer with batch of inputs."""
layer = Dense(input_size=2, output_size=3)
# Batch of 4 samples, 2 features each
batch_input = SimpleTensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0], [7.0, 8.0]])
# Test batch processing
batch_input = SimpleTensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
batch_output = layer(batch_input)
assert batch_output.shape == (2, 2), f"Expected batch output shape (2, 2), got {batch_output.shape}"
output = layer(batch_input)
assert output.shape == (4, 3), f"Expected batch output shape (4, 3), got {output.shape}"
def test_edge_cases(self):
"""Test Dense layer with edge cases."""
layer = Dense(input_size=1, output_size=1)
# Single feature, single output
single_input = SimpleTensor([[5.0]])
output = layer(single_input)
assert output.shape == (1, 1)
# Large batch
large_batch = SimpleTensor([[1.0]] * 100) # 100 samples
output = layer(large_batch)
assert output.shape == (100, 1)
# Test edge cases
edge_input = SimpleTensor([[0.0, 0.0, 0.0]])
edge_output = layer(edge_input)
assert edge_output.shape == (1, 2), "Should handle zero input"
```
**Characteristics**:
- **Self-contained**: No dependencies on other TinyTorch modules
- **Mock-based**: Simple, visible mocks that document interfaces
- **Comprehensive**: Test all functionality, edge cases, error conditions
- **Clear interfaces**: Mocks show exactly what the module expects
- **Debuggable**: Students can easily understand and modify mocks
- **Professional**: Use pytest structure and best practices
- **Grading-focused**: Designed for instructor assessment
### 3. Integration Tests (With Vetted Solutions)
**Goal**: Verify new module composes correctly with other vetted modules
### 3. Integration Tests (Cross-Module Validation)
**Goal**: Verify modules work together using vetted solutions
**Location**: `tests/integration/` directory
**Dependencies**: Instructor-provided working implementations of prerequisite modules
**Dependencies**: Instructor-provided working implementations
**Scope**: Cross-module workflows and realistic ML scenarios
**Purpose**: Ensure modules work together in real ML pipelines
**Purpose**: Ensure modules compose correctly without cascade failures
**Example**:
```python
# tests/integration/test_layers_integration.py
# tests/integration/test_basic_ml_pipeline.py
"""
Integration Tests for Layers Module
Integration Tests - Basic ML Pipeline
Tests how student's layer implementation works with vetted Tensor and Activation modules.
Uses instructor-provided working implementations to avoid dependency cascades.
Tests how student modules work together with vetted solutions.
No cascade failures - student code tested with known-working dependencies.
"""
from tinytorch.solutions.tensor import Tensor # Instructor-provided working version
from tinytorch.solutions.activations import ReLU # Instructor-provided working version
from tinytorch.solutions.tensor import Tensor # Working implementation
from tinytorch.solutions.activations import ReLU # Working implementation
from student_layers import Dense # Student's implementation
class TestLayersIntegration:
"""Test student's layers with working tensor and activation implementations."""
class TestBasicMLPipeline:
"""Test student modules in realistic ML workflows."""
def test_neural_network_forward_pass(self):
"""Test complete neural network forward pass using student's Dense layer."""
# Create network components
layer1 = Dense(input_size=4, output_size=3) # Student's implementation
"""Test complete neural network using student's Dense layer."""
# Create network with student's Dense layers
layer1 = Dense(input_size=4, output_size=3) # Student implementation
activation = ReLU() # Working implementation
layer2 = Dense(input_size=3, output_size=2) # Student's implementation
layer2 = Dense(input_size=3, output_size=2) # Student implementation
# Create input data
# Create input with working Tensor
x = Tensor([[1.0, 2.0, 3.0, 4.0]]) # Working tensor
# Forward pass through network
h1 = layer1(x) # Student's layer with working tensor
h1 = layer1(x) # Student layer with working tensor
h1_activated = activation(h1) # Working activation
output = layer2(h1_activated) # Student's layer
output = layer2(h1_activated) # Student layer
# Verify complete pipeline works
# Verify pipeline works end-to-end
assert output.shape == (1, 2), "Network should produce correct output shape"
assert isinstance(output, Tensor), "Network should produce Tensor output"
print("✅ Student's Dense layers work in complete neural network!")
def test_image_classification_pipeline(self):
"""Test realistic image classification scenario."""
# Simulate flattened MNIST image (28x28 = 784 pixels)
image_data = Tensor([np.random.randn(1, 784)])
# Create classification network
hidden_layer = Dense(784, 128) # Student's implementation
relu = ReLU() # Working activation
output_layer = Dense(128, 10) # Student's implementation (10 classes)
# Forward pass
hidden = hidden_layer(image_data)
activated = relu(hidden)
predictions = output_layer(activated)
# Verify realistic ML workflow
assert predictions.shape == (1, 10), "Should output 10 class predictions"
print("✅ Student's layers work for image classification!")
```
**Characteristics**:
- **Realistic workflows**: Test actual ML scenarios students will encounter
- **Vetted dependencies**: Use working implementations to isolate testing
- **No cascade failures**: Student's module tested independently
- **Production-like**: Mirror real-world ML development patterns
### 4. System Tests (Production Scenarios)
**Goal**: Validate performance, scalability, and robustness in production-like scenarios
**Location**: `tests/system/` directory
**Dependencies**: Complete working system
**Scope**: Performance, scalability, robustness, production workflows
**Purpose**: Ensure system works at scale and handles real-world conditions
**Example**:
```python
# tests/system/test_performance.py
"""
System Performance Tests
Tests TinyTorch performance with realistic datasets and workloads.
Ensures system can handle production-scale scenarios.
"""
import time
import psutil
import numpy as np
from tinytorch.core.tensor import Tensor
from tinytorch.core.layers import Dense
from tinytorch.core.networks import Sequential
class TestSystemPerformance:
"""Test system performance with realistic workloads."""
def test_large_batch_processing(self):
"""Test system with large batch sizes."""
# Create large network
network = Sequential([
Dense(1000, 500),
Dense(500, 250),
Dense(250, 10)
])
# Large batch (1000 samples)
large_batch = Tensor(np.random.randn(1000, 1000))
# Time the forward pass
start_time = time.time()
output = network(large_batch)
duration = time.time() - start_time
# Verify performance
assert duration < 5.0, f"Large batch processing took {duration:.2f}s, expected < 5s"
assert output.shape == (1000, 10), "Should handle large batches correctly"
print(f"✅ Processed 1000 samples in {duration:.2f}s")
def test_memory_usage(self):
"""Test memory usage with realistic workloads."""
# Monitor memory before
process = psutil.Process()
memory_before = process.memory_info().rss / 1024 / 1024 # MB
# Create and use multiple large tensors
tensors = []
for i in range(10):
tensor = Tensor(np.random.randn(1000, 1000))
tensors.append(tensor)
# Monitor memory after
memory_after = process.memory_info().rss / 1024 / 1024 # MB
memory_used = memory_after - memory_before
# Verify reasonable memory usage
assert memory_used < 500, f"Memory usage {memory_used:.1f}MB seems excessive"
print(f"✅ Memory usage: {memory_used:.1f}MB for large tensor operations")
def test_cifar10_training_simulation(self):
"""Test system with CIFAR-10 scale workload."""
# Simulate CIFAR-10 training batch
batch_size = 32
image_size = 32 * 32 * 3 # 3072 pixels
num_classes = 10
# Create realistic CNN-like network
network = Sequential([
Dense(image_size, 512),
Dense(512, 256),
Dense(256, 128),
Dense(128, num_classes)
])
# Simulate training batches
total_time = 0
num_batches = 100
for batch in range(num_batches):
# Create batch
images = Tensor(np.random.randn(batch_size, image_size))
# Forward pass
start = time.time()
predictions = network(images)
batch_time = time.time() - start
total_time += batch_time
# Verify batch processing
assert predictions.shape == (batch_size, num_classes)
avg_batch_time = total_time / num_batches
# Performance requirements
assert avg_batch_time < 0.1, f"Average batch time {avg_batch_time:.3f}s too slow"
print(f"✅ Processed {num_batches} CIFAR-10 batches, avg time: {avg_batch_time:.3f}s")
```
**Characteristics**:
- **Production scale**: Test with realistic dataset sizes and batch sizes
- **Performance monitoring**: Measure time, memory, throughput
- **Robustness testing**: Handle edge cases and stress conditions
- **Real-world scenarios**: Mirror actual ML training and inference workloads
## Testing Workflow
### For Students
1. **Develop with unit tests**: Get immediate feedback in notebooks
2. **Validate with module tests**: Run comprehensive tests before moving on
3. **Verify integration**: See how module works with broader system
4. **Optional system tests**: Understand production requirements
### For Students (Primary Path)
1. **Implement in notebook**: Write code with educational guidance
2. **Test inline**: Get immediate feedback without leaving notebook
3. **Build confidence**: See progress with visual indicators
4. **Complete module**: All inline tests pass before moving on
### For Instructors
1. **Grade module tests**: Assess individual module functionality
2. **Verify integration**: Ensure modules compose correctly
3. **Monitor system performance**: Track overall system health
4. **Provide solutions**: Maintain working implementations for integration tests
### For Instructors (Grading Path)
1. **Run module tests**: Comprehensive validation with mocks
2. **Run integration tests**: Verify cross-module functionality
3. **Grade systematically**: Clear separation of concerns
## Key Principles
### 1. **Dependency Isolation**
- Unit tests: No dependencies
- Module tests: Simple, visible mocks only
- Integration tests: Vetted solutions for dependencies
- System tests: Complete working system
### 1. **Learning-First Design**
- Prioritize student understanding over tool complexity
- Minimize context switching and cognitive overhead
- Provide immediate, encouraging feedback
- Build confidence step by step
### 2. **Clear Interfaces**
- Mocks document expected interfaces explicitly
- Students can see exactly what their module needs to provide
- Interface evolution is visible and documented
### 2. **Flow State Preservation**
- Keep students in their notebooks
- Provide instant validation
- Use visual, encouraging feedback
- Avoid workflow disruption
### 3. **Educational Value**
- Each test level serves a specific learning purpose
- Tests explain what they're checking and why
- Failures provide actionable feedback
### 3. **Comprehensive Coverage**
- Inline tests are thorough, not just quick checks
- Test functionality, edge cases, and error conditions
- Provide educational context for each test
- Explain what's being tested and why
### 4. **Professional Standards**
- Use pytest structure and best practices
- Include comprehensive edge case testing
- Mirror real-world ML development patterns
### 5. **Scalable Architecture**
- No cascade failures from broken dependencies
- Independent module development and grading
- Realistic integration without penalty for past bugs
### 4. **Professional Structure (Where Appropriate)**
- Module tests use professional pytest structure
- Integration tests mirror real-world workflows
- Maintain quality standards for grading
- Prepare students for industry practices
## Implementation Guidelines
### Mock Design Principles
1. **Minimal**: Only implement what the module actually needs
2. **Visible**: Put mocks at the top of test files with clear documentation
3. **Simple**: Easy to understand and modify
4. **Evolving**: Update mocks as interfaces grow
### Inline Test Design
1. **Comprehensive**: Test all functionality thoroughly
2. **Educational**: Explain what's being tested
3. **Visual**: Use emojis and clear progress indicators
4. **Helpful**: Provide hints when tests fail
5. **Encouraging**: Build confidence with positive feedback
### Mock Design for Module Tests
1. **Simple**: Only implement what's needed
2. **Visible**: Put mocks at top of files with clear documentation
3. **Educational**: Show exactly what interfaces are expected
4. **Minimal**: Don't over-engineer the mocks
### Test Organization
```
tests/
├── test_{module}.py # Module tests with mocks
├── integration/ # Cross-module integration tests
│ ├── test_basic_ml.py # Tensor → Layers → Networks
│ ├── test_vision.py # CNN pipelines
│ └── test_data.py # DataLoader → Networks
└── system/ # Production-scale tests
├── test_performance.py # Speed and memory
├── test_scalability.py # Large datasets
└── test_robustness.py # Error handling
modules/source/{module}/{module}_dev.py # Implementation + comprehensive inline tests
tests/test_{module}.py # Module tests with mocks (for grading)
tests/integration/ # Cross-module tests with vetted solutions
```
### CLI Integration
```bash
# Run unit tests (embedded in notebooks)
tito test --unit --module tensor
# Primary workflow - students work in notebooks
# Tests run automatically as cells execute
# Run module tests (with mocks)
tito test --module tensor
# Run integration tests
tito test --integration
# Run system tests
tito test --system
# Run all tests
tito test --all
# Instructor grading workflow
tito test --module tensor # Run module tests with mocks
tito test --integration # Run integration tests
tito test --all # Run all tests
```
## Benefits
### For Students
- **Clear progression**: Unit → Module → Integration → System
- **Immediate feedback**: Catch issues early
- **No cascade failures**: Broken dependencies don't block progress
- **Realistic experience**: See how modules work in complete systems
- **No context switching**: Stay in flow state
- **Immediate feedback**: Know instantly if code works
- **Confidence building**: Step-by-step validation
- **Clear guidance**: Helpful error messages and hints
- **Educational value**: Learn what good testing looks like
### For Instructors
- **Independent grading**: Assess modules separately
- **Clear diagnostics**: Know exactly where issues are
- **Flexible pacing**: Students can progress at different rates
- **Quality assurance**: Comprehensive validation at every level
- **Comprehensive validation**: Thorough testing for grading
- **Clear diagnostics**: Know exactly what's working/broken
- **Independent assessment**: Module tests don't depend on student's other modules
- **Professional standards**: Maintain quality without overwhelming students
### For the System
- **Maintainable**: Clear separation of concerns
- **Scalable**: Add new modules without breaking existing tests
- **Professional**: Industry-standard testing practices
- **Maintainable**: Clear separation of learning vs grading concerns
- **Scalable**: Easy to add new modules
- **Educational**: Every test serves a learning purpose
- **Practical**: Balances learning effectiveness with quality assurance
This four-tier architecture ensures comprehensive testing while maintaining educational clarity and avoiding the dependency cascade problem that plagued our earlier approaches.
## Conclusion
The inline-first testing approach prioritizes student learning effectiveness over tool complexity. Students get comprehensive testing within their learning context, building confidence and understanding step by step. Instructors maintain professional testing standards for grading while avoiding the cognitive overhead that disrupts the learning process.
**Key insight**: Context switching is expensive for learners. Keep them in flow state while ensuring comprehensive validation.