github-starred/TinyTorch
2
0
Fork 0
mirror of https://github.com/MLSysBook/TinyTorch.git synced 2026-04-28 01:38:09 -05:00
Code Issues 7 Packages Projects Releases Wiki Activity

fix: Correct testing architecture in module development best practices

Browse Source 
🔧 CRITICAL CORRECTION - Testing Architecture:

**Inline Unit Tests** (Immediate after each feature):
- Location: Inline in notebook/Python development code
- Purpose: Immediate feedback after implementing each component
- Data: Simple, hardcoded test data
- Example: assert result.shape == expected_shape right after implementation

**Module-Level Tests** (Isolated with stubs):
- Location: modules/source/XX_module/tests/test_module.py
- Purpose: Test module logic in isolation using fake/stubbed dependencies
- Data: FakeTensor, FakeLayer classes to avoid real module dependencies
- Example: Test Sequential with FakeLayer objects, not real Dense/ReLU

**Integration Tests** (Real cross-module):
- Location: tests/integration/ directory
- Purpose: Test how modules actually work together
- Data: Real implementations from tinytorch.core.*
- Example: Real Tensor → Real Dense → Real ReLU → Real Sequential

**System Tests** (Full end-to-end):
- Location: tests/system/ directory
- Purpose: Complete ML pipelines with real datasets
- Data: Real datasets and complete workflows

This correction ensures proper testing isolation and eliminates the confusion between unit tests (inline) and module-level tests (stubbed dependencies). Students learn proper testing architecture while maintaining development velocity.
This commit is contained in:
Vijay Janapa Reddi
2025-07-13 15:37:04 -04:00
parent 14b3733d08
commit 2292aab5e3
1 changed files with 221 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files

347
.cursor/rules/module-development-best-practices.mdc
View File

@@ -31,10 +31,11 @@ modules/source/XX_module_name/
```
### Test Organization (Critical)
- **Module-level tests**: `modules/source/XX_module/tests/` - ONLY unit tests for this module
- **Integration tests**: `tests/` (main directory) - Cross-module and system integration tests
- **Test after each feature**: Write unit tests immediately after implementing each component
- **No integration in module tests**: Module tests should NOT test interactions with other modules
- **Unit tests**: Inline in the notebook/Python code - immediate tests after each feature
- **Module-level tests**: `modules/source/XX_module/tests/` - test module in isolation with stubbed/fake data
- **Integration tests**: `tests/` (main directory) - real cross-module testing with actual dependencies
- **Test after each feature**: Write inline unit tests immediately after implementing each component
- **Module tests use fake data**: Module-level tests should stub out other modules with fake inputs/outputs
## Development Workflow: Test-Driven Feature Development
@@ -46,18 +47,23 @@ class ComponentName:
# Implementation here
pass
# Step 2: Use the feature (immediate inline test)
# Step 2: Use the feature (immediate inline unit test)
component = ComponentName()
result = component.method()
print(f"✅ Component working: {result}")
assert result.shape == expected_shape # Inline unit test
# Step 3: Analyze/Test the feature (unit test)
# Step 3: Analyze/Test the feature (more comprehensive inline testing)
def test_component_method():
"""Unit test for the specific method just implemented."""
"""Inline unit test for the specific method just implemented."""
component = ComponentName()
result = component.method()
assert result.shape == expected_shape
assert np.allclose(result.data, expected_data)
print("✅ Component unit test passed")
# Run the test immediately
test_component_method()
```
### 2. Build → Use → Analyze Pattern
@@ -79,73 +85,127 @@ Our best modules follow specific third-stage verbs:
- **Questions**: "How can we make this faster?", "What about memory usage?"
- **Focus**: Production-ready systems engineering
## Testing Architecture (Based on Networks Module)
## Testing Architecture (Corrected)
### Module-Level Unit Tests Pattern
### 1. Inline Unit Tests (Immediate After Each Feature)
```python
# In the notebook/Python development file
# Immediately after implementing each feature
class Sequential:
def __init__(self, layers):
self.layers = layers
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x
# INLINE UNIT TEST - Immediate testing
print("🔬 Unit Test: Sequential Network...")
network = Sequential([Dense(3, 2), ReLU()])
x = Tensor([[1.0, 2.0, 3.0]])
y = network(x)
assert y.shape == (1, 2), f"Expected shape (1, 2), got {y.shape}"
assert np.all(y.data >= 0), "ReLU output should be non-negative"
print("✅ Sequential network unit test passed")
```
### 2. Module-Level Tests (Isolated with Stubbed Data)
```python
# modules/source/04_networks/tests/test_networks.py
"""
Unit tests for the Networks module ONLY.
Tests Sequential, create_mlp, and other network components in isolation.
Module-level tests for Networks module.
Tests the module in isolation using stubbed/fake data from other modules.
"""
import pytest
from tinytorch.core.networks import Sequential, create_mlp
from tinytorch.core.layers import Dense
from tinytorch.core.activations import ReLU, Sigmoid
import numpy as np
class TestSequential:
"""Unit tests for Sequential network class."""
# Create fake/stubbed versions of dependencies
class FakeTensor:
"""Stubbed Tensor class for isolated testing."""
def __init__(self, data):
self.data = np.array(data)
self.shape = self.data.shape
class FakeLayer:
"""Stubbed Layer class for isolated testing."""
def __init__(self, output_shape):
self.output_shape = output_shape
def test_sequential_creation(self):
"""Test Sequential network creation and structure."""
def __call__(self, x):
# Return fake output with expected shape
return FakeTensor(np.random.randn(*self.output_shape))
# Import the actual module being tested
from networks_dev import Sequential
class TestSequentialIsolated:
"""Test Sequential class in isolation with fake dependencies."""
def test_sequential_with_fake_layers(self):
"""Test Sequential network with stubbed layers."""
# Use fake layers that don't depend on other modules
fake_layer1 = FakeLayer(output_shape=(1, 4))
fake_layer2 = FakeLayer(output_shape=(1, 2))
network = Sequential([fake_layer1, fake_layer2])
fake_input = FakeTensor([[1.0, 2.0, 3.0]])
output = network(fake_input)
# Test only the Sequential logic, not the actual layers
assert len(network.layers) == 2
assert output.shape == (1, 2)
def test_sequential_layer_composition(self):
"""Test that Sequential properly composes layers."""
# Test with minimal fake layers
layers = [FakeLayer((1, 3)), FakeLayer((1, 2))]
network = Sequential(layers)
assert network.layers == layers
assert len(network.layers) == 2
```
### 3. Integration Tests (Real Cross-Module Testing)
```python
# tests/integration/test_ml_pipeline.py
"""
Integration tests using real implementations from all modules.
Tests how modules actually work together in realistic scenarios.
"""
import pytest
from tinytorch.core.tensor import Tensor # Real Tensor
from tinytorch.core.layers import Dense # Real Dense layer
from tinytorch.core.activations import ReLU # Real ReLU
from tinytorch.core.networks import Sequential # Real Sequential
class TestRealMLPipeline:
"""Test real ML pipeline with actual module implementations."""
def test_tensor_to_network_integration(self):
"""Test real tensor flowing through real network."""
# Real tensor
x = Tensor([[1.0, 2.0, 3.0]])
# Real network with real layers
network = Sequential([
Dense(input_size=3, output_size=4),
ReLU(),
Dense(input_size=4, output_size=2),
Sigmoid()
Dense(input_size=4, output_size=2)
])
assert len(network.layers) == 4
assert isinstance(network.layers[0], Dense)
assert isinstance(network.layers[1], ReLU)
# Real forward pass
output = network(x)
def test_sequential_forward_pass(self):
"""Test Sequential network forward pass."""
network = Sequential([Dense(3, 2), ReLU()])
x = Tensor([[1.0, 2.0, 3.0]])
y = network(x)
assert y.shape == (1, 2)
assert np.all(y.data >= 0) # ReLU ensures non-negative
```
### Integration Tests Pattern
```python
# tests/test_integration.py (main directory)
"""
Integration tests for cross-module functionality.
Tests how modules work together in complete ML workflows.
"""
def test_complete_ml_pipeline():
"""Test complete ML pipeline: data → network → training."""
# This tests multiple modules working together
from tinytorch.core.dataloader import DataLoader, SimpleDataset
from tinytorch.core.networks import Sequential
from tinytorch.core.layers import Dense
from tinytorch.core.activations import ReLU
# Create complete pipeline
dataset = SimpleDataset(size=100, num_features=10, num_classes=3)
dataloader = DataLoader(dataset, batch_size=16)
network = Sequential([Dense(10, 5), ReLU(), Dense(5, 3)])
# Test integration
for batch_data, batch_labels in dataloader:
output = network(batch_data)
assert output.shape == (16, 3)
break
# Test real integration
assert output.shape == (1, 2)
assert isinstance(output, Tensor)
assert np.all(output.data >= 0) # ReLU ensures non-negative
```
## Student Implementation Structure
@@ -218,42 +278,33 @@ print(" Maintains batch dimension")
## Comprehensive Testing Strategy
### 1. Immediate Unit Tests (After Each Feature)
### 1. Inline Unit Tests (Immediate After Each Feature)
- **When**: Immediately after implementing each component
- **What**: Test the specific feature in isolation
- **Where**: Inline in the development notebook
- **What**: Test the specific feature with simple assertions
- **Where**: Inline in the development notebook/Python file
- **Purpose**: Ensure feature works before moving to next
- **Data**: Use simple, hardcoded test data
### 2. Module Unit Tests (Module Completion)
### 2. Module-Level Tests (Isolated Testing)
- **When**: After completing the entire module
- **What**: Comprehensive tests for all module components
- **What**: Test module components in isolation with stubbed dependencies
- **Where**: `modules/source/XX_module/tests/test_module.py`
- **Purpose**: Ensure module is ready for package export
- **Purpose**: Ensure module logic works independently of other modules
- **Data**: Use fake/stubbed data to avoid dependencies
### 3. Integration Tests (Cross-Module)
### 3. Integration Tests (Real Cross-Module Testing)
- **When**: After multiple modules are complete
- **What**: Test how modules work together
- **Where**: `tests/` (main directory)
- **What**: Test how modules work together with real implementations
- **Where**: `tests/integration/` directory
- **Purpose**: Ensure modules integrate properly in ML workflows
- **Data**: Use real implementations and actual data
### 4. Comprehensive Test Suites (Module-Level)
```python
# Based on our best modules' comprehensive testing
def run_comprehensive_module_tests():
"""Run all comprehensive module tests"""
print("🧪 Running Comprehensive Module Test Suite...")
test_results = []
test_results.append(test_basic_functionality())
test_results.append(test_edge_cases())
test_results.append(test_performance())
test_results.append(test_real_world_scenarios())
all_passed = all(test_results)
print(f"🎯 Overall Result: {'ALL TESTS PASSED! 🎉' if all_passed else 'SOME TESTS FAILED ❌'}")
return all_passed
```
### 4. System Tests (Full End-to-End)
- **When**: After major functionality is complete
- **What**: Test complete ML pipelines from data to results
- **Where**: `tests/system/` directory
- **Purpose**: Ensure entire system works in production scenarios
- **Data**: Use real datasets and complete workflows
## Progress Feedback Pattern
@@ -329,29 +380,66 @@ def implement_all_features():
def test_everything():
# Test all features together
# GOOD: Test after each feature
# GOOD: Inline unit tests after each feature
def implement_feature_1():
# Implementation
pass
# Immediate inline unit test
assert feature_1_works()
print("✅ Feature 1 working")
def test_feature_1():
# Immediate unit test
def implement_feature_2():
# Implementation
pass
# Immediate inline unit test
assert feature_2_works()
print("✅ Feature 2 working")
```
### ❌ Don't Mix Integration and Unit Tests
### ❌ Don't Use Real Dependencies in Module Tests
```python
# BAD: Integration tests in module directory
# BAD: Module tests with real dependencies
# modules/source/04_networks/tests/test_networks.py
def test_networks_with_dataloader():
# Tests networks + dataloader integration
# This should be in tests/ main directory
from tinytorch.core.layers import Dense # Real dependency
from tinytorch.core.activations import ReLU # Real dependency
def test_sequential_with_real_layers():
# This creates coupling between modules
network = Sequential([Dense(3, 2), ReLU()])
# GOOD: Unit tests only in module directory
# GOOD: Module tests with stubbed dependencies
class FakeDense:
def __call__(self, x):
return FakeTensor(np.random.randn(1, 2))
class FakeReLU:
def __call__(self, x):
return x # Simplified fake behavior
def test_sequential_with_fake_layers():
# This tests only Sequential logic
network = Sequential([FakeDense(), FakeReLU()])
```
### ❌ Don't Mix Testing Levels
```python
# BAD: Integration testing in module-level tests
# modules/source/04_networks/tests/test_networks.py
def test_sequential_creation():
# Tests only Sequential class
pass
def test_networks_with_real_dataloader():
# This should be in tests/integration/
from tinytorch.core.dataloader import DataLoader
# Testing cross-module integration
# GOOD: Keep testing levels separate
# modules/source/04_networks/tests/test_networks.py - Module-level with stubs
def test_sequential_isolated():
# Test with fake data only
# tests/integration/test_ml_pipeline.py - Integration with real modules
def test_networks_with_real_dataloader():
# Test real cross-module integration
```
### ❌ Don't Use Mock Data
@@ -373,16 +461,19 @@ class CIFAR10Dataset:
- [ ] Uses real data, not synthetic/mock data
- [ ] Includes progress feedback for long operations
- [ ] Visual feedback functions (development only, not exported)
- [ ] Unit tests after each feature implementation
- [ ] Comprehensive test suite at module completion
- [ ] Clear separation: module tests vs integration tests
- [ ] Inline unit tests after each feature implementation
- [ ] Module-level tests use stubbed/fake dependencies for isolation
- [ ] Integration tests use real cross-module implementations
- [ ] Clear separation: inline → module → integration → system testing
- [ ] Follows "Build → Use → Analyze/Test" progression
- [ ] TODO guidance includes systems thinking
- [ ] Clean separation between development and exports
### Student Experience Requirements
- [ ] Clear learning progression with immediate feedback
- [ ] Unit tests provide confidence after each feature
- [ ] Clear learning progression with immediate inline feedback
- [ ] Inline unit tests provide confidence after each feature
- [ ] Module tests demonstrate isolation and stubbing concepts
- [ ] Integration tests show real-world module interactions
- [ ] Real-world relevance and connections
- [ ] Smooth transition to next modules
- [ ] Test-driven development workflow
@@ -390,23 +481,26 @@ class CIFAR10Dataset:
## Success Metrics
**Students should be able to:**
- Test their code immediately after each feature
- Understand the difference between unit and integration tests
- Write inline unit tests immediately after each feature
- Understand the difference between inline, module, integration, and system tests
- Create stubbed/fake dependencies for module-level testing
- Explain what they built in simple terms
- Modify code to solve related problems
- Connect module concepts to real ML systems
- Debug issues by understanding the test failures
- Debug issues by understanding the different testing levels
**Modules should achieve:**
- High student engagement and completion rates
- Clear testing patterns and immediate feedback
- Clear testing patterns with immediate inline feedback
- Proper isolation in module-level tests using stubs
- Realistic integration testing with real dependencies
- Smooth progression to next modules
- Real-world relevance and production quality
- Consistent test-driven development workflow
---
**Remember**: We're teaching ML systems engineering with test-driven development. Every feature should be tested immediately, every module should have comprehensive unit tests, and integration tests should be separate. Follow the "Build → Use → Analyze/Test" cycle religiously.
**Remember**: We're teaching ML systems engineering with proper testing architecture. Inline unit tests provide immediate feedback after each feature, module-level tests use stubbed dependencies for isolation, integration tests use real cross-module implementations, and system tests validate complete workflows. Follow the "Build → Use → Analyze/Test" cycle with proper testing separation.
## Development Workflow Summary
@@ -416,7 +510,8 @@ class CIFAR10Dataset:
```bash
cd modules/source/XX_module_name/
# Work in module_name_dev.py (Jupytext format)
# Tests go in tests/test_module_name.py
# Module tests go in tests/test_module_name.py (with stubs)
# Integration tests go in tests/integration/ (with real modules)
```
2. **Feature Development Phase** (Repeat for each component)
@@ -427,17 +522,19 @@ class CIFAR10Dataset:
# Implementation
pass
# Step 2: Use the feature (immediate test)
# Step 2: Use the feature (immediate inline unit test)
component = NewComponent()
result = component.method()
assert result.shape == expected_shape # Inline unit test
print(f"✅ {component.__class__.__name__} working: {result}")
# Step 3: Analyze/Test the feature (unit test)
# Step 3: Analyze/Test the feature (more comprehensive inline testing)
def test_new_component():
component = NewComponent()
result = component.method()
assert result.shape == expected_shape
assert np.allclose(result.data, expected_data)
print("✅ Component inline unit test passed")
# Run the test immediately
test_new_component()
@@ -445,37 +542,35 @@ class CIFAR10Dataset:
3. **Module Completion Phase**
```bash
# Run comprehensive module tests
# Run module-level tests (with stubbed dependencies)
python -m pytest modules/source/XX_module_name/tests/test_module_name.py -v
# Export to package
tito package nbdev --export XX_module_name
# Test package integration
tito module test XX_module_name
```
4. **Integration Testing Phase**
```bash
# Run integration tests (cross-module)
# Run integration tests (with real cross-module dependencies)
python -m pytest tests/integration/ -v
# Run system tests
# Run system tests (full end-to-end)
python -m pytest tests/system/ -v
```
### Daily Development Rhythm
- **Morning**: Review previous day's tests, ensure all passing
- **Feature work**: Build → Use → Test for each component
- **End of day**: Run module tests, commit working features
- **Module completion**: Comprehensive testing, integration verification
- **Morning**: Review previous day's inline tests, ensure all passing
- **Feature work**: Build → Use → Inline Test for each component
- **Module work**: Create stubbed module-level tests for isolation
- **Integration work**: Test real cross-module interactions
- **End of day**: Run all test levels, commit working features
### Quality Gates
- **Feature Level**: Immediate unit test must pass
- **Module Level**: All module tests must pass before export
- **Integration Level**: Cross-module tests must pass before merge
- **System Level**: Full system tests must pass before release
- **Feature Level**: Inline unit test must pass immediately
- **Module Level**: Stubbed isolation tests must pass before export
- **Integration Level**: Real cross-module tests must pass before merge
- **System Level**: Full end-to-end tests must pass before release
This workflow ensures students build confidence incrementally while maintaining professional development standards.
This workflow ensures students understand proper testing architecture while building confidence incrementally through immediate feedback.