mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-03-11 18:33:34 -05:00
Remove temporary documentation and planning files
Deleted Category 1 temporary documentation files: - Root directory: review reports, fix summaries, implementation checklists - docs/development: testing plans, review checklists, quick references - instructor/guides: analysis reports and implementation plans - tests: testing strategy document These were completed work logs and planning documents no longer needed. All active documentation (site content, module ABOUT files, READMEs) preserved.
This commit is contained in:
Vijay Janapa Reddi
@@ -1,366 +0,0 @@
|
||||
# `tito module reset` Quick Reference Guide
|
||||
|
||||
Quick reference for the new module reset functionality.
|
||||
|
||||
---
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Full Reset (Default - Recommended)
|
||||
```bash
|
||||
tito module reset 01
|
||||
```
|
||||
**What it does:**
|
||||
1. ✅ Backs up your current work
|
||||
2. ✅ Removes package exports
|
||||
3. ✅ Restores module from git
|
||||
4. ✅ Updates progress tracking
|
||||
|
||||
**Use when:** You want to completely start over on a module
|
||||
|
||||
---
|
||||
|
||||
## Common Scenarios
|
||||
|
||||
### 1. Made Mistakes - Start Fresh
|
||||
```bash
|
||||
# Completely reset module 01
|
||||
tito module reset 01
|
||||
|
||||
# Then start working again
|
||||
tito module start 01
|
||||
```
|
||||
|
||||
### 2. Keep Package Exports (Soft Reset)
|
||||
```bash
|
||||
# Reset source but keep package
|
||||
tito module reset 01 --soft
|
||||
|
||||
# Good for: Testing different approaches without re-exporting
|
||||
```
|
||||
|
||||
### 3. View Available Backups
|
||||
```bash
|
||||
# List all backups for module 01
|
||||
tito module reset 01 --list-backups
|
||||
|
||||
# Output shows:
|
||||
# - Timestamp of backup
|
||||
# - Git hash at time of backup
|
||||
# - Number of files backed up
|
||||
```
|
||||
|
||||
### 4. Restore from Specific Backup
|
||||
```bash
|
||||
# First, list backups to find timestamp
|
||||
tito module reset 01 --list-backups
|
||||
|
||||
# Then restore from specific backup
|
||||
tito module reset 01 --restore-backup 20251112_143022
|
||||
```
|
||||
|
||||
### 5. Quick Reset (Skip Confirmation)
|
||||
```bash
|
||||
# Useful for automation or when you're sure
|
||||
tito module reset 01 --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## All Command Flags
|
||||
|
||||
| Flag | Description | Use When |
|
||||
|------|-------------|----------|
|
||||
| `--soft` | Keep package exports | You want to preserve exports |
|
||||
| `--hard` | Remove everything [DEFAULT] | You want a complete reset |
|
||||
| `--from-git` | Restore from git HEAD [DEFAULT] | Reset to repository version |
|
||||
| `--restore-backup <timestamp>` | Restore from backup | You want previous work back |
|
||||
| `--list-backups` | Show available backups | Checking backup history |
|
||||
| `--no-backup` | Skip backup creation | **DANGEROUS** - only for testing |
|
||||
| `--force` | Skip confirmations | You're absolutely sure |
|
||||
|
||||
---
|
||||
|
||||
## Safety Features
|
||||
|
||||
### Automatic Backup
|
||||
**Every reset creates a backup** (unless `--no-backup`)
|
||||
|
||||
**Backup location:**
|
||||
```
|
||||
.tito/backups/01_tensor_20251112_143022/
|
||||
├── tensor.py # Your work
|
||||
└── backup_metadata.json # Info about backup
|
||||
```
|
||||
|
||||
### Confirmation Prompts
|
||||
**Always asks before destructive actions** (unless `--force`)
|
||||
|
||||
Example:
|
||||
```
|
||||
This will reset the module to a clean state.
|
||||
Your current work will be backed up.
|
||||
|
||||
Continue with reset? (y/N):
|
||||
```
|
||||
|
||||
### Git Status Check
|
||||
**Warns if you have uncommitted changes:**
|
||||
```
|
||||
⚠️ You have uncommitted changes in your repository!
|
||||
Consider committing your work before resetting.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
### Natural Learning Flow
|
||||
```bash
|
||||
# 1. Start module
|
||||
tito module start 01
|
||||
|
||||
# 2. Work in Jupyter (save changes)
|
||||
|
||||
# 3. Try to complete
|
||||
tito module complete 01
|
||||
# Tests fail? Code doesn't work?
|
||||
|
||||
# 4. Reset and try again
|
||||
tito module reset 01
|
||||
|
||||
# 5. Start fresh with new approach
|
||||
tito module resume 01
|
||||
```
|
||||
|
||||
### Progress Tracking
|
||||
**Reset updates your progress:**
|
||||
- Removes module from "completed"
|
||||
- Clears completion date
|
||||
- Preserves backup history
|
||||
|
||||
**View progress:**
|
||||
```bash
|
||||
tito module status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Questions
|
||||
|
||||
### Q: Will I lose my work?
|
||||
**A:** No! Unless you use `--no-backup`, your work is automatically backed up to `.tito/backups/`
|
||||
|
||||
### Q: Can I undo a reset?
|
||||
**A:** Yes! Use `--restore-backup` with the timestamp of the backup you want:
|
||||
```bash
|
||||
tito module reset 01 --restore-backup 20251112_143022
|
||||
```
|
||||
|
||||
### Q: What's the difference between soft and hard reset?
|
||||
**A:**
|
||||
- **Hard (default):** Removes everything (source + package exports)
|
||||
- **Soft:** Only resets source, keeps package exports intact
|
||||
|
||||
### Q: Where are my backups stored?
|
||||
**A:** In `.tito/backups/<module_name>_<timestamp>/`
|
||||
|
||||
Each backup includes:
|
||||
- Your Python files
|
||||
- Metadata (timestamp, git hash, file list)
|
||||
|
||||
### Q: How do I see all my backups?
|
||||
**A:**
|
||||
```bash
|
||||
tito module reset 01 --list-backups
|
||||
```
|
||||
|
||||
### Q: Can I reset multiple modules?
|
||||
**A:** Not in one command, but you can run reset for each:
|
||||
```bash
|
||||
tito module reset 01
|
||||
tito module reset 02
|
||||
tito module reset 03
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples by Use Case
|
||||
|
||||
### Beginner: Made a mistake, start over
|
||||
```bash
|
||||
tito module reset 01
|
||||
# Easy! Just reset and start fresh
|
||||
```
|
||||
|
||||
### Intermediate: Want to try different approach
|
||||
```bash
|
||||
# Keep exports but reset source
|
||||
tito module reset 01 --soft
|
||||
|
||||
# Work on new approach
|
||||
tito module resume 01
|
||||
```
|
||||
|
||||
### Advanced: Restore previous working version
|
||||
```bash
|
||||
# Check what backups exist
|
||||
tito module reset 01 --list-backups
|
||||
|
||||
# Restore the one that was working
|
||||
tito module reset 01 --restore-backup 20251112_140000
|
||||
```
|
||||
|
||||
### Developer: Quick iteration during testing
|
||||
```bash
|
||||
# No confirmations, fast workflow
|
||||
tito module reset 01 --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Backup failed. Reset aborted."
|
||||
**Cause:** Can't create backup directory or files
|
||||
|
||||
**Solution:**
|
||||
1. Check `.tito/` directory permissions
|
||||
2. Ensure you have write access
|
||||
3. Try: `mkdir -p .tito/backups`
|
||||
|
||||
### "Git checkout failed"
|
||||
**Cause:** File not tracked in git or git issues
|
||||
|
||||
**Solutions:**
|
||||
1. Check if file is in git: `git ls-files modules/01_tensor/tensor.py`
|
||||
2. Commit the file first: `git add modules/01_tensor/tensor.py && git commit`
|
||||
3. Or use backup restore: `--restore-backup` instead
|
||||
|
||||
### "Export file not found"
|
||||
**Cause:** Module wasn't exported yet
|
||||
|
||||
**Solution:** This is fine! If there's no export, nothing to remove. Reset will still restore source.
|
||||
|
||||
### "Module directory not found"
|
||||
**Cause:** Invalid module number or name
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Use correct module number (01-20)
|
||||
tito module reset 01 # ✅ Correct
|
||||
tito module reset 1 # ✅ Also works (normalized to 01)
|
||||
tito module reset 99 # ❌ Invalid - no module 99
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. **Commit Before Major Changes**
|
||||
```bash
|
||||
# Before trying something risky
|
||||
git add .
|
||||
git commit -m "Working version before experiment"
|
||||
|
||||
# Now safe to reset if needed
|
||||
tito module reset 01
|
||||
```
|
||||
|
||||
### 2. **Use Soft Reset for Iteration**
|
||||
```bash
|
||||
# When experimenting with implementations
|
||||
tito module reset 01 --soft
|
||||
# Keeps package exports, faster iteration
|
||||
```
|
||||
|
||||
### 3. **Check Backups Periodically**
|
||||
```bash
|
||||
# See your backup history
|
||||
tito module reset 01 --list-backups
|
||||
|
||||
# Clean old backups manually if needed
|
||||
rm -rf .tito/backups/01_tensor_20251110_*
|
||||
```
|
||||
|
||||
### 4. **Force Only When Sure**
|
||||
```bash
|
||||
# Interactive (asks confirmation) - SAFE
|
||||
tito module reset 01
|
||||
|
||||
# Force (no confirmation) - FAST but RISKY
|
||||
tito module reset 01 --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with Other Commands
|
||||
|
||||
### Complete Workflow
|
||||
```bash
|
||||
# Start
|
||||
tito module start 01
|
||||
|
||||
# Work...
|
||||
|
||||
# Try to complete
|
||||
tito module complete 01
|
||||
|
||||
# If tests fail, reset and try again
|
||||
tito module reset 01
|
||||
|
||||
# Resume work
|
||||
tito module resume 01
|
||||
|
||||
# Complete successfully
|
||||
tito module complete 01
|
||||
|
||||
# Check progress
|
||||
tito module status
|
||||
```
|
||||
|
||||
### With Package Management
|
||||
```bash
|
||||
# Reset removes exports
|
||||
tito module reset 01
|
||||
|
||||
# Manually re-export if needed
|
||||
tito module export 01_tensor
|
||||
|
||||
# Or just use complete (exports automatically)
|
||||
tito module complete 01
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
| What You Want | Command |
|
||||
|---------------|---------|
|
||||
| Full reset | `tito module reset 01` |
|
||||
| Keep exports | `tito module reset 01 --soft` |
|
||||
| List backups | `tito module reset 01 --list-backups` |
|
||||
| Restore backup | `tito module reset 01 --restore-backup <time>` |
|
||||
| Quick reset | `tito module reset 01 --force` |
|
||||
| No backup | `tito module reset 01 --no-backup --force` |
|
||||
|
||||
---
|
||||
|
||||
## Help Text
|
||||
|
||||
```bash
|
||||
# Show reset help
|
||||
tito module reset --help
|
||||
|
||||
# Show module workflow help
|
||||
tito module --help
|
||||
|
||||
# Show all TITO commands
|
||||
tito --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Remember:** Reset is your safety net! Don't be afraid to use it when you need a fresh start. Your work is backed up automatically.
|
||||
|
||||
**Pro Tip:** Check your backups occasionally with `--list-backups` to see your learning progress history!
|
||||
@@ -1,176 +0,0 @@
|
||||
# TinyTorch Testing Quick Reference
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### **For Students**
|
||||
```bash
|
||||
# 1. Run inline tests (fast feedback)
|
||||
python modules/XX_modulename/modulename.py
|
||||
|
||||
# 2. Export to package
|
||||
tito export XX_modulename
|
||||
|
||||
# 3. Run module tests
|
||||
pytest tests/XX_modulename/ -v
|
||||
|
||||
# 4. Run critical integration tests
|
||||
pytest tests/integration/test_gradient_flow.py -v
|
||||
```
|
||||
|
||||
### **For Maintainers**
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest tests/ -v
|
||||
|
||||
# Run critical tests only
|
||||
pytest tests/integration/test_gradient_flow.py -v
|
||||
|
||||
# Run tests for specific module
|
||||
pytest tests/XX_modulename/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Test Categories Checklist
|
||||
|
||||
For each module, verify:
|
||||
|
||||
- [ ] **Core Functionality** - Does it work?
|
||||
- [ ] **Gradient Flow** - Do gradients flow? (if trainable)
|
||||
- [ ] **Integration** - Works with other modules?
|
||||
- [ ] **Shape Correctness** - Shapes handled correctly?
|
||||
- [ ] **Edge Cases** - Handles edge cases?
|
||||
- [ ] **Export/Import** - Exports correctly?
|
||||
|
||||
---
|
||||
|
||||
## 🔥 Critical Tests (Must Pass)
|
||||
|
||||
These tests **must pass** before merging:
|
||||
|
||||
1. **Gradient Flow**: `tests/integration/test_gradient_flow.py`
|
||||
- If this fails, training is broken
|
||||
|
||||
2. **Module Integration**: `tests/XX_modulename/test_progressive_integration.py`
|
||||
- Ensures module works with previous modules
|
||||
|
||||
3. **Export/Import**: Verify module exports to `tinytorch.*`
|
||||
- Students need to import from package
|
||||
|
||||
---
|
||||
|
||||
## 📊 Module Status Quick Check
|
||||
|
||||
| Module | Core | Gradients | Integration | Status |
|
||||
|--------|------|-----------|-------------|--------|
|
||||
| 01_tensor | ✅ | N/A | ✅ | ✅ Good |
|
||||
| 02_activations | ✅ | ⚠️ | ✅ | ⚠️ Missing gradients |
|
||||
| 03_layers | ✅ | ✅ | ✅ | ✅ Good |
|
||||
| 04_losses | ✅ | ✅ | ✅ | ✅ Good |
|
||||
| 05_autograd | ✅ | ✅ | ✅ | ✅ Excellent |
|
||||
| 06_optimizers | ⚠️ | ⚠️ | ✅ | ⚠️ Missing core |
|
||||
| 07_training | ✅ | ✅ | ⚠️ | ⚠️ Missing convergence |
|
||||
| 08_dataloader | ✅ | N/A | ⚠️ | ⚠️ Missing edge cases |
|
||||
| 09_spatial | ✅ | ✅ | ✅ | ✅ Good |
|
||||
| 10_tokenization | ⚠️ | N/A | ✅ | ⚠️ Missing core |
|
||||
| 11_embeddings | ✅ | ✅ | ✅ | ✅ Good |
|
||||
| 12_attention | ⚠️ | ⚠️ | ✅ | ⚠️ Missing core |
|
||||
| 13_transformers | ✅ | ✅ | ✅ | ✅ Excellent |
|
||||
| 14_profiling | ✅ | N/A | ✅ | ✅ Good |
|
||||
| 15-20 | ⚠️ | ⚠️ | ⚠️ | ⚠️ Needs assessment |
|
||||
|
||||
**Legend**: ✅ Complete | ⚠️ Gaps | ❌ Missing | N/A Not Applicable
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Priority Actions
|
||||
|
||||
### **High Priority** (Do First)
|
||||
1. Module 02_activations: Add gradient flow tests
|
||||
2. Module 06_optimizers: Add core functionality tests
|
||||
3. Module 07_training: Add convergence tests
|
||||
|
||||
### **Medium Priority** (Do Next)
|
||||
4. Module 08_dataloader: Add edge case tests
|
||||
5. Module 10_tokenization: Add core tests
|
||||
6. Module 12_attention: Add core tests
|
||||
|
||||
### **Low Priority** (Nice to Have)
|
||||
7. Modules 15-20: Assess and add tests
|
||||
8. All modules: Add export/import tests
|
||||
|
||||
---
|
||||
|
||||
## 📝 Test File Structure
|
||||
|
||||
For module `XX_modulename`:
|
||||
|
||||
```
|
||||
tests/XX_modulename/
|
||||
├── test_[modulename]_core.py # Core functionality
|
||||
├── test_gradient_flow.py # Gradient flow (if applicable)
|
||||
├── test_[modulename]_integration.py # Integration
|
||||
├── test_progressive_integration.py # Progressive integration
|
||||
├── test_edge_cases.py # Edge cases
|
||||
└── test_real_world_usage.py # Real-world usage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Common Test Patterns
|
||||
|
||||
### **Gradient Flow Test**
|
||||
```python
|
||||
def test_component_gradient_flow():
|
||||
component = Component(...)
|
||||
x = Tensor(..., requires_grad=True)
|
||||
output = component(x)
|
||||
loss = output.sum()
|
||||
loss.backward()
|
||||
|
||||
assert x.grad is not None
|
||||
for param in component.parameters():
|
||||
assert param.grad is not None
|
||||
```
|
||||
|
||||
### **Integration Test**
|
||||
```python
|
||||
def test_module_integration():
|
||||
from tinytorch.core.tensor import Tensor
|
||||
from tinytorch.core.layers import Linear
|
||||
|
||||
# Test components work together
|
||||
x = Tensor([[1.0, 2.0]])
|
||||
layer = Linear(2, 3)
|
||||
output = layer(x)
|
||||
assert output.shape == (1, 3)
|
||||
```
|
||||
|
||||
### **Edge Case Test**
|
||||
```python
|
||||
def test_edge_cases():
|
||||
# Empty input
|
||||
result = component(Tensor([]))
|
||||
|
||||
# Zero values
|
||||
result = component(Tensor([0.0]))
|
||||
|
||||
# Large values
|
||||
result = component(Tensor([1e10]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Full Documentation
|
||||
|
||||
- **Test Separation Plan**: `docs/development/TEST_SEPARATION_PLAN.md` - **START HERE** - What goes where
|
||||
- **Master Plan**: `docs/development/MASTER_TESTING_PLAN.md`
|
||||
- **Testing Architecture**: `docs/development/testing-architecture.md`
|
||||
- **Gradient Flow Strategy**: `docs/development/gradient-flow-testing-strategy.md`
|
||||
- **Comprehensive Plan**: `docs/development/comprehensive-module-testing-plan.md`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-XX
|
||||
**Quick Reference**: For detailed plans, see MASTER_TESTING_PLAN.md
|
||||
|
||||
@@ -1,705 +0,0 @@
|
||||
# Comprehensive Module Testing Plan
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
This document defines a **systematic testing strategy** for all TinyTorch modules. It identifies what critical checks each module needs, ensuring both students and maintainers can catch issues early and build robust systems.
|
||||
|
||||
**Key Principle**: Every module needs tests that validate:
|
||||
1. **Correctness** - Does it work as intended?
|
||||
2. **Integration** - Does it work with other modules?
|
||||
3. **Robustness** - Does it handle edge cases?
|
||||
4. **Usability** - Can students actually use it?
|
||||
|
||||
---
|
||||
|
||||
## 📊 Test Categories: What to Test
|
||||
|
||||
### **Category 1: Core Functionality** ✅
|
||||
**Purpose**: Verify the module does what it's supposed to do
|
||||
|
||||
**Checks**:
|
||||
- ✅ Forward pass correctness
|
||||
- ✅ Output shapes match expectations
|
||||
- ✅ Mathematical correctness (compare to reference implementations)
|
||||
- ✅ API correctness (methods exist, signatures correct)
|
||||
- ✅ Parameter initialization (if applicable)
|
||||
|
||||
**Example**: For `03_layers`:
|
||||
- Linear layer computes `output = input @ weight + bias` correctly
|
||||
- Output shape is `(batch, out_features)` when input is `(batch, in_features)`
|
||||
- Weight and bias are initialized properly
|
||||
|
||||
---
|
||||
|
||||
### **Category 2: Gradient Flow** 🔥
|
||||
**Purpose**: Verify gradients flow correctly (critical for training)
|
||||
|
||||
**Checks**:
|
||||
- ✅ Gradients exist after backward pass
|
||||
- ✅ Gradients are non-zero (not all zeros)
|
||||
- ✅ All trainable parameters receive gradients
|
||||
- ✅ Gradient shapes match parameter shapes
|
||||
- ✅ Gradients flow through the component correctly
|
||||
|
||||
**Example**: For `02_activations`:
|
||||
- ReLU preserves `requires_grad` flag
|
||||
- Backward pass computes correct gradients
|
||||
- Gradient is 0 for negative inputs, 1 for positive inputs
|
||||
|
||||
**Modules That Need This**: All modules with trainable parameters or that process gradients
|
||||
- ✅ 02_activations, 03_layers, 04_losses, 05_autograd, 06_optimizers, 07_training, 09_spatial, 11_embeddings, 12_attention, 13_transformers
|
||||
|
||||
**Modules That Don't Need This**: Modules that don't process gradients
|
||||
- ❌ 01_tensor (foundation, no gradients yet), 08_dataloader (data only), 10_tokenization (text processing), 14_profiling (analysis), 15_quantization (post-training), 16_compression (post-training), 17_memoization (caching), 18_acceleration (optimization), 19_benchmarking (evaluation)
|
||||
|
||||
---
|
||||
|
||||
### **Category 3: Integration with Previous Modules** 🔗
|
||||
**Purpose**: Verify module N works with modules 1 through N-1
|
||||
|
||||
**Checks**:
|
||||
- ✅ Imports from previous modules work
|
||||
- ✅ Components from previous modules integrate correctly
|
||||
- ✅ Data flows correctly through the stack
|
||||
- ✅ No breaking changes to previous modules
|
||||
|
||||
**Example**: For `07_training`:
|
||||
- Uses Tensor (01), Layers (03), Losses (04), Autograd (05), Optimizers (06)
|
||||
- All components work together in a training loop
|
||||
- Training loop actually trains (loss decreases)
|
||||
|
||||
**All Modules Need This**: Every module should test integration with previous modules
|
||||
|
||||
---
|
||||
|
||||
### **Category 4: Shape Correctness** 📐
|
||||
**Purpose**: Verify shapes are handled correctly (common source of bugs)
|
||||
|
||||
**Checks**:
|
||||
- ✅ Output shapes match expected dimensions
|
||||
- ✅ Broadcasting works correctly
|
||||
- ✅ Reshape operations preserve data
|
||||
- ✅ Batch dimensions handled correctly
|
||||
- ✅ Edge cases (empty tensors, single samples, etc.)
|
||||
|
||||
**Example**: For `09_spatial`:
|
||||
- Conv2d output shape: `(batch, out_channels, height_out, width_out)`
|
||||
- MaxPool2d reduces spatial dimensions correctly
|
||||
- Shapes work with Linear layers downstream
|
||||
|
||||
**Modules That Need This**: All modules that transform shapes
|
||||
- ✅ 01_tensor, 03_layers, 09_spatial, 11_embeddings, 12_attention, 13_transformers
|
||||
|
||||
---
|
||||
|
||||
### **Category 5: Edge Cases & Error Handling** ⚠️
|
||||
**Purpose**: Verify robustness and helpful error messages
|
||||
|
||||
**Checks**:
|
||||
- ✅ Handles empty inputs gracefully
|
||||
- ✅ Handles zero values correctly
|
||||
- ✅ Handles very large/small values
|
||||
- ✅ Provides helpful error messages for invalid inputs
|
||||
- ✅ Handles NaN/Inf correctly
|
||||
- ✅ Handles out-of-bounds indices
|
||||
|
||||
**Example**: For `08_dataloader`:
|
||||
- Empty dataset handled gracefully
|
||||
- Batch size larger than dataset handled correctly
|
||||
- Invalid indices raise clear error messages
|
||||
|
||||
**All Modules Need This**: Every module should handle edge cases
|
||||
|
||||
---
|
||||
|
||||
### **Category 6: Numerical Stability** 🔢
|
||||
**Purpose**: Verify numerical correctness and stability
|
||||
|
||||
**Checks**:
|
||||
- ✅ No NaN values in outputs
|
||||
- ✅ No Inf values in outputs
|
||||
- ✅ Numerical precision is acceptable
|
||||
- ✅ Operations are numerically stable
|
||||
- ✅ Compare to reference implementations (NumPy, PyTorch)
|
||||
|
||||
**Example**: For `02_activations`:
|
||||
- Sigmoid doesn't overflow for large inputs
|
||||
- Softmax is numerically stable (uses log-sum-exp trick)
|
||||
- No NaN/Inf in outputs
|
||||
|
||||
**Modules That Need This**: Modules with numerical operations
|
||||
- ✅ 01_tensor, 02_activations, 03_layers, 04_losses, 05_autograd, 09_spatial, 11_embeddings, 12_attention, 13_transformers
|
||||
|
||||
---
|
||||
|
||||
### **Category 7: Memory & Performance** ⚡
|
||||
**Purpose**: Verify reasonable performance (not exhaustive, but catch major issues)
|
||||
|
||||
**Checks**:
|
||||
- ✅ No memory leaks
|
||||
- ✅ Operations complete in reasonable time
|
||||
- ✅ Memory usage is reasonable
|
||||
- ✅ Can handle realistic batch sizes
|
||||
|
||||
**Example**: For `13_transformers`:
|
||||
- Forward pass completes in reasonable time for small models
|
||||
- Memory usage scales linearly with batch size
|
||||
- No memory leaks across multiple forward passes
|
||||
|
||||
**Modules That Need This**: Modules with performance-sensitive operations
|
||||
- ✅ 05_autograd, 09_spatial, 12_attention, 13_transformers, 14_profiling, 18_acceleration, 19_benchmarking
|
||||
|
||||
---
|
||||
|
||||
### **Category 8: Real-World Usage** 🌍
|
||||
**Purpose**: Verify the module works in realistic scenarios
|
||||
|
||||
**Checks**:
|
||||
- ✅ Can solve the intended problem
|
||||
- ✅ Works with real datasets (if applicable)
|
||||
- ✅ Matches expected behavior from documentation
|
||||
- ✅ Can be used in production-like scenarios
|
||||
|
||||
**Example**: For `07_training`:
|
||||
- Can train a simple model on real data
|
||||
- Loss decreases over epochs
|
||||
- Model actually learns (accuracy improves)
|
||||
|
||||
**Modules That Need This**: All modules should have at least one real-world usage test
|
||||
|
||||
---
|
||||
|
||||
### **Category 9: Export/Import Correctness** 📦
|
||||
**Purpose**: Verify code exports correctly and can be imported
|
||||
|
||||
**Checks**:
|
||||
- ✅ Code exports to `tinytorch/` correctly
|
||||
- ✅ Can import from `tinytorch.*` package
|
||||
- ✅ Exported API matches module API
|
||||
- ✅ No import errors
|
||||
|
||||
**All Modules Need This**: Every module should test export/import
|
||||
|
||||
---
|
||||
|
||||
### **Category 10: API Consistency** 🔌
|
||||
**Purpose**: Verify API matches conventions and is usable
|
||||
|
||||
**Checks**:
|
||||
- ✅ Methods have expected names
|
||||
- ✅ Parameters match expected signatures
|
||||
- ✅ Return types are consistent
|
||||
- ✅ Follows TinyTorch conventions
|
||||
|
||||
**All Modules Need This**: Every module should test API consistency
|
||||
|
||||
---
|
||||
|
||||
## 📋 Module-by-Module Testing Plan
|
||||
|
||||
### **Module 01: Tensor** (Foundation)
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: All operations work (add, mul, matmul, etc.)
|
||||
- ❌ Gradient Flow: Not applicable (no gradients yet)
|
||||
- ❌ Integration: No previous modules
|
||||
- ✅ Shape Correctness: Broadcasting, reshaping, indexing
|
||||
- ✅ Edge Cases: Empty tensors, zero values, large arrays
|
||||
- ✅ Numerical Stability: Precision, overflow handling
|
||||
- ⚠️ Memory & Performance: Large tensor operations
|
||||
- ✅ Real-World Usage: Can build neural networks with tensors
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.tensor`
|
||||
- ✅ API Consistency: Matches NumPy-like API
|
||||
|
||||
**Test Files**:
|
||||
- `tests/01_tensor/test_tensor_core.py` - Core functionality
|
||||
- `tests/01_tensor/test_tensor_integration.py` - Integration with NumPy
|
||||
- `tests/01_tensor/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
---
|
||||
|
||||
### **Module 02: Activations**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Forward pass correctness
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - All activations preserve gradients
|
||||
- ✅ Integration: Works with Tensor (01)
|
||||
- ✅ Shape Correctness: Output shape matches input shape
|
||||
- ✅ Edge Cases: Large values, zero values, negative values
|
||||
- ✅ **Numerical Stability**: **CRITICAL** - No overflow/underflow
|
||||
- ⚠️ Memory & Performance: Fast forward/backward passes
|
||||
- ✅ Real-World Usage: Can use in neural networks
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.activations`
|
||||
- ✅ API Consistency: All activations have same interface
|
||||
|
||||
**Test Files**:
|
||||
- `tests/02_activations/test_activations_core.py` - Core functionality
|
||||
- `tests/02_activations/test_gradient_flow.py` - **MISSING** - Gradient flow tests
|
||||
- `tests/02_activations/test_activations_integration.py` - Integration
|
||||
- `tests/02_activations/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Gap**: Missing comprehensive gradient flow tests for all activations
|
||||
|
||||
---
|
||||
|
||||
### **Module 03: Layers**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Forward pass, parameter initialization
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - All layers compute gradients correctly
|
||||
- ✅ Integration: Works with Tensor (01), Activations (02)
|
||||
- ✅ **Shape Correctness**: **CRITICAL** - Output shapes match expectations
|
||||
- ✅ Edge Cases: Zero inputs, single samples, large batches
|
||||
- ✅ Numerical Stability: No NaN/Inf in outputs
|
||||
- ⚠️ Memory & Performance: Reasonable memory usage
|
||||
- ✅ Real-World Usage: Can build neural networks
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.layers`
|
||||
- ✅ API Consistency: All layers follow Module interface
|
||||
|
||||
**Test Files**:
|
||||
- `tests/03_layers/test_layers_core.py` - Core functionality
|
||||
- `tests/03_layers/test_layers_integration.py` - Integration
|
||||
- `tests/03_layers/test_layers_networks_integration.py` - Network integration
|
||||
- `tests/03_layers/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Gap**: Missing gradient flow tests for Dropout, LayerNorm (if in layers module)
|
||||
|
||||
---
|
||||
|
||||
### **Module 04: Losses**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Loss computation correctness
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Loss functions compute gradients
|
||||
- ✅ Integration: Works with Tensor (01), Layers (03)
|
||||
- ✅ Shape Correctness: Handles different batch sizes
|
||||
- ✅ Edge Cases: Perfect predictions, zero loss, large losses
|
||||
- ✅ **Numerical Stability**: **CRITICAL** - Log operations stable
|
||||
- ⚠️ Memory & Performance: Efficient computation
|
||||
- ✅ Real-World Usage: Can use in training loops
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.losses`
|
||||
- ✅ API Consistency: All losses have same interface
|
||||
|
||||
**Test Files**:
|
||||
- `tests/04_losses/test_dense_layer.py` - Layer tests
|
||||
- `tests/04_losses/test_dense_integration.py` - Integration
|
||||
- `tests/04_losses/test_network_capability.py` - Network capability
|
||||
- `tests/04_losses/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Status**: Good coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 05: Autograd**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Forward/backward pass correctness
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Gradients computed correctly
|
||||
- ✅ Integration: Works with all previous modules
|
||||
- ✅ Shape Correctness: Gradient shapes match parameter shapes
|
||||
- ✅ **Edge Cases**: **CRITICAL** - Broadcasting, reshape, chain rule
|
||||
- ✅ **Numerical Stability**: **CRITICAL** - Gradient computation stable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - No memory leaks
|
||||
- ✅ Real-World Usage: Can train models
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.autograd`
|
||||
- ✅ API Consistency: Matches PyTorch-like API
|
||||
|
||||
**Test Files**:
|
||||
- `tests/05_autograd/test_gradient_flow.py` - Gradient flow ✅
|
||||
- `tests/05_autograd/test_batched_matmul_backward.py` - Batched operations
|
||||
- `tests/05_autograd/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Status**: Excellent coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 06: Optimizers**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Parameter updates work
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Optimizers use gradients correctly
|
||||
- ✅ Integration: Works with Autograd (05), Layers (03)
|
||||
- ✅ Shape Correctness: Parameter shapes preserved
|
||||
- ✅ Edge Cases: Zero gradients, very small/large learning rates
|
||||
- ✅ Numerical Stability: Updates don't cause overflow
|
||||
- ⚠️ Memory & Performance: Efficient updates
|
||||
- ✅ Real-World Usage: Can train models
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.optimizers`
|
||||
- ✅ API Consistency: All optimizers have same interface
|
||||
|
||||
**Test Files**:
|
||||
- `tests/06_optimizers/test_progressive_integration.py` - Progressive integration
|
||||
- `tests/06_optimizers/test_cnn_networks_integration.py` - CNN integration
|
||||
- `tests/06_optimizers/test_cnn_pipeline_integration.py` - Pipeline integration
|
||||
|
||||
**Gap**: Missing dedicated optimizer functionality tests
|
||||
|
||||
---
|
||||
|
||||
### **Module 07: Training**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Training loops work
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Full training stack gradients work
|
||||
- ✅ Integration: Works with all previous modules (01-06)
|
||||
- ✅ Shape Correctness: Batch handling, loss aggregation
|
||||
- ✅ Edge Cases: Single sample, empty batches, convergence
|
||||
- ✅ Numerical Stability: Training doesn't diverge
|
||||
- ⚠️ Memory & Performance: Reasonable training speed
|
||||
- ✅ **Real-World Usage**: **CRITICAL** - Can actually train models
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.training`
|
||||
- ✅ API Consistency: Training API is usable
|
||||
|
||||
**Test Files**:
|
||||
- `tests/07_training/test_autograd_integration.py` - Autograd integration
|
||||
- `tests/07_training/test_tensor_autograd_integration.py` - Tensor integration
|
||||
- `tests/07_training/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Gap**: Missing end-to-end training convergence tests
|
||||
|
||||
---
|
||||
|
||||
### **Module 08: Dataloader**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Batching, shuffling, iteration work
|
||||
- ❌ Gradient Flow: Not applicable (data only)
|
||||
- ✅ Integration: Works with Tensor (01), doesn't break gradients
|
||||
- ✅ Shape Correctness: Batch shapes correct
|
||||
- ✅ **Edge Cases**: **CRITICAL** - Empty dataset, batch > dataset size
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Efficient data loading
|
||||
- ✅ Real-World Usage: Can load real datasets
|
||||
- ✅ Export/Import: Exports to `tinytorch.data.dataloader`
|
||||
- ✅ API Consistency: Iterator interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/08_dataloader/test_autograd_core.py` - Core functionality
|
||||
- `tests/08_dataloader/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Gap**: Missing comprehensive edge case tests, missing tests that verify dataloader doesn't break gradient flow
|
||||
|
||||
---
|
||||
|
||||
### **Module 09: Spatial (CNNs)**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Conv2d, Pooling work correctly
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Conv2d gradients work
|
||||
- ✅ Integration: Works with Tensor (01), Layers (03), Autograd (05)
|
||||
- ✅ **Shape Correctness**: **CRITICAL** - Output shapes match expectations
|
||||
- ✅ Edge Cases: Kernel size > image size, stride > kernel size
|
||||
- ✅ Numerical Stability: No NaN/Inf in outputs
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Efficient convolution
|
||||
- ✅ Real-World Usage: Can build CNNs
|
||||
- ✅ Export/Import: Exports to `tinytorch.core.spatial`
|
||||
- ✅ API Consistency: Matches PyTorch Conv2d API
|
||||
|
||||
**Test Files**:
|
||||
- `tests/integration/test_cnn_integration.py` - CNN integration ✅
|
||||
- `tests/09_spatial/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Status**: Good coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 10: Tokenization**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Tokenization works correctly
|
||||
- ❌ Gradient Flow: Not applicable (text processing)
|
||||
- ✅ Integration: Works with Tensor (01)
|
||||
- ✅ Shape Correctness: Token sequences have correct shapes
|
||||
- ✅ Edge Cases: Empty strings, special characters, long sequences
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ⚠️ Memory & Performance: Efficient tokenization
|
||||
- ✅ Real-World Usage: Can tokenize real text
|
||||
- ✅ Export/Import: Exports to `tinytorch.text.tokenization`
|
||||
- ✅ API Consistency: Tokenizer interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/10_tokenization/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Gap**: Missing comprehensive tokenization tests
|
||||
|
||||
---
|
||||
|
||||
### **Module 11: Embeddings**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Embedding lookup works
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Embedding gradients work
|
||||
- ✅ Integration: Works with Tokenization (10), Tensor (01)
|
||||
- ✅ Shape Correctness: Embedding shapes correct
|
||||
- ✅ Edge Cases: Out-of-vocab tokens, zero embeddings
|
||||
- ✅ Numerical Stability: Embedding values reasonable
|
||||
- ⚠️ Memory & Performance: Efficient embedding lookup
|
||||
- ✅ Real-World Usage: Can embed real text
|
||||
- ✅ Export/Import: Exports to `tinytorch.text.embeddings`
|
||||
- ✅ API Consistency: Embedding interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/11_embeddings/test_training_integration.py` - Training integration
|
||||
- `tests/11_embeddings/test_ml_pipeline.py` - ML pipeline
|
||||
- `tests/11_embeddings/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Status**: Good coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 12: Attention**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Attention mechanism works
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Attention gradients work
|
||||
- ✅ Integration: Works with Embeddings (11), Tensor (01)
|
||||
- ✅ **Shape Correctness**: **CRITICAL** - Attention output shapes correct
|
||||
- ✅ Edge Cases: Causal masking, padding masks, long sequences
|
||||
- ✅ **Numerical Stability**: **CRITICAL** - Softmax stability
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - O(n²) complexity handled
|
||||
- ✅ Real-World Usage: Can use in transformers
|
||||
- ✅ Export/Import: Exports to `tinytorch.models.attention`
|
||||
- ✅ API Consistency: Attention interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/12_attention/test_progressive_integration.py` - Progressive integration
|
||||
- `tests/12_attention/test_compression_integration.py` - Compression integration
|
||||
|
||||
**Gap**: Missing dedicated attention mechanism tests
|
||||
|
||||
---
|
||||
|
||||
### **Module 13: Transformers**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Transformer blocks work
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Full transformer gradients work
|
||||
- ✅ Integration: Works with all previous modules (01-12)
|
||||
- ✅ **Shape Correctness**: **CRITICAL** - Transformer output shapes correct
|
||||
- ✅ Edge Cases: Variable sequence lengths, masking
|
||||
- ✅ **Numerical Stability**: **CRITICAL** - LayerNorm, attention stability
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Efficient transformer forward/backward
|
||||
- ✅ **Real-World Usage**: **CRITICAL** - Can train transformers
|
||||
- ✅ Export/Import: Exports to `tinytorch.models.transformer`
|
||||
- ✅ API Consistency: Transformer API works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/13_transformers/test_transformer_gradient_flow.py` - Gradient flow ✅
|
||||
- `tests/13_transformers/test_training_simple.py` - Training tests
|
||||
- `tests/13_transformers/test_kernels_integration.py` - Kernel integration
|
||||
- `tests/13_transformers/test_progressive_integration.py` - Progressive integration
|
||||
|
||||
**Status**: Excellent coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 14: Profiling**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Profiling works correctly
|
||||
- ❌ Gradient Flow: Not applicable (analysis only)
|
||||
- ✅ Integration: Works with all modules
|
||||
- ⚠️ Shape Correctness: Not applicable
|
||||
- ✅ Edge Cases: Empty profiles, very fast operations
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Profiling overhead minimal
|
||||
- ✅ Real-World Usage: Can profile real models
|
||||
- ✅ Export/Import: Exports to `tinytorch.profiling`
|
||||
- ✅ API Consistency: Profiler interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/14_profiling/test_progressive_integration.py` - Progressive integration
|
||||
- `tests/14_profiling/test_benchmarking_integration.py` - Benchmarking integration
|
||||
- `tests/14_profiling/test_kv_cache_integration.py` - KV cache integration
|
||||
|
||||
**Status**: Good coverage
|
||||
|
||||
---
|
||||
|
||||
### **Module 15: Quantization**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Quantization works correctly
|
||||
- ⚠️ Gradient Flow: May need gradient tests if quantization-aware training
|
||||
- ✅ Integration: Works with trained models
|
||||
- ✅ Shape Correctness: Quantized model shapes preserved
|
||||
- ✅ Edge Cases: Extreme values, zero values
|
||||
- ✅ Numerical Stability: Quantization doesn't cause overflow
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Memory reduction achieved
|
||||
- ✅ Real-World Usage: Can quantize real models
|
||||
- ✅ Export/Import: Exports to `tinytorch.quantization`
|
||||
- ✅ API Consistency: Quantization API works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/15_memoization/test_progressive_integration.py` - Progressive integration
|
||||
- `tests/15_memoization/test_mlops_integration.py` - MLOps integration
|
||||
- `tests/15_memoization/test_tinygpt_integration.py` - TinyGPT integration
|
||||
|
||||
**Gap**: Missing quantization-specific tests
|
||||
|
||||
---
|
||||
|
||||
### **Module 16: Compression**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Compression works correctly
|
||||
- ⚠️ Gradient Flow: May need gradient tests if compression-aware training
|
||||
- ✅ Integration: Works with trained models
|
||||
- ✅ Shape Correctness: Compressed model shapes handled
|
||||
- ✅ Edge Cases: Already sparse models, extreme compression
|
||||
- ✅ Numerical Stability: Compression doesn't cause instability
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Compression ratio achieved
|
||||
- ✅ Real-World Usage: Can compress real models
|
||||
- ✅ Export/Import: Exports to `tinytorch.compression`
|
||||
- ✅ API Consistency: Compression API works
|
||||
|
||||
**Test Files**: Need to check
|
||||
|
||||
**Gap**: Unknown - needs assessment
|
||||
|
||||
---
|
||||
|
||||
### **Module 17: Memoization**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Caching works correctly
|
||||
- ❌ Gradient Flow: Not applicable (caching only)
|
||||
- ✅ Integration: Works with all modules
|
||||
- ⚠️ Shape Correctness: Not applicable
|
||||
- ✅ Edge Cases: Cache invalidation, memory limits
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Caching improves performance
|
||||
- ✅ Real-World Usage: Can cache real computations
|
||||
- ✅ Export/Import: Exports to `tinytorch.memoization`
|
||||
- ✅ API Consistency: Cache interface works
|
||||
|
||||
**Test Files**:
|
||||
- `tests/17_compression/` - Need to check
|
||||
|
||||
**Gap**: Unknown - needs assessment
|
||||
|
||||
---
|
||||
|
||||
### **Module 18: Acceleration**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Acceleration works correctly
|
||||
- ❌ Gradient Flow: Not applicable (optimization only)
|
||||
- ✅ Integration: Works with all modules
|
||||
- ⚠️ Shape Correctness: Not applicable
|
||||
- ✅ Edge Cases: Already optimized code, edge cases
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Speedup achieved
|
||||
- ✅ Real-World Usage: Can accelerate real models
|
||||
- ✅ Export/Import: Exports to `tinytorch.acceleration`
|
||||
- ✅ API Consistency: Acceleration API works
|
||||
|
||||
**Test Files**: Need to check
|
||||
|
||||
**Gap**: Unknown - needs assessment
|
||||
|
||||
---
|
||||
|
||||
### **Module 19: Benchmarking**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Benchmarking works correctly
|
||||
- ❌ Gradient Flow: Not applicable (evaluation only)
|
||||
- ✅ Integration: Works with all modules
|
||||
- ⚠️ Shape Correctness: Not applicable
|
||||
- ✅ Edge Cases: Very fast/slow operations, edge cases
|
||||
- ⚠️ Numerical Stability: Not applicable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - Benchmarking overhead minimal
|
||||
- ✅ Real-World Usage: Can benchmark real models
|
||||
- ✅ Export/Import: Exports to `tinytorch.benchmarking`
|
||||
- ✅ API Consistency: Benchmarking API works
|
||||
|
||||
**Test Files**: Need to check
|
||||
|
||||
**Gap**: Unknown - needs assessment
|
||||
|
||||
---
|
||||
|
||||
### **Module 20: Capstone**
|
||||
**Critical Checks Needed**:
|
||||
- ✅ Core Functionality: Complete system works
|
||||
- ✅ **Gradient Flow**: **CRITICAL** - Full system gradients work
|
||||
- ✅ Integration: Works with ALL modules (01-19)
|
||||
- ✅ Shape Correctness: End-to-end shapes correct
|
||||
- ✅ Edge Cases: All edge cases from all modules
|
||||
- ✅ Numerical Stability: Full system stable
|
||||
- ✅ **Memory & Performance**: **CRITICAL** - System performance acceptable
|
||||
- ✅ **Real-World Usage**: **CRITICAL** - Can train and use TinyGPT
|
||||
- ✅ Export/Import: Exports to `tinytorch.applications.tinygpt`
|
||||
- ✅ API Consistency: Complete API works
|
||||
|
||||
**Test Files**: Need to check
|
||||
|
||||
**Gap**: Unknown - needs assessment
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Priority Implementation Plan
|
||||
|
||||
### **Phase 1: Critical Gaps** (Must Fix)
|
||||
1. **Module 02_activations**: Add comprehensive gradient flow tests
|
||||
2. **Module 08_dataloader**: Add edge case tests, verify doesn't break gradients
|
||||
3. **Module 06_optimizers**: Add dedicated optimizer functionality tests
|
||||
4. **Module 07_training**: Add end-to-end convergence tests
|
||||
|
||||
### **Phase 2: Important Gaps** (Should Fix)
|
||||
5. **Module 03_layers**: Add gradient flow tests for Dropout, LayerNorm
|
||||
6. **Module 10_tokenization**: Add comprehensive tokenization tests
|
||||
7. **Module 12_attention**: Add dedicated attention mechanism tests
|
||||
8. **All modules**: Add export/import correctness tests
|
||||
|
||||
### **Phase 3: Nice to Have** (Can Fix)
|
||||
9. **All modules**: Add numerical stability tests
|
||||
10. **All modules**: Add memory/performance tests
|
||||
11. **All modules**: Add real-world usage tests
|
||||
|
||||
---
|
||||
|
||||
## 📝 Test File Naming Convention
|
||||
|
||||
For each module `XX_modulename`, create:
|
||||
|
||||
```
|
||||
tests/XX_modulename/
|
||||
├── test_[modulename]_core.py # Core functionality
|
||||
├── test_gradient_flow.py # Gradient flow (if applicable)
|
||||
├── test_[modulename]_integration.py # Integration with previous modules
|
||||
├── test_progressive_integration.py # Progressive integration (module N with 1-N-1)
|
||||
├── test_edge_cases.py # Edge cases and error handling
|
||||
├── test_numerical_stability.py # Numerical stability (if applicable)
|
||||
└── test_real_world_usage.py # Real-world usage scenarios
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Success Criteria
|
||||
|
||||
A module has **complete test coverage** when:
|
||||
|
||||
1. ✅ Core functionality tests pass
|
||||
2. ✅ Gradient flow tests pass (if applicable)
|
||||
3. ✅ Integration tests pass
|
||||
4. ✅ Progressive integration tests pass
|
||||
5. ✅ Edge case tests pass
|
||||
6. ✅ Export/import tests pass
|
||||
7. ✅ At least one real-world usage test passes
|
||||
|
||||
---
|
||||
|
||||
## 🎓 For Students
|
||||
|
||||
This testing plan helps you:
|
||||
- **Understand what to test**: Clear categories of what matters
|
||||
- **Catch bugs early**: Test as you build
|
||||
- **Learn best practices**: See how professional ML systems are tested
|
||||
- **Build confidence**: Know your code works correctly
|
||||
|
||||
---
|
||||
|
||||
## 🔧 For Maintainers
|
||||
|
||||
This testing plan helps you:
|
||||
- **Catch regressions**: Comprehensive tests catch breaking changes
|
||||
- **Ensure quality**: All modules meet quality standards
|
||||
- **Document behavior**: Tests document expected behavior
|
||||
- **Maintain system**: Keep TinyTorch robust as it evolves
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-XX
|
||||
**Status**: Comprehensive plan complete, implementation in progress
|
||||
**Priority**: High - Systematic testing ensures robust system
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,341 +0,0 @@
|
||||
# Gradient Flow Testing Strategy
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
Gradient flow tests are **critical** for TinyTorch because they validate that the autograd system works correctly end-to-end. A component might work perfectly in isolation, but if gradients don't flow through it, training will fail silently.
|
||||
|
||||
**Key Principle**: Every module that has trainable parameters or processes gradients should have gradient flow tests.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Current Gradient Flow Test Coverage
|
||||
|
||||
### **Comprehensive Integration Tests** ✅
|
||||
- `tests/integration/test_gradient_flow.py` - **CRITICAL**: Tests entire training stack
|
||||
- Basic tensor operations
|
||||
- Layer gradients (Linear)
|
||||
- Activation gradients (Sigmoid, ReLU, Tanh)
|
||||
- Loss gradients (MSE, BCE, CrossEntropy)
|
||||
- Optimizer integration (SGD, AdamW)
|
||||
- Full training loops
|
||||
- Edge cases
|
||||
|
||||
- `tests/test_gradient_flow.py` - Comprehensive suite
|
||||
- Simple linear networks
|
||||
- MLP networks
|
||||
- CNN networks
|
||||
- Gradient accumulation
|
||||
|
||||
### **Module-Specific Gradient Tests** ✅
|
||||
- `tests/05_autograd/test_gradient_flow.py` - Autograd operations
|
||||
- Arithmetic operations (add, sub, mul, div)
|
||||
- GELU activation
|
||||
- LayerNorm operations
|
||||
- Reshape operations
|
||||
|
||||
- `tests/13_transformers/test_transformer_gradient_flow.py` - Transformer components
|
||||
- MultiHeadAttention gradients
|
||||
- LayerNorm gradients
|
||||
- MLP gradients
|
||||
- Full GPT model gradients
|
||||
- Attention masking gradients
|
||||
|
||||
- `tests/integration/test_cnn_integration.py` - CNN components
|
||||
- Conv2d gradient flow
|
||||
- Complete CNN forward/backward
|
||||
- Pooling operations
|
||||
|
||||
- `tests/regression/test_nlp_components_gradient_flow.py` - NLP components
|
||||
- Tokenization
|
||||
- Embeddings
|
||||
- Positional encoding
|
||||
- Attention mechanisms
|
||||
- Full GPT model
|
||||
|
||||
### **System-Level Tests** ✅
|
||||
- `tests/system/test_gradients.py` - System validation
|
||||
- Gradient existence in single layers
|
||||
- Gradient existence in deep networks
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Gap Analysis: What's Missing?
|
||||
|
||||
### **Module-by-Module Coverage**
|
||||
|
||||
| Module | Has Gradient Flow Tests? | Status | Notes |
|
||||
|--------|-------------------------|--------|-------|
|
||||
| 01_tensor | ✅ Partial | Good | Basic operations covered in integration tests |
|
||||
| 02_activations | ⚠️ Partial | Needs Work | Some activations tested, not all |
|
||||
| 03_layers | ✅ Good | Good | Linear layer well tested |
|
||||
| 04_losses | ✅ Good | Good | All major losses tested |
|
||||
| 05_autograd | ✅ Excellent | Complete | Comprehensive autograd tests |
|
||||
| 06_optimizers | ✅ Good | Good | Optimizer integration tested |
|
||||
| 07_training | ✅ Good | Good | Training loops tested |
|
||||
| 08_dataloader | ❌ Missing | **Gap** | No gradient flow tests |
|
||||
| 09_spatial | ✅ Good | Good | CNN tests cover Conv2d |
|
||||
| 10_tokenization | ✅ Partial | Good | Covered in NLP regression tests |
|
||||
| 11_embeddings | ✅ Good | Good | Covered in NLP regression tests |
|
||||
| 12_attention | ✅ Good | Good | Covered in transformer tests |
|
||||
| 13_transformers | ✅ Excellent | Complete | Comprehensive transformer tests |
|
||||
| 14_profiling | ⚠️ N/A | N/A | Profiling doesn't need gradients |
|
||||
| 15_memoization | ⚠️ N/A | N/A | Caching doesn't need gradients |
|
||||
| 16_quantization | ⚠️ Unknown | Needs Check | Quantization might need gradient tests |
|
||||
| 17_compression | ⚠️ Unknown | Needs Check | Compression might need gradient tests |
|
||||
| 18_acceleration | ⚠️ N/A | N/A | Acceleration doesn't need gradients |
|
||||
| 19_benchmarking | ⚠️ N/A | N/A | Benchmarking doesn't need gradients |
|
||||
|
||||
### **Specific Gaps Identified**
|
||||
|
||||
1. **Module 02_activations** - Not all activations have gradient tests
|
||||
- ✅ Sigmoid tested
|
||||
- ✅ ReLU tested (partial)
|
||||
- ⚠️ Tanh not fully tested
|
||||
- ⚠️ GELU tested in autograd but not in activations module
|
||||
- ⚠️ Softmax not tested
|
||||
|
||||
2. **Module 08_dataloader** - No gradient flow tests
|
||||
- Dataloader doesn't have trainable parameters, but should test:
|
||||
- Data doesn't break gradient flow
|
||||
- Batched operations preserve gradients
|
||||
|
||||
3. **Module 03_layers** - Missing some layer types
|
||||
- ✅ Linear well tested
|
||||
- ⚠️ Dropout not tested
|
||||
- ⚠️ BatchNorm not tested (if exists)
|
||||
- ⚠️ LayerNorm tested in transformers but not in layers module
|
||||
|
||||
4. **Edge Cases** - Some gaps
|
||||
- ⚠️ Vanishing gradients detection
|
||||
- ⚠️ Exploding gradients detection
|
||||
- ⚠️ Gradient clipping
|
||||
- ⚠️ Mixed precision (if applicable)
|
||||
|
||||
---
|
||||
|
||||
## 📋 Recommended Test Structure
|
||||
|
||||
### **For Each Module with Trainable Parameters**
|
||||
|
||||
Create: `tests/XX_modulename/test_gradient_flow.py`
|
||||
|
||||
**Template**:
|
||||
```python
|
||||
"""
|
||||
Gradient Flow Tests for Module XX: [Module Name]
|
||||
|
||||
Tests that gradients flow correctly through all components in this module.
|
||||
"""
|
||||
|
||||
def test_[component]_gradient_flow():
|
||||
"""Test that [Component] preserves gradient flow."""
|
||||
# 1. Create component
|
||||
component = Component(...)
|
||||
|
||||
# 2. Forward pass
|
||||
x = Tensor(..., requires_grad=True)
|
||||
output = component(x)
|
||||
|
||||
# 3. Backward pass
|
||||
loss = output.sum()
|
||||
loss.backward()
|
||||
|
||||
# 4. Verify gradients exist
|
||||
assert x.grad is not None, "Input should have gradients"
|
||||
|
||||
# 5. Verify component parameters have gradients (if trainable)
|
||||
if hasattr(component, 'parameters'):
|
||||
for param in component.parameters():
|
||||
assert param.grad is not None, f"{param} should have gradient"
|
||||
assert np.abs(param.grad).max() > 1e-10, "Gradient should be non-zero"
|
||||
|
||||
def test_[component]_with_previous_modules():
|
||||
"""Test that [Component] works with modules 01 through XX-1."""
|
||||
# Use previous modules
|
||||
from tinytorch.core.tensor import Tensor
|
||||
from tinytorch.core.layers import Linear # if applicable
|
||||
|
||||
# Test integration
|
||||
...
|
||||
```
|
||||
|
||||
### **Critical Checks for Every Module**
|
||||
|
||||
1. **Gradient Existence**: Do gradients exist after backward?
|
||||
2. **Gradient Non-Zero**: Are gradients actually computed (not all zeros)?
|
||||
3. **Parameter Coverage**: Do all trainable parameters receive gradients?
|
||||
4. **Shape Correctness**: Do gradient shapes match parameter shapes?
|
||||
5. **Integration**: Does it work with previous modules?
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Priority Recommendations
|
||||
|
||||
### **High Priority** (Must Have)
|
||||
|
||||
1. **Complete Module 02_activations gradient tests**
|
||||
- Create `tests/02_activations/test_gradient_flow.py`
|
||||
- Test all activations: Sigmoid, ReLU, Tanh, GELU, Softmax
|
||||
- Verify gradients are correct (not just exist)
|
||||
|
||||
2. **Add Module 08_dataloader gradient flow tests**
|
||||
- Create `tests/08_dataloader/test_gradient_flow.py`
|
||||
- Test that dataloader doesn't break gradient flow
|
||||
- Test batched operations preserve gradients
|
||||
|
||||
3. **Complete Module 03_layers gradient tests**
|
||||
- Add Dropout gradient tests
|
||||
- Add LayerNorm gradient tests (if in layers module)
|
||||
- Add BatchNorm gradient tests (if exists)
|
||||
|
||||
### **Medium Priority** (Should Have)
|
||||
|
||||
4. **Add vanishing/exploding gradient detection**
|
||||
- Create `tests/debugging/test_gradient_vanishing.py`
|
||||
- Create `tests/debugging/test_gradient_explosion.py`
|
||||
- Provide helpful error messages for students
|
||||
|
||||
5. **Add per-module progressive integration gradient tests**
|
||||
- Each module should test: "Do gradients flow through module N with modules 1-N-1?"
|
||||
- Example: `tests/07_training/test_gradient_flow_progressive.py`
|
||||
|
||||
### **Low Priority** (Nice to Have)
|
||||
|
||||
6. **Add numerical stability gradient tests**
|
||||
- Test with very small values
|
||||
- Test with very large values
|
||||
- Test with NaN/Inf handling
|
||||
|
||||
7. **Add gradient accumulation tests per module**
|
||||
- Test that gradients accumulate correctly
|
||||
- Test zero_grad() works correctly
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Implementation Plan
|
||||
|
||||
### **Step 1: Create Missing Module Gradient Flow Tests**
|
||||
|
||||
For each module missing gradient flow tests:
|
||||
|
||||
```bash
|
||||
# Create test file
|
||||
touch tests/XX_modulename/test_gradient_flow.py
|
||||
|
||||
# Add template with:
|
||||
# - Component gradient flow tests
|
||||
# - Integration with previous modules
|
||||
# - Edge cases
|
||||
```
|
||||
|
||||
### **Step 2: Enhance Existing Tests**
|
||||
|
||||
For modules with partial coverage:
|
||||
|
||||
1. Review existing tests
|
||||
2. Identify missing components
|
||||
3. Add tests for missing components
|
||||
4. Ensure all trainable parameters are tested
|
||||
|
||||
### **Step 3: Add Debugging Tests**
|
||||
|
||||
Create helpful debugging tests:
|
||||
|
||||
```python
|
||||
# tests/debugging/test_gradient_vanishing.py
|
||||
def test_detect_vanishing_gradients():
|
||||
"""Detect and diagnose vanishing gradients."""
|
||||
# Deep network
|
||||
# Check gradient magnitudes
|
||||
# Provide helpful error message
|
||||
```
|
||||
|
||||
### **Step 4: Add Progressive Integration Gradient Tests**
|
||||
|
||||
For each module, add:
|
||||
|
||||
```python
|
||||
# tests/XX_modulename/test_gradient_flow_progressive.py
|
||||
def test_module_N_gradients_with_all_previous():
|
||||
"""Test that module N gradients work with modules 1 through N-1."""
|
||||
# Use all previous modules
|
||||
# Test gradient flow through complete stack
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Test Execution Strategy
|
||||
|
||||
### **During Development**
|
||||
```bash
|
||||
# Test specific module gradient flow
|
||||
pytest tests/XX_modulename/test_gradient_flow.py -v
|
||||
|
||||
# Test integration gradient flow
|
||||
pytest tests/integration/test_gradient_flow.py -v
|
||||
|
||||
# Test all gradient flow tests
|
||||
pytest tests/ -k "gradient" -v
|
||||
```
|
||||
|
||||
### **Before Committing**
|
||||
```bash
|
||||
# Run all gradient flow tests
|
||||
pytest tests/integration/test_gradient_flow.py tests/*/test_gradient_flow.py -v
|
||||
|
||||
# Critical: Must pass before merging
|
||||
pytest tests/integration/test_gradient_flow.py -v
|
||||
```
|
||||
|
||||
### **CI/CD Integration**
|
||||
- Add gradient flow tests to CI pipeline
|
||||
- Fail build if critical gradient flow tests fail
|
||||
- Report gradient flow test coverage
|
||||
|
||||
---
|
||||
|
||||
## ✅ Success Criteria
|
||||
|
||||
A module has **complete gradient flow coverage** when:
|
||||
|
||||
1. ✅ All trainable components have gradient flow tests
|
||||
2. ✅ All activations preserve gradient flow
|
||||
3. ✅ Integration with previous modules is tested
|
||||
4. ✅ Edge cases are covered (zero gradients, small values, etc.)
|
||||
5. ✅ Tests verify gradients are non-zero (not just exist)
|
||||
6. ✅ Tests verify gradient shapes match parameter shapes
|
||||
7. ✅ Tests provide helpful error messages when they fail
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Educational Value
|
||||
|
||||
Gradient flow tests teach students:
|
||||
|
||||
1. **Gradient flow is critical**: Components must preserve gradients
|
||||
2. **Integration matters**: Components must work together
|
||||
3. **Debugging skills**: How to diagnose gradient flow issues
|
||||
4. **Best practices**: Proper gradient handling patterns
|
||||
|
||||
---
|
||||
|
||||
## 📚 References
|
||||
|
||||
- **Critical Test**: `tests/integration/test_gradient_flow.py` - Must pass before merging
|
||||
- **Comprehensive Suite**: `tests/test_gradient_flow.py` - Full coverage
|
||||
- **Module Tests**: `tests/XX_modulename/test_gradient_flow.py` - Per-module coverage
|
||||
- **Transformer Tests**: `tests/13_transformers/test_transformer_gradient_flow.py` - Example of comprehensive module tests
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-XX
|
||||
**Status**: Analysis complete, implementation in progress
|
||||
**Priority**: High - Gradient flow is critical for training to work
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,188 +0,0 @@
|
||||
# NBGrader Standardized Testing Framework
|
||||
|
||||
## 🎯 The Perfect Solution
|
||||
|
||||
Your suggestion to use **dedicated, locked NBGrader cells** for testing is brilliant! This approach provides:
|
||||
|
||||
✅ **Protected Infrastructure** - Students can't break the testing framework
|
||||
✅ **Consistent Placement** - Same location in every module (before final summary)
|
||||
✅ **Educational Flow** - Learn → Implement → Test → Reflect
|
||||
✅ **Professional Standards** - Mirrors real software development practices
|
||||
✅ **Quality Assurance** - Ensures comprehensive validation of all student work
|
||||
|
||||
## 📋 Module Structure
|
||||
|
||||
Every TinyTorch module follows this standardized structure:
|
||||
|
||||
```
|
||||
1. 📖 Educational Content & Implementation Guidance
|
||||
2. 💻 Student Implementation Sections (unlocked)
|
||||
3. 🧪 Standardized Testing (LOCKED NBGrader cell)
|
||||
4. 🎯 Module Summary & Takeaways
|
||||
```
|
||||
|
||||
## 🔒 The Locked Testing Cell
|
||||
|
||||
### NBGrader Configuration
|
||||
```python
|
||||
# %% nbgrader={"grade": false, "grade_id": "standardized-testing", "locked": true, "schema_version": 3, "solution": false, "task": false}
|
||||
```
|
||||
|
||||
### Key Settings Explained:
|
||||
- **`grade: false`** - Testing cell is not graded (provides feedback only)
|
||||
- **`locked: true`** - Students cannot modify this cell
|
||||
- **`solution: false`** - This is not a solution cell
|
||||
- **`task: false`** - This is not a task for students to complete
|
||||
|
||||
### Cell Structure:
|
||||
```python
|
||||
# =============================================================================
|
||||
# STANDARDIZED MODULE TESTING - DO NOT MODIFY
|
||||
# This cell is locked to ensure consistent testing across all TinyTorch modules
|
||||
# =============================================================================
|
||||
|
||||
from tinytorch.utils.testing import create_test_runner
|
||||
|
||||
def test_core_functionality():
|
||||
"""Test core module functionality."""
|
||||
# Module-specific tests here
|
||||
print("✅ Core functionality tests passed!")
|
||||
|
||||
def test_edge_cases():
|
||||
"""Test edge cases and error handling."""
|
||||
# Edge case tests here
|
||||
print("✅ Edge case tests passed!")
|
||||
|
||||
def test_ml_integration():
|
||||
"""Test integration with ML workflows."""
|
||||
# Integration tests here
|
||||
print("✅ ML integration tests passed!")
|
||||
|
||||
# Execute standardized testing
|
||||
if __name__ == "__main__":
|
||||
test_runner = create_test_runner("ModuleName")
|
||||
|
||||
test_runner.register_test("Core Functionality", test_core_functionality)
|
||||
test_runner.register_test("Edge Cases", test_edge_cases)
|
||||
test_runner.register_test("ML Integration", test_ml_integration)
|
||||
|
||||
success = test_runner.run_all_tests()
|
||||
```
|
||||
|
||||
## 🎭 Consistent Student Experience
|
||||
|
||||
Every module produces **identical testing output**:
|
||||
|
||||
```
|
||||
🔬 Running ModuleName Module Tests...
|
||||
==================================================
|
||||
🧪 Testing Core Functionality... ✅ PASSED
|
||||
🧪 Testing Edge Cases... ✅ PASSED
|
||||
🧪 Testing ML Integration... ✅ PASSED
|
||||
|
||||
============================================================
|
||||
🎯 MODULENAME MODULE TESTING COMPLETE
|
||||
============================================================
|
||||
🎉 CONGRATULATIONS! All tests passed!
|
||||
|
||||
✅ ModuleName Module Status: 3/3 tests passed (100%)
|
||||
|
||||
📊 Detailed Results:
|
||||
Core Functionality: ✅ PASSED
|
||||
Edge Cases: ✅ PASSED
|
||||
ML Integration: ✅ PASSED
|
||||
|
||||
📈 Progress: ModuleName Module ✓ COMPLETE
|
||||
|
||||
🚀 Ready for the next module!
|
||||
```
|
||||
|
||||
## 📚 Educational Benefits
|
||||
|
||||
### For Students:
|
||||
1. **Consistent Experience** - Same testing format across all modules
|
||||
2. **Immediate Feedback** - Clear validation of their implementations
|
||||
3. **Professional Exposure** - Experience with real testing practices
|
||||
4. **Protected Learning** - Cannot accidentally break testing infrastructure
|
||||
5. **Quality Confidence** - Assurance their implementations work correctly
|
||||
|
||||
### For Instructors:
|
||||
1. **Standardized Quality** - Consistent validation across all modules
|
||||
2. **Protected Infrastructure** - Testing framework cannot be compromised
|
||||
3. **Easy Maintenance** - Single source of truth for testing format
|
||||
4. **Educational Focus** - More time on content, less on testing logistics
|
||||
5. **Scalable Assessment** - Efficient evaluation of student progress
|
||||
|
||||
## 🔄 Module Flow
|
||||
|
||||
### 1. Educational Introduction
|
||||
```markdown
|
||||
# Module X: Topic Name
|
||||
Learn about [concept] and its importance in ML systems...
|
||||
```
|
||||
|
||||
### 2. Implementation Guidance
|
||||
```python
|
||||
# Student implementation sections (UNLOCKED)
|
||||
# Clear TODOs and guidance for student work
|
||||
```
|
||||
|
||||
### 3. Testing Validation (LOCKED)
|
||||
```markdown
|
||||
## 🧪 Module Testing
|
||||
Time to test your implementation! This section is locked to ensure consistency.
|
||||
```
|
||||
|
||||
### 4. Learning Summary
|
||||
```markdown
|
||||
## 🎯 Module Summary: Topic Mastery!
|
||||
Congratulations! You've successfully implemented...
|
||||
```
|
||||
|
||||
## 🏗️ Implementation Strategy
|
||||
|
||||
### Phase 1: Infrastructure
|
||||
- ✅ **Shared testing utilities** - `tinytorch.utils.testing` module
|
||||
- ✅ **NBGrader template** - Standardized cell structure
|
||||
- ✅ **Documentation** - Clear guidelines for implementation
|
||||
|
||||
### Phase 2: Module Migration
|
||||
1. **Add testing section** to each module before final summary
|
||||
2. **Lock testing cells** with NBGrader configuration
|
||||
3. **Register module tests** with shared test runner
|
||||
4. **Validate consistency** across all modules
|
||||
|
||||
### Phase 3: Quality Assurance
|
||||
1. **Test each module** individually for correctness
|
||||
2. **Verify consistent output** across all modules
|
||||
3. **Ensure NBGrader compatibility** with locked cells
|
||||
4. **Document any module-specific considerations**
|
||||
|
||||
## 🎯 Benefits Achieved
|
||||
|
||||
### Technical Benefits:
|
||||
- **Zero Code Duplication** - Shared testing infrastructure
|
||||
- **Perfect Consistency** - Identical output format across modules
|
||||
- **Protected Quality** - Testing framework cannot be broken
|
||||
- **Easy Maintenance** - Single point of update for improvements
|
||||
|
||||
### Educational Benefits:
|
||||
- **Professional Standards** - Real-world software development practices
|
||||
- **Immediate Feedback** - Clear validation of student implementations
|
||||
- **Consistent Experience** - Same quality across all learning modules
|
||||
- **Focus on Learning** - Students focus on concepts, not testing setup
|
||||
|
||||
### Assessment Benefits:
|
||||
- **Standardized Evaluation** - Consistent criteria across modules
|
||||
- **Automated Validation** - Reliable testing of student implementations
|
||||
- **Quality Assurance** - Comprehensive coverage of learning objectives
|
||||
- **Scalable Grading** - Efficient instructor workflow
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
1. **Apply template** to all existing modules
|
||||
2. **Test NBGrader integration** with locked cells
|
||||
3. **Validate student experience** across all modules
|
||||
4. **Document module-specific testing** requirements
|
||||
|
||||
This NBGrader standardized testing framework provides the **perfect balance** of consistency, protection, and educational value!
|
||||
@@ -1,174 +0,0 @@
|
||||
# NBGrader Testing Cell Template
|
||||
|
||||
## 🎯 Standardized Module Structure
|
||||
|
||||
Every TinyTorch module should follow this structure for consistent testing:
|
||||
|
||||
### 1. Educational Content
|
||||
```python
|
||||
# %% [markdown]
|
||||
"""
|
||||
# Module X: Topic Name
|
||||
[Educational content, implementation guidance, etc.]
|
||||
"""
|
||||
|
||||
# %%
|
||||
#| default_exp core.module_name
|
||||
|
||||
# Student implementation sections...
|
||||
# [Student code here]
|
||||
```
|
||||
|
||||
### 2. Individual Test Functions
|
||||
```python
|
||||
# Test functions that students can run during development
|
||||
def test_feature_1_comprehensive():
|
||||
"""Test feature 1 functionality comprehensively."""
|
||||
# Detailed test implementation
|
||||
assert feature_works()
|
||||
print("✅ Feature 1 tests passed!")
|
||||
|
||||
def test_feature_2_integration():
|
||||
"""Test feature 2 integration with other components."""
|
||||
# Integration test implementation
|
||||
assert integration_works()
|
||||
print("✅ Feature 2 integration tests passed!")
|
||||
|
||||
def test_module_integration():
|
||||
"""Test overall module integration."""
|
||||
# Overall integration tests
|
||||
assert module_works()
|
||||
print("✅ Module integration tests passed!")
|
||||
```
|
||||
|
||||
### 3. Dedicated Testing Section (Auto-Discovery)
|
||||
```python
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 🧪 Module Testing
|
||||
|
||||
Time to test your implementation! This section uses TinyTorch's standardized testing framework with **automatic test discovery**.
|
||||
|
||||
**This testing section is locked** - it provides consistent feedback across all modules and cannot be modified.
|
||||
"""
|
||||
|
||||
# %% nbgrader={"grade": false, "grade_id": "standardized-testing", "locked": true, "schema_version": 3, "solution": false, "task": false}
|
||||
# =============================================================================
|
||||
# STANDARDIZED MODULE TESTING - DO NOT MODIFY
|
||||
# This cell is locked to ensure consistent testing across all TinyTorch modules
|
||||
# =============================================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
from tinytorch.utils.testing import run_module_tests_auto
|
||||
|
||||
# Automatically discover and run all tests in this module
|
||||
success = run_module_tests_auto("ModuleName")
|
||||
```
|
||||
|
||||
### 4. Module Summary (After Testing)
|
||||
```python
|
||||
# %% [markdown]
|
||||
"""
|
||||
## 🎯 Module Summary: [Topic] Mastery!
|
||||
|
||||
Congratulations! You've successfully implemented [module topic]:
|
||||
|
||||
### What You've Accomplished
|
||||
✅ **Feature 1**: Description of what was implemented
|
||||
✅ **Feature 2**: Description of what was implemented
|
||||
✅ **Integration**: How features work together
|
||||
|
||||
### Key Concepts You've Learned
|
||||
- **Concept 1**: Explanation
|
||||
- **Concept 2**: Explanation
|
||||
- **Concept 3**: Explanation
|
||||
|
||||
### Next Steps
|
||||
1. **Export your code**: `tito package nbdev --export module_name`
|
||||
2. **Test your implementation**: `tito test module_name`
|
||||
3. **Move to next module**: Brief description of what's next
|
||||
"""
|
||||
```
|
||||
|
||||
## 🎯 **Critical: Correct Section Ordering**
|
||||
|
||||
The order of sections **must** follow this logical flow:
|
||||
|
||||
1. **Educational Content** - Students learn the concepts
|
||||
2. **Implementation Sections** - Students build the functionality
|
||||
3. **🧪 Module Testing** - Students verify their implementation works
|
||||
4. **🎯 Module Summary** - Students celebrate success and move forward
|
||||
|
||||
### ❌ **Wrong Order (Confusing)**:
|
||||
```
|
||||
Implementation → Summary ("Congratulations!") → Testing → "Wait, did it work?"
|
||||
```
|
||||
|
||||
### ✅ **Correct Order (Natural)**:
|
||||
```
|
||||
Implementation → Testing → Summary ("Congratulations! It works!") → Next Steps
|
||||
```
|
||||
|
||||
**Why This Matters**:
|
||||
- Testing **validates** the implementation before celebrating
|
||||
- Summary **confirms** success after verification
|
||||
- Natural flow: Build → Test → Celebrate → Advance
|
||||
- Mirrors real software development practices
|
||||
|
||||
## 🔍 Automatic Test Discovery
|
||||
|
||||
The new testing framework **automatically discovers** test functions, eliminating manual registration:
|
||||
|
||||
### ✅ **Discovered Test Patterns**
|
||||
The system automatically finds and runs functions matching these patterns:
|
||||
- `test_*_comprehensive`: Comprehensive testing of individual features
|
||||
- `test_*_integration`: Integration testing with other components
|
||||
- `test_*_activation`: Specific activation function tests (ReLU, Sigmoid, etc.)
|
||||
|
||||
### ✅ **Benefits**
|
||||
- **Zero Manual Work**: No need to register functions manually
|
||||
- **Error Prevention**: Won't miss test functions
|
||||
- **Consistent Naming**: Enforces good test naming conventions
|
||||
- **Automatic Ordering**: Tests run in alphabetical order
|
||||
- **Clean Output**: Standardized reporting format
|
||||
|
||||
### ✅ **Example Output**
|
||||
```
|
||||
🔍 Auto-discovered 4 test functions
|
||||
|
||||
🧪 Running Tensor Module Tests...
|
||||
==================================================
|
||||
✅ Tensor Arithmetic: PASSED
|
||||
✅ Tensor Creation: PASSED
|
||||
✅ Tensor Integration: PASSED
|
||||
✅ Tensor Properties: PASSED
|
||||
==================================================
|
||||
🎉 All tests passed! (4/4)
|
||||
✅ Tensor module is working correctly!
|
||||
```
|
||||
|
||||
### ✅ **Safety Features**
|
||||
- **Pattern Matching**: Only discovers functions matching expected patterns
|
||||
- **Protected Framework**: NBGrader locked cells prevent student modifications
|
||||
- **Fallback Support**: Manual registration still available if needed
|
||||
- **Error Handling**: Graceful handling of malformed test functions
|
||||
|
||||
## 📝 Implementation Notes
|
||||
|
||||
### Test Function Requirements
|
||||
1. **Naming Convention**: Must start with `test_` and contain expected patterns
|
||||
2. **Self-Contained**: Each test should be independent
|
||||
3. **Clear Output**: Print success messages for educational feedback
|
||||
4. **Proper Assertions**: Use assert statements for validation
|
||||
|
||||
### Module Integration
|
||||
1. **Single Entry Point**: Each module has one standardized testing entry
|
||||
2. **Consistent Interface**: Same API across all modules
|
||||
3. **CLI Integration**: `tito test module_name` uses the auto-discovery
|
||||
4. **Development Workflow**: Students can run individual tests during development
|
||||
|
||||
### Educational Benefits
|
||||
1. **Immediate Feedback**: Students see results as they develop
|
||||
2. **Professional Practices**: Mirrors real software development workflows
|
||||
3. **Consistent Experience**: Same testing approach across all modules
|
||||
4. **Assessment Ready**: NBGrader can evaluate student implementations
|
||||
@@ -1,425 +0,0 @@
|
||||
# TinyTorch Testing Architecture
|
||||
|
||||
## 🎯 Overview: Two-Tier Testing Strategy
|
||||
|
||||
TinyTorch uses a **two-tier testing approach** that separates component validation from system integration:
|
||||
|
||||
1. **Inline Tests** (`modules/`) - Component validation, unit tests
|
||||
2. **Integration Tests** (`tests/`) - Inter-module integration, edge cases, system tests
|
||||
|
||||
This separation follows ML engineering best practices: validate components in isolation, then test how they work together.
|
||||
|
||||
---
|
||||
|
||||
## 📋 Tier 1: Inline Tests (Component Validation)
|
||||
|
||||
### **Location**: `modules/XX_modulename/*.py`
|
||||
|
||||
### **Purpose**:
|
||||
- ✅ Validate individual components work correctly **in isolation**
|
||||
- ✅ Test single module functionality
|
||||
- ✅ Provide immediate feedback during development
|
||||
- ✅ Educate students about expected behavior
|
||||
- ✅ Fast execution for rapid iteration
|
||||
|
||||
### **What Gets Tested**:
|
||||
- Individual class/function correctness
|
||||
- Mathematical operations (forward passes)
|
||||
- Shape transformations
|
||||
- Basic edge cases and error handling
|
||||
- Component-level functionality
|
||||
|
||||
### **Test Pattern**:
|
||||
```python
|
||||
def test_unit_componentname():
|
||||
"""🧪 Unit Test: Component Name
|
||||
|
||||
**This is a unit test** - it tests [component] in isolation.
|
||||
"""
|
||||
print("🔬 Unit Test: Component...")
|
||||
|
||||
# Test implementation
|
||||
assert condition, "✅ Component works"
|
||||
|
||||
print("✅ Component test passed")
|
||||
```
|
||||
|
||||
### **Example**: `modules/01_tensor/tensor.py`
|
||||
- `test_unit_tensor_creation()` - Tests tensor creation
|
||||
- `test_unit_arithmetic_operations()` - Tests +, -, *, /
|
||||
- `test_unit_matrix_multiplication()` - Tests @ operator
|
||||
- `test_unit_shape_manipulation()` - Tests reshape, transpose
|
||||
- `test_unit_reduction_operations()` - Tests sum, mean, max
|
||||
|
||||
### **Execution**:
|
||||
```bash
|
||||
# Run inline tests only
|
||||
tito test 01_tensor --inline-only
|
||||
|
||||
# Tests run when you execute the module file
|
||||
python modules/01_tensor/tensor.py
|
||||
```
|
||||
|
||||
### **Key Characteristics**:
|
||||
- ✅ **Fast**: Run during development for immediate feedback
|
||||
- ✅ **Isolated**: No dependencies on other modules
|
||||
- ✅ **Educational**: Shows students what "correct" looks like
|
||||
- ✅ **Component-focused**: Tests one thing at a time
|
||||
|
||||
---
|
||||
|
||||
## 📊 Tier 2: Integration Tests (`tests/` Directory)
|
||||
|
||||
### **Location**: `tests/`
|
||||
|
||||
### **Purpose**:
|
||||
- ✅ Test how **multiple modules work together**
|
||||
- ✅ Validate cross-module dependencies
|
||||
- ✅ Test realistic workflows and use cases
|
||||
- ✅ Ensure system-level correctness
|
||||
- ✅ Catch bugs that unit tests miss
|
||||
- ✅ Test edge cases and corner scenarios
|
||||
- ✅ Validate exported code (`tinytorch/`) works correctly
|
||||
|
||||
### **Key Insight**:
|
||||
**Component correctness ≠ System correctness**
|
||||
|
||||
A tensor might work perfectly in isolation, but fail when gradients flow through layers → activations → losses → optimizers. Integration tests catch these "seam" bugs.
|
||||
|
||||
---
|
||||
|
||||
## 🗂️ Structure of `tests/` Directory
|
||||
|
||||
### 1. **Module-Specific Integration Tests** (`tests/XX_modulename/`)
|
||||
|
||||
**Purpose**: Test that module N works correctly **with all previous modules** (1 through N-1)
|
||||
|
||||
**Example**: `tests/05_autograd/test_progressive_integration.py`
|
||||
- Tests autograd with Tensor (01), Activations (02), Layers (03), Losses (04)
|
||||
- Validates that gradients flow correctly through the entire stack built so far
|
||||
|
||||
**Pattern**: Progressive integration
|
||||
```python
|
||||
# tests/05_autograd/test_progressive_integration.py
|
||||
def test_autograd_with_all_previous_modules():
|
||||
# Uses real Tensor, real Layers, real Activations, real Losses
|
||||
# Then tests Autograd (05) with all of them
|
||||
x = Tensor([[1.0, 2.0]], requires_grad=True)
|
||||
layer = Linear(2, 3)
|
||||
activation = ReLU()
|
||||
loss_fn = MSELoss()
|
||||
|
||||
output = activation(layer(x))
|
||||
loss = loss_fn(output, target)
|
||||
loss.backward()
|
||||
|
||||
assert x.grad is not None # Gradient flowed through everything!
|
||||
```
|
||||
|
||||
**Why This Matters**:
|
||||
- Catches integration bugs early
|
||||
- Ensures modules don't break previous functionality
|
||||
- Validates the "seams" between modules
|
||||
|
||||
---
|
||||
|
||||
### 2. **Cross-Module Integration Tests** (`tests/integration/`)
|
||||
|
||||
**Purpose**: Test **multiple modules working together** in realistic scenarios
|
||||
|
||||
**Key Files**:
|
||||
- `test_gradient_flow.py` - **CRITICAL**: Validates gradients flow through entire training stack
|
||||
- `test_end_to_end_training.py` - Full training loops
|
||||
- `test_module_compatibility.py` - Module interfaces
|
||||
|
||||
**Example**: `tests/integration/test_gradient_flow.py`
|
||||
```python
|
||||
def test_complete_training_stack():
|
||||
"""Test that gradients flow through: Tensor → Layers → Activations → Loss → Autograd → Optimizer"""
|
||||
# Uses modules 01, 02, 03, 04, 05, 06, 07
|
||||
# Validates the entire training pipeline works
|
||||
```
|
||||
|
||||
**Why This Matters**:
|
||||
- Catches bugs that unit tests miss
|
||||
- Validates the "seams" between modules
|
||||
- Ensures training actually works end-to-end
|
||||
- Tests realistic ML workflows
|
||||
|
||||
---
|
||||
|
||||
### 3. **Edge Cases & Stress Tests** (`tests/05_autograd/`, `tests/debugging/`)
|
||||
|
||||
**Purpose**: Test **corner cases** and **common pitfalls**
|
||||
|
||||
**Examples**:
|
||||
- `tests/05_autograd/test_broadcasting.py` - Broadcasting gradient bugs
|
||||
- `tests/05_autograd/test_computation_graph.py` - Graph construction edge cases
|
||||
- `tests/debugging/test_gradient_vanishing.py` - Detect vanishing gradients
|
||||
- `tests/debugging/test_common_mistakes.py` - "Did you forget backward()?" style tests
|
||||
|
||||
**Philosophy**: When these tests fail, the error message should **teach the student** what went wrong and how to fix it.
|
||||
|
||||
**Why This Matters**:
|
||||
- Catches numerical stability issues
|
||||
- Tests edge cases that break in production
|
||||
- Pedagogical: teaches debugging skills
|
||||
|
||||
---
|
||||
|
||||
### 4. **Regression Tests** (`tests/regression/`)
|
||||
|
||||
**Purpose**: Ensure **previously fixed bugs don't come back**
|
||||
|
||||
**Pattern**: Each bug gets a test file
|
||||
- `test_issue_20241125_conv_fc_shapes.py` - Tests a specific bug that was fixed
|
||||
- Documents the bug, root cause, fix, and prevention
|
||||
|
||||
**Why This Matters**:
|
||||
- Prevents regressions
|
||||
- Documents historical bugs
|
||||
- Ensures fixes persist
|
||||
|
||||
---
|
||||
|
||||
### 5. **Performance Tests** (`tests/performance/`)
|
||||
|
||||
**Purpose**: Validate **systems performance** characteristics
|
||||
|
||||
**Examples**:
|
||||
- Memory profiling
|
||||
- Speed benchmarks
|
||||
- Scalability tests
|
||||
|
||||
**Why This Matters**:
|
||||
- Ensures implementations are efficient
|
||||
- Validates performance characteristics
|
||||
- Catches performance regressions
|
||||
|
||||
---
|
||||
|
||||
### 6. **System Tests** (`tests/system/`)
|
||||
|
||||
**Purpose**: Test **entire system workflows**
|
||||
|
||||
**Examples**:
|
||||
- End-to-end training pipelines
|
||||
- Model export/import
|
||||
- Checkpoint system tests
|
||||
|
||||
**Why This Matters**:
|
||||
- Validates complete workflows
|
||||
- Tests production scenarios
|
||||
- Ensures system-level correctness
|
||||
|
||||
---
|
||||
|
||||
### 7. **Checkpoint Tests** (`tests/checkpoints/`)
|
||||
|
||||
**Purpose**: Validate **milestone capabilities**
|
||||
|
||||
**Examples**:
|
||||
- `checkpoint_01_foundation.py` - Tensor operations mastered
|
||||
- `checkpoint_05_learning.py` - Autograd working correctly
|
||||
|
||||
**Why This Matters**:
|
||||
- Validates student progress
|
||||
- Ensures milestones are met
|
||||
- Provides clear success criteria
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Code Flow: Development → Export → Testing
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ DEVELOPMENT WORKFLOW │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
1. DEVELOP in modules/
|
||||
└─> modules/01_tensor/tensor.py
|
||||
├─> Write code
|
||||
├─> Write inline tests (test_unit_*)
|
||||
└─> Run: python modules/01_tensor/tensor.py
|
||||
|
||||
2. EXPORT to tinytorch/
|
||||
└─> tito export 01_tensor
|
||||
└─> Code exported to tinytorch/core/tensor.py
|
||||
|
||||
3. TEST integration
|
||||
└─> tests/01_tensor/test_progressive_integration.py
|
||||
├─> Imports from tinytorch.core.tensor (exported code!)
|
||||
├─> Tests module works with previous modules
|
||||
└─> Run: pytest tests/01_tensor/
|
||||
|
||||
4. TEST cross-module
|
||||
└─> tests/integration/test_gradient_flow.py
|
||||
├─> Imports from tinytorch.* (all exported modules)
|
||||
├─> Tests multiple modules working together
|
||||
└─> Run: pytest tests/integration/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Decision Tree: Where Should This Test Go?
|
||||
|
||||
```
|
||||
Is it testing a single component in isolation?
|
||||
├─ YES → modules/XX_modulename/*.py (inline test_unit_*)
|
||||
│
|
||||
└─ NO → Is it testing module N with previous modules?
|
||||
├─ YES → tests/XX_modulename/test_progressive_integration.py
|
||||
│
|
||||
└─ NO → Is it testing multiple modules together?
|
||||
├─ YES → tests/integration/test_*.py
|
||||
│
|
||||
└─ NO → Is it an edge case or stress test?
|
||||
├─ YES → tests/XX_modulename/test_*_edge_cases.py
|
||||
│ OR tests/debugging/test_*.py
|
||||
│
|
||||
└─ NO → Is it a regression test?
|
||||
├─ YES → tests/regression/test_issue_*.py
|
||||
│
|
||||
└─ NO → Is it a performance test?
|
||||
├─ YES → tests/performance/test_*.py
|
||||
│
|
||||
└─ NO → Is it a system test?
|
||||
└─ YES → tests/system/test_*.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Best Practices
|
||||
|
||||
### **DO**:
|
||||
✅ Write inline tests immediately after implementing a component
|
||||
✅ Test one thing per inline test function
|
||||
✅ Use descriptive test function names (`test_unit_sigmoid`, not `test1`)
|
||||
✅ Add integration tests when combining multiple modules
|
||||
✅ Run inline tests frequently during development
|
||||
✅ Run full test suite before committing
|
||||
✅ Test exported code (`tinytorch/`), not development code (`modules/`)
|
||||
✅ Write tests that catch real bugs you've encountered
|
||||
|
||||
### **DON'T**:
|
||||
❌ Mix inline and integration test concerns
|
||||
❌ Test implementation details in integration tests
|
||||
❌ Skip inline tests and jump to integration
|
||||
❌ Test mocked/fake components (use real ones)
|
||||
❌ Create dependencies between test files
|
||||
❌ Test code in `modules/` directly in `tests/` (test `tinytorch/` instead)
|
||||
❌ Duplicate inline tests in `tests/` directory
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Key Distinctions
|
||||
|
||||
| Aspect | Inline Tests (`modules/`) | Integration Tests (`tests/`) |
|
||||
|--------|-------------------------|----------------------------|
|
||||
| **Location** | `modules/XX_name/*.py` | `tests/XX_name/` or `tests/integration/` |
|
||||
| **Scope** | Single component | Multiple modules |
|
||||
| **Dependencies** | None (isolated) | Previous modules |
|
||||
| **Speed** | Fast | Slower |
|
||||
| **Purpose** | Component correctness | System correctness |
|
||||
| **When to run** | During development | Before commit/export |
|
||||
| **What gets tested** | `modules/` code directly | `tinytorch/` exported code |
|
||||
| **Example** | `test_unit_tensor_creation()` | `test_tensor_with_layers()` |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Testing Workflow
|
||||
|
||||
### For Students:
|
||||
|
||||
```bash
|
||||
# 1. Work on module
|
||||
cd modules/01_tensor
|
||||
vim tensor.py
|
||||
|
||||
# 2. Run inline tests (fast feedback)
|
||||
python tensor.py
|
||||
# or
|
||||
tito test 01_tensor --inline-only
|
||||
|
||||
# 3. Export to package
|
||||
tito export 01_tensor
|
||||
|
||||
# 4. Run integration tests (full validation)
|
||||
tito test 01_tensor
|
||||
# or
|
||||
pytest tests/01_tensor/
|
||||
|
||||
# 5. Run cross-module tests (ensure nothing broke)
|
||||
pytest tests/integration/
|
||||
```
|
||||
|
||||
### For Instructors:
|
||||
|
||||
```bash
|
||||
# Comprehensive test suite
|
||||
tito test --comprehensive
|
||||
|
||||
# Specific module deep dive
|
||||
tito test 05_autograd --detailed
|
||||
|
||||
# All inline tests only (quick check)
|
||||
tito test --all --inline-only
|
||||
|
||||
# Critical integration tests
|
||||
pytest tests/integration/test_gradient_flow.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Why This Architecture?
|
||||
|
||||
### **Separation of Concerns**:
|
||||
- **Inline tests** = "Does this component work?"
|
||||
- **Integration tests** = "Do these components work together?"
|
||||
|
||||
### **Educational Value**:
|
||||
- Students learn component testing first
|
||||
- Then learn integration testing
|
||||
- Mirrors professional ML engineering workflows
|
||||
|
||||
### **Practical Benefits**:
|
||||
- Fast feedback during development (inline tests)
|
||||
- Comprehensive validation before commit (integration tests)
|
||||
- Catches bugs at the right level
|
||||
- Clear mental model: component vs. system
|
||||
|
||||
### **Real-World Alignment**:
|
||||
- Professional ML teams use this pattern
|
||||
- Unit tests for components
|
||||
- Integration tests for pipelines
|
||||
- System tests for workflows
|
||||
|
||||
---
|
||||
|
||||
## 📚 Summary
|
||||
|
||||
**Think of `tests/` as the "system validation layer":**
|
||||
|
||||
1. **`modules/` inline tests** = "Does my component work?"
|
||||
2. **`tests/XX_modulename/`** = "Does my module work with previous modules?"
|
||||
3. **`tests/integration/`** = "Do multiple modules work together?"
|
||||
4. **`tests/debugging/`** = "Are there edge cases I'm missing?"
|
||||
5. **`tests/regression/`** = "Did I break something that was working?"
|
||||
6. **`tests/performance/`** = "Is my implementation efficient?"
|
||||
7. **`tests/system/`** = "Does the entire system work?"
|
||||
|
||||
**The key insight**: `tests/` validates that exported code (`tinytorch/`) works correctly in realistic scenarios, catching bugs that isolated unit tests miss.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-01-XX
|
||||
**Test Infrastructure**: Complete (20/20 modules have test directories)
|
||||
**Philosophy**: Component correctness ≠ System correctness
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user