TinyTorch/book/testing-framework.md

Testing Framework

Comprehensive Testing with TinyTorch

TinyTorch includes a robust testing framework that ensures all modules work correctly and integrate seamlessly. The testing system provides multiple levels of validation from individual module testing to full system integration.

Quick Start: Run All Tests

# Run comprehensive test suite (recommended)
tito test --comprehensive

# Output shows module tests, integration tests, and example validation
# ✅ Overall TinyTorch Health: 100.0%

Testing Levels

1. Module-Level Testing

Test individual components in isolation:

# Test specific modules
tito test --module tensor
tito test --module training
tito test --module autograd

# All modules have dedicated test suites
python tests/run_all_modules.py

2. Integration Testing

Verify modules work together correctly:

# Run integration tests
python tests/integration/run_integration_tests.py

# Test module dependencies
python tests/integration/test_module_dependencies.py

3. Checkpoint Testing

Validate learning progression capabilities:

# Test specific capabilities
tito checkpoint test 01  # Foundation capabilities
tito checkpoint test 10  # Training capabilities

# Run all checkpoint tests
for i in {00..15}; do tito checkpoint test $i; done

4. Example Validation

Ensure real-world examples work:

# Example tests are included in comprehensive testing
# Validates XOR training and CIFAR-10 baseline performance

Test Architecture

Progressive Testing Pattern

Every module follows the same testing pattern:

• Core functionality tests: Basic operations work correctly
• Integration tests: Module exports and imports properly
• Progressive tests: Dependencies on previous modules work
• Capability tests: Real-world usage scenarios pass

Test Files Organization

tests/
├── checkpoints/           # Capability validation tests
├── integration/           # Cross-module integration tests
├── module_XX/            # Individual module test suites
└── run_all_modules.py    # Master test runner

Test Results Interpretation

Health Status

• 100.0%: All tests passing, framework fully functional
• 95%+: Minor issues, core functionality intact
• 90%+: Some failing tests, investigate specific modules
• <90%: Significant issues, check recent changes

Module Status Indicators

• ✅ Passing: Module works correctly
• ⚠️ Warning: Minor issues detected
• ❌ Failing: Module has errors requiring fixes

Best Practices

During Development

1. Test early and often: Run tito test --module X after each change
2. Use progressive testing: Ensure dependencies work before building on them
3. Validate with examples: Real usage should work, not just unit tests

Before Commits

1. Run comprehensive tests: tito test --comprehensive
2. Check all modules pass: Verify 100% health status
3. Test checkpoint progression: Ensure learning path works

Troubleshooting Failed Tests

1. Check specific module: tito test --module failing_module
2. Review integration: Look for dependency issues
3. Validate exports: Ensure tito module complete works
4. Check environment: Run tito system doctor

Testing Philosophy

The TinyTorch testing framework embodies our core principles:

KISS Principle in Testing

• Simple test patterns: Every module follows the same structure
• Clear feedback: Tests provide specific, actionable error messages
• Minimal complexity: Tests focus on essential functionality

Systems Thinking

• Integration focus: Tests verify components work together
• Real-world validation: Examples demonstrate practical usage
• Performance awareness: Tests include basic performance checks

Educational Value

• Learning verification: Tests confirm understanding, not just implementation
• Progressive validation: Capabilities build on each other systematically
• Immediate feedback: Students know instantly if their code works

This testing framework ensures that your TinyTorch implementation is not just educational, but production-ready and reliable.