56f374efa3
🎯 NORTH STAR VISION DOCUMENTED: 'Don't Just Import It, Build It' - Training AI Engineers, not just ML users AI Engineering emerges as a foundational discipline like Computer Engineering, bridging algorithms and systems to build the AI infrastructure of the future. 🧪 ROBUST TESTING FRAMEWORK ESTABLISHED: - Created tests/regression/ for sandbox integrity tests - Implemented test-driven bug prevention workflow - Clear separation: student tests (pedagogical) vs system tests (robustness) - Every bug becomes a test to prevent recurrence ✅ KEY IMPLEMENTATIONS: - NORTH_STAR.md: Vision for AI Engineering discipline - Testing best practices: Focus on robust student sandbox - Git workflow standards: Professional development practices - Regression test suite: Prevent infrastructure issues - Conv->Linear dimension tests (found CNN bug) - Transformer reshaping tests (found GPT bug) 🏗️ SANDBOX INTEGRITY: Students need a solid, predictable environment where they focus on ML concepts, not debugging framework issues. The framework must be invisible. 📚 EDUCATIONAL PHILOSOPHY: TinyTorch isn't just teaching a framework - it's founding the AI Engineering discipline by training engineers who understand how to BUILD ML systems. This establishes the foundation for training the first generation of true AI Engineers who will define this emerging discipline.
3.3 KiB
TinyTorch Regression Tests
Ensuring Core Infrastructure Works Correctly
This directory contains regression tests that ensure TinyTorch's core functionality works correctly so students don't get stuck on infrastructure issues.
📋 Test Coverage
Shape Compatibility Tests
File: test_conv_linear_dimensions.py
What it tests: Convolution output dimensions match Linear layer expectations
Why it matters: Students shouldn't debug dimension mismatches in their CNNs
Tensor Reshaping Tests
File: test_transformer_reshaping.py
What it tests: Transformer 3D outputs work with Linear 2D layers
Why it matters: Language model architectures should "just work"
🧪 Running Regression Tests
Run All Regression Tests
pytest tests/regression/
Run Specific Bug Test
pytest tests/regression/test_issue_20241125_conv_fc_shapes.py -v
Run with Coverage
pytest tests/regression/ --cov=tinytorch --cov-report=html
📝 Adding New Regression Tests
When you discover a bug:
-
Create Test File:
test_issue_YYYYMMDD_description.py -
Use Bug Tracking Template:
"""
BUG TRACKING:
============
Bug ID: BUG-YYYY-MM-DD-XXX
Date Found: YYYY-MM-DD
Found By: [Name/System]
Severity: [Critical/High/Medium/Low]
DESCRIPTION:
[What broke and under what conditions]
REPRODUCTION:
[Exact steps to reproduce]
ROOT CAUSE:
[Why it happened]
FIX:
[What was changed to fix it]
PREVENTION:
[How this test prevents recurrence]
"""
-
Write Specific Test: Test the EXACT scenario that failed
-
Verify Test Catches Bug:
- Test should FAIL without the fix
- Test should PASS with the fix
-
Update This README: Add entry to Bug Index
🎯 Testing Philosophy
Every bug tells a story about a gap in our testing.
When we find a bug, we ask:
- Why didn't existing tests catch this?
- What test would have prevented it?
- Are there similar bugs we haven't found yet?
The goal: Build a test suite so comprehensive that bugs become impossible.
📊 Regression Test Statistics
- Total Bugs Found: 2
- Bugs with Regression Tests: 2 (100%)
- Test Coverage: 100% of discovered issues
- Last Updated: 2024-11-25
🔄 Integration with CI/CD
These regression tests run automatically on:
- Every commit to main branch
- Every pull request
- Nightly comprehensive test suite
Failures in regression tests block deployment to ensure fixed bugs never return.
🏆 Success Metrics
We measure success by:
- Zero Regressions: No bug returns after being fixed
- Fast Detection: Regression tests catch issues immediately
- Clear Documentation: Every test explains the bug it prevents
- Continuous Growth: New bugs always get new tests
📚 Learning from Bugs
Each bug teaches us something:
- Conv Shape Mismatch: Always calculate dimensions programmatically, never manually
- Transformer Reshape: Consider tensor dimensionality at module boundaries
- [Future bugs will add lessons here]
🚀 Future Improvements
- Add performance regression tests
- Create fuzz testing for edge cases
- Build automatic bug report generation
- Implement regression test metrics dashboard
Remember: A bug fixed without a test is a bug waiting to return.