- Update TransformerBlock to use mlp_ratio instead of hidden_dim
- Update PositionalEncoding argument order
- Fix MultiHeadAttention to use self-attention API
- Add missing MultiHeadAttention import
- tests/performance/: Referenced non-existent modules/ directory
- tests/system/: Required tinytorch.nn.functional which does not exist
- tests/regression/test_conv_linear_dimensions.py: Same issue
- These tests predated the API consolidation
Removed files created during debugging:
- tests/regression/GRADIENT_FLOW_TEST_SUMMARY.md (info now in test docstrings)
- tests/debug_posenc.py (temporary debug script)
Test organization is clean:
- Module tests: tests/XX_modulename/
- Integration tests: tests/integration/
- Regression tests: tests/regression/ (gradient flow tests)
- Milestone tests: tests/milestones/
- System tests: tests/system/
All actual test files remain and pass.
Removed files created during debugging:
- tests/regression/GRADIENT_FLOW_TEST_SUMMARY.md (info now in test docstrings)
- tests/debug_posenc.py (temporary debug script)
Test organization is clean:
- Module tests: tests/XX_modulename/
- Integration tests: tests/integration/
- Regression tests: tests/regression/ (gradient flow tests)
- Milestone tests: tests/milestones/
- System tests: tests/system/
All actual test files remain and pass.
TransposeBackward:
- New backward function for transpose operation
- Patch Tensor.transpose() to track gradients
- Critical for attention (Q @ K.T) gradient flow
MatmulBackward batched fix:
- Change np.dot to np.matmul for batched 3D+ tensors
- Use np.swapaxes instead of .T for proper batched transpose
- Fixes gradient shapes in attention mechanisms
Tests added:
- tests/05_autograd/test_batched_matmul_backward.py (3 tests)
- Updated tests/regression/test_gradient_flow_fixes.py (9 tests total)
All gradient flow issues for transformer training are now resolved!
TransposeBackward:
- New backward function for transpose operation
- Patch Tensor.transpose() to track gradients
- Critical for attention (Q @ K.T) gradient flow
MatmulBackward batched fix:
- Change np.dot to np.matmul for batched 3D+ tensors
- Use np.swapaxes instead of .T for proper batched transpose
- Fixes gradient shapes in attention mechanisms
Tests added:
- tests/05_autograd/test_batched_matmul_backward.py (3 tests)
- Updated tests/regression/test_gradient_flow_fixes.py (9 tests total)
All gradient flow issues for transformer training are now resolved!
🎯 NORTH STAR VISION DOCUMENTED:
'Don't Just Import It, Build It' - Training AI Engineers, not just ML users
AI Engineering emerges as a foundational discipline like Computer Engineering,
bridging algorithms and systems to build the AI infrastructure of the future.
🧪 ROBUST TESTING FRAMEWORK ESTABLISHED:
- Created tests/regression/ for sandbox integrity tests
- Implemented test-driven bug prevention workflow
- Clear separation: student tests (pedagogical) vs system tests (robustness)
- Every bug becomes a test to prevent recurrence
✅ KEY IMPLEMENTATIONS:
- NORTH_STAR.md: Vision for AI Engineering discipline
- Testing best practices: Focus on robust student sandbox
- Git workflow standards: Professional development practices
- Regression test suite: Prevent infrastructure issues
- Conv->Linear dimension tests (found CNN bug)
- Transformer reshaping tests (found GPT bug)
🏗️ SANDBOX INTEGRITY:
Students need a solid, predictable environment where they focus on ML concepts,
not debugging framework issues. The framework must be invisible.
📚 EDUCATIONAL PHILOSOPHY:
TinyTorch isn't just teaching a framework - it's founding the AI Engineering
discipline by training engineers who understand how to BUILD ML systems.
This establishes the foundation for training the first generation of true
AI Engineers who will define this emerging discipline.
🎯 NORTH STAR VISION DOCUMENTED:
'Don't Just Import It, Build It' - Training AI Engineers, not just ML users
AI Engineering emerges as a foundational discipline like Computer Engineering,
bridging algorithms and systems to build the AI infrastructure of the future.
🧪 ROBUST TESTING FRAMEWORK ESTABLISHED:
- Created tests/regression/ for sandbox integrity tests
- Implemented test-driven bug prevention workflow
- Clear separation: student tests (pedagogical) vs system tests (robustness)
- Every bug becomes a test to prevent recurrence
✅ KEY IMPLEMENTATIONS:
- NORTH_STAR.md: Vision for AI Engineering discipline
- Testing best practices: Focus on robust student sandbox
- Git workflow standards: Professional development practices
- Regression test suite: Prevent infrastructure issues
- Conv->Linear dimension tests (found CNN bug)
- Transformer reshaping tests (found GPT bug)
🏗️ SANDBOX INTEGRITY:
Students need a solid, predictable environment where they focus on ML concepts,
not debugging framework issues. The framework must be invisible.
📚 EDUCATIONAL PHILOSOPHY:
TinyTorch isn't just teaching a framework - it's founding the AI Engineering
discipline by training engineers who understand how to BUILD ML systems.
This establishes the foundation for training the first generation of true
AI Engineers who will define this emerging discipline.
