Vijay Janapa Reddi
0e135f1aea
Implement Tensor slicing with progressive disclosure and fix embedding gradient flow
...
WHAT: Added Tensor.__getitem__ (slicing) following progressive disclosure principles
MODULE 01 (Tensor):
- Added __getitem__ method for basic slicing operations
- Clean implementation with NO gradient mentions (progressive disclosure)
- Supports all NumPy-style indexing: x[0], x[:3], x[1:4], x[:, 1]
- Ensures scalar results are wrapped in arrays
MODULE 05 (Autograd):
- Added SliceBackward function for gradient computation
- Implements proper gradient scatter: zeros everywhere except sliced positions
- Added monkey-patching in enable_autograd() for __getitem__
- Follows same pattern as existing operations (add, mul, matmul)
MODULE 11 (Embeddings):
- Updated PositionalEncoding to use Tensor slicing instead of .data
- Fixed multiple .data accesses that broke computation graphs
- Removed Tensor() wrapping that created gradient-disconnected leafs
- Uses proper Tensor operations to preserve gradient flow
TESTING:
- All 6 component tests PASS (Embedding, Attention, FFN, Residual, Forward, Training)
- 19/19 parameters get gradients (was 18/19 before)
- Loss dropping better: 1.54→1.08 (vs 1.62→1.24 before)
- Model still not learning (0% accuracy) - needs fresh session to test monkey-patching
WHY THIS MATTERS:
- Tensor slicing is FUNDAMENTAL - needed by transformers for position embeddings
- Progressive disclosure maintains educational integrity
- Follows existing TinyTorch architecture patterns
- Enables position embeddings to potentially learn (pending verification)
DOCUMENTS CREATED:
- milestones/05_2017_transformer/TENSOR_SLICING_IMPLEMENTATION.md
- milestones/05_2017_transformer/STATUS.md
- milestones/05_2017_transformer/FIXES_SUMMARY.md
- milestones/05_2017_transformer/DEBUG_REVERSAL.md
- tests/milestones/test_reversal_debug.py (component tests)
ARCHITECTURAL PRINCIPLE:
Progressive disclosure is not just nice-to-have, it's CRITICAL for educational systems.
Don't expose Module 05 concepts (gradients) in Module 01 (basic operations).
Monkey-patch when features are needed, not before.
2025-11-22 18:26:12 -05:00
Vijay Janapa Reddi
41b132f55f
Update tinytorch and tito with module exports
...
Re-exported all modules after restructuring:
- Updated _modidx.py with new module locations
- Removed outdated autogeneration headers
- Updated all core modules (tensor, autograd, layers, etc.)
- Updated optimization modules (quantization, compression, etc.)
- Updated TITO commands for new structure
Changes include:
- 24 tinytorch/ module files
- 24 tito/ command and core files
- Updated references from modules/source/ to modules/
All modules re-exported via nbdev from their new locations.
2025-11-10 19:42:03 -05:00
Vijay Janapa Reddi
ae330dd477
Regenerate tinytorch package from all module exports
...
- Run tito export --all to update all exported code
- Fix file permissions (chmod u+w) to allow export writes
- Update 12 modified files with latest module code
- Add 3 new files (tinygpt, acceleration, compression)
- All 21 modules successfully exported
2025-11-10 06:23:47 -05:00
Vijay Janapa Reddi
4e1b536fac
Module 17: Export QuantizationComplete for INT8 quantization
...
- Added QuantizationComplete class with quantize/dequantize methods
- Exported quantization functions to tinytorch/optimization/quantization.py
- Provides 4x memory reduction with minimal accuracy loss
- Removed pedagogical QuantizedLinear export to avoid conflicts
- Added proper imports to export block
2025-11-06 15:50:48 -05:00
Vijay Janapa Reddi
caff73a75b
Reset package and export modules 01-07 only (skip broken spatial module)
2025-09-30 13:41:00 -04:00
Vijay Janapa Reddi
da6e4374e0
Add exported package files and cleanup
...
This commit includes:
- Exported tinytorch package files from nbdev (autograd, losses, optimizers, training, etc.)
- Updated activations.py and layers.py with __call__ methods
- New module exports: attention, spatial, tokenization, transformer, etc.
- Removed old _modidx.py file
- Cleanup of duplicate milestone directories
These are the generated package files that correspond to the source modules
we've been developing. Students will import from these when using TinyTorch.
2025-09-30 12:38:56 -04:00
