mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-05-09 01:52:37 -05:00
92781736a1
Major changes: - Moved TinyGPT from Module 16 to examples/tinygpt (capstone demo) - Fixed Module 10 (optimizers) and Module 11 (training) bugs - All 16 modules now passing tests (100% health) - Added comprehensive testing with 'tito test --comprehensive' - Renamed example files for clarity (train_xor_network.py, etc.) - Created working TinyGPT example structure - Updated documentation to reflect 15 core modules + examples - Added KISS principle and testing framework documentation
TinyGPT - Language Model Example
The Capstone Achievement: Build GPT from your own components!
TinyGPT demonstrates that the ML framework you built from scratch can create a working language model. This isn't a toy - it's a real transformer that generates text.
What TinyGPT Proves
This example shows that everything you built works together:
- Tensors & Autograd (Modules 2, 9) - Gradient computation through transformers
- Layers & Attention (Modules 4-7) - Multi-head attention and feed-forward networks
- Optimizers & Training (Modules 10-11) - Adam optimizer training language models
- Data Processing (Module 8) - Tokenization and sequence handling
Quick Start
# Train TinyGPT on Shakespeare
python train_shakespeare.py
# Generate text interactively
python generate.py
# See a simple working example
python train_simple.py
Architecture
TinyGPT implements a GPT-style transformer:
- Character-level tokenization (simplest approach)
- Multi-head self-attention
- Positional encodings
- Layer normalization
- Autoregressive generation
Files
tinygpt.py- The complete model implementationtrain_shakespeare.py- Train on Shakespeare texttrain_simple.py- Simple pattern learning (for debugging)generate.py- Interactive text generationutils.py- Tokenization and data utilities
Performance
With the components you built:
- Learns simple patterns (like "abcabc...") perfectly
- Generates character-level text after training
- Shows clear learning curves on small datasets
This proves your framework is complete and working!