Merge transformer-training into dev

Complete Milestone 05 - 2017 Transformer implementation

Major Features:
- TinyTalks interactive dashboard with rich CLI
- Complete gradient flow fixes (13 tests passing)
- Multiple training examples (5-min, 10-min, levels 1-2)
- Milestone celebration card (perceptron style)
- Comprehensive documentation

Gradient Flow Fixes:
- Fixed reshape, matmul (3D), embedding, sqrt, mean, sub, div, GELU
- All transformer components now fully differentiable
- Hybrid attention approach for educational clarity + gradients

Training Results:
- 10-min training: 96.6% loss improvement, 62.5% accuracy
- 5-min training: 97.8% loss improvement, 66.7% accuracy
- Working chatbot with coherent responses

Files Added:
- tinytalks_dashboard.py (main demo)
- tinytalks_chatbot.py, tinytalks_dataset.py
- level1_memorization.py, level2_patterns.py
- Comprehensive docs and test suites

Ready for student use 2>&1
This commit is contained in:
Vijay Janapa Reddi
2025-10-30 17:48:11 -04:00
36 changed files with 7365 additions and 2240 deletions

View File

@@ -15,6 +15,12 @@
#| default_exp text.tokenization
#| export
import numpy as np
from typing import List, Dict, Tuple, Optional, Set
import json
import re
from collections import defaultdict, Counter
# %% [markdown]
"""
# Module 10: Tokenization - Converting Text to Numbers