mirror of
https://github.com/MLSysBook/TinyTorch.git
synced 2026-08-01 18:59:41 -05:00
Vijay Janapa Reddi
a16bfc8a32
- Enhanced module-developer agent with Dr. Sarah Rodriguez persona - Added comprehensive educational frameworks and Golden Rules - Implemented Progressive Disclosure Principle (no forward references) - Added Immediate Testing Pattern (test after each implementation) - Integrated package structure template (📦 where code exports to) - Applied clean NBGrader structure with proper scaffolding - Fixed tensor module formatting and scope boundaries - Removed confusing transparent analysis patterns - Added visual impact icons system for consistent motivation 🎯 Ready to apply these proven educational principles to all modules
32 lines
1.0 KiB
YAML
32 lines
1.0 KiB
YAML
name: "Tokenization"
|
|
number: 11
|
|
description: "Text processing systems that convert raw text into numerical sequences for language models"
|
|
learning_objectives:
|
|
- "Implement character-level tokenization with special token handling"
|
|
- "Build BPE (Byte Pair Encoding) tokenizer for subword units"
|
|
- "Understand tokenization trade-offs: vocabulary size vs sequence length"
|
|
- "Optimize tokenization performance for production systems"
|
|
- "Analyze how tokenization affects model memory and training efficiency"
|
|
|
|
prerequisites:
|
|
- "02_tensor"
|
|
|
|
exports:
|
|
- "CharTokenizer"
|
|
- "BPETokenizer"
|
|
- "TokenizationProfiler"
|
|
- "OptimizedTokenizer"
|
|
|
|
systems_concepts:
|
|
- "Memory efficiency of token representations"
|
|
- "Vocabulary size vs model size tradeoffs"
|
|
- "Tokenization throughput optimization"
|
|
- "String processing performance"
|
|
- "Cache-friendly text processing patterns"
|
|
|
|
ml_systems_focus: "Text processing pipelines, tokenization throughput, memory-efficient vocabulary management"
|
|
|
|
estimated_time: "4-5 hours"
|
|
|
|
next_modules:
|
|
- "12_embeddings" |