Vijay Janapa Reddi 62636fa92a fix: Add missing typing imports to Module 10 tokenization ...

Issue: CharTokenizer was failing with NameError: name 'List' is not defined
Root cause: typing imports were not marked with #| export

Fix:
✅ Added #| export directive to import block in tokenization_dev.py
✅ Re-exported module using 'tito export 10_tokenization'
✅ typing.List, Dict, Tuple, Optional, Set now properly exported

Verification:
- CharTokenizer.build_vocab() works ✅
- encode() and decode() work ✅
- Tested on Shakespeare sample text ✅

This fixes the integration with vaswani_shakespeare.py which now properly
uses CharTokenizer from Module 10 instead of manual tokenization.

2025-10-28 09:44:24 -04:00

..

source

fix: Add missing typing imports to Module 10 tokenization

2025-10-28 09:44:24 -04:00