github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity
Files
v0.23.1
ollama/tokenizer
History
Daniel Hiltgen ec9b4e9e47 tokenizer: fix multi-regex BPE offset handling (#15844) 
Use the current fragment offset when emitting unmatched spans during multi-regex BPE splitting. This avoids duplicating earlier prompt text and inflating token counts for multi-stage BPE tokenizers.
2026-04-27 14:14:27 -07:00
..
testdata
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
bytepairencoding_test.go
tokenizer: fix multi-regex BPE offset handling (#15844)
2026-04-27 14:14:27 -07:00
bytepairencoding.go
tokenizer: fix multi-regex BPE offset handling (#15844)
2026-04-27 14:14:27 -07:00
sentencepiece_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
sentencepiece.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
tokenizer.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
vocabulary_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
vocabulary.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
wordpiece_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
wordpiece.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00