Daniel HiltgenandGitHub 195f4cdca1 mlx: match publisher tokenizer semantics (#18779) ...

* mlx: match publisher tokenizer semantics

Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.

Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.

* address comments

2026-10-04 07:45:15 -07:00

..

testdata

…

tokenizer_added.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_benchmark_test.go

…

tokenizer_bpe_spans_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_bpe.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_config_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_correctness_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_decode.go

…

tokenizer_encode.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_ggml_parity_test.go

…

tokenizer_load_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_load.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_normalization_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_pretokenizer_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_pretokenizer.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_reference_benchmark_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_reference_model_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer_reference_test.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00

tokenizer.go

mlx: match publisher tokenizer semantics (#18779)

2026-10-04 07:45:15 -07:00