[GH-ISSUE #9116] Phi-4 with official tokenizer changes #52443

Open
opened 2026-04-28 23:18:31 -05:00 by GiteaMirror · 1 comment
Owner

Originally created by @l2dy on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9116

Microsoft made several changes to tokenizer config in 6fbb3d3bbe.

The changes were suggested by unsloth.ai, who claim it makes Phi-4 inference better.

Originally created by @l2dy on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9116 Microsoft made several changes to tokenizer config in https://huggingface.co/microsoft/phi-4/commit/6fbb3d3bbe726c99b4188087b4deeec1bceac5ae. The changes were suggested by unsloth.ai, who claim it makes Phi-4 inference better.
GiteaMirror added the model label 2026-04-28 23:18:31 -05:00
Author
Owner

@ivanbaldo commented on GitHub (May 21, 2025):

Mmm seems it's still not fixed unfortunately.

<!-- gh-comment-id:2898755713 --> @ivanbaldo commented on GitHub (May 21, 2025): Mmm seems it's still not fixed unfortunately.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#52443