github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9815] Gemma3 Model Tokenization Issue with Unicode token. #6424

New Issue
Closed
opened 2026-04-12 17:59:04 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 17:59:04 -05:00
Owner
Copy Link

Originally created by @LETS-BEE on GitHub (Mar 17, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9815

Originally assigned to: @jmorganca on GitHub.

What is the issue?

The Gemma3 model exhibits an issue where specific Korean words are not properly recognized. Specifically, complex Korean words expressed as Unicode characters are being tokenized at the Unicode level instead of being recognized as complete words.

Steps to Reproduce:

  1. Run the Gemma-3 model using Ollama (e.g., ollama run gemma3:27b).
  2. Input the following Korean sentence: " '괭' 을 초성, 중성, 종성으로 나눠서 설명해줘. "
  3. Check the model's response. You will find that the input sentence is not properly recognized or is awkwardly expressed due to being tokenized at the Unicode level.

Expected Result:

  • Content related to '괭' should be printed.

Actual Result:

  • The model fails to properly recognize the input Korean sentence or awkwardly expresses it due to tokenization at the Unicode level. For example, It tries to analyze the empty ' ' without indicating '괭' or indicates the next letter '을'. It is also separated into ['<0xEA>', '<0xB4>', '<0xAD>'].

Additional Information:

  • The same sentence is recognized correctly by the Mistral-Small3 or qwq models.
  • Other Korean users have confirmed that the issue is occurring, with the same Unicode-based tokenization observed.
  • This suggests that the tokenizer in the Gemma-3 model may not adequately reflect the characteristics of the Korean language.
  • It seems that it doesn't recognize letters like '뫄' and '뼉' as well as '괭'.
  • I confirmed that Google's aistudio recognizes it normally.
  • Question 5 in the screenshot below is an example.
    Image

Relevant log output

OS

Linux

GPU

AMD

CPU

AMD

Ollama version

0.6.1

Originally created by @LETS-BEE on GitHub (Mar 17, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9815 Originally assigned to: @jmorganca on GitHub. ### What is the issue? The Gemma3 model exhibits an issue where specific Korean words are not properly recognized. Specifically, complex Korean words expressed as Unicode characters are being tokenized at the Unicode level instead of being recognized as complete words. **Steps to Reproduce:** 1. Run the Gemma-3 model using Ollama (e.g., ollama run gemma3:27b). 2. Input the following Korean sentence: " '괭' 을 초성, 중성, 종성으로 나눠서 설명해줘. " 3. Check the model's response. You will find that the input sentence is not properly recognized or is awkwardly expressed due to being tokenized at the Unicode level. **Expected Result:** - Content related to '괭' should be printed. **Actual Result:** - The model fails to properly recognize the input Korean sentence or awkwardly expresses it due to tokenization at the Unicode level. For example, It tries to analyze the empty ' ' without indicating '괭' or indicates the next letter '을'. It is also separated into ['<0xEA>', '<0xB4>', '<0xAD>']. **Additional Information:** - The same sentence is recognized correctly by the Mistral-Small3 or qwq models. - Other Korean users have confirmed that the issue is occurring, with the same Unicode-based tokenization observed. - This suggests that the tokenizer in the Gemma-3 model may not adequately reflect the characteristics of the Korean language. - It seems that it doesn't recognize letters like '뫄' and '뼉' as well as '괭'. - I confirmed that Google's aistudio recognizes it normally. - Question 5 in the screenshot below is an example. ![Image](https://github.com/user-attachments/assets/48a66c15-7a54-4e3a-8852-3dfc092f1a2b) ### Relevant log output ```shell ``` ### OS Linux ### GPU AMD ### CPU AMD ### Ollama version 0.6.1
GiteaMirror added the bug label 2026-04-12 17:59:04 -05:00
GiteaMirror commented 2026-04-12 17:59:05 -05:00
Author
Owner
Copy Link

@LETS-BEE commented on GitHub (Mar 18, 2025):

For less frequently used characters (e.g. '꿹', '𪚥', '𰻞'), the tokenizer generates Unicode tokens, but Ollama does not seem to recognize them. I can see that they are recognized normally in AI Studio or other Spaces in Huggingface.

Can check the tokenized text here: https://huggingface.co/spaces/barttee/tokenizers

<!-- gh-comment-id:2732429997 --> @LETS-BEE commented on GitHub (Mar 18, 2025): For less frequently used characters (e.g. '꿹', '𪚥', '𰻞'), the tokenizer generates Unicode tokens, but Ollama does not seem to recognize them. I can see that they are recognized normally in AI Studio or other Spaces in Huggingface. Can check the tokenized text here: https://huggingface.co/spaces/barttee/tokenizers
GiteaMirror commented 2026-04-12 17:59:05 -05:00
Author
Owner
Copy Link

@bjj commented on GitHub (Mar 18, 2025):

I also see this ℵ (U+2135). For example, ask the model How are the cardinalities of infinity written? and ollama will produce <0xE2><0x84><0xAC>. Note that that's not even the right Unicode codepoint. It is trying to write ℬ (U+212C). This is consistent with another gguf I tried with llama.cpp directly, which produces ℬ where it means ℵ.

So in this case, there are two issues (incorrect model output from both ggufs, and ollama also writes out the UTF-8 bytes).

Unfortunately I can't directly test the ollama library gemma3:27b q4_k_m gguf in llama.cpp because it fails with:
llama_model_load: error loading model: error loading model hyperparameters: key not found in model: gemma3.attention.layer_norm_rms_epsilon

An FP8 quant running in vLLM can correctly produce ℵ in this case (although it appears to produce a spurious RTL marker which causes the subscript to go on the wrong side).

qwen2.5:32b produces correct output on ollama with the same question.

<!-- gh-comment-id:2733769847 --> @bjj commented on GitHub (Mar 18, 2025): I also see this ℵ (U+2135). For example, ask the model `How are the cardinalities of infinity written?` and `ollama` will produce `<0xE2><0x84><0xAC>`. Note that that's not even the right Unicode codepoint. It is trying to write ℬ (U+212C). This is consistent with another gguf I tried with llama.cpp directly, which produces ℬ where it means ℵ. So in this case, there are two issues (incorrect model output from both ggufs, **and** ollama also writes out the UTF-8 bytes). Unfortunately I can't directly test the ollama library gemma3:27b q4_k_m gguf in llama.cpp because it fails with: `llama_model_load: error loading model: error loading model hyperparameters: key not found in model: gemma3.attention.layer_norm_rms_epsilon` An FP8 quant running in vLLM can correctly produce ℵ in this case (although it appears to produce a spurious RTL marker which causes the subscript to go on the wrong side). qwen2.5:32b produces correct output on ollama with the same question.
GiteaMirror commented 2026-04-12 17:59:06 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Apr 2, 2025):

Hi all, sorry about the issue. This should be fixed in https://github.com/ollama/ollama/pull/10081 and released with Ollama 0.6.4 which is currently being built

<!-- gh-comment-id:2773746778 --> @jmorganca commented on GitHub (Apr 2, 2025): Hi all, sorry about the issue. This should be fixed in https://github.com/ollama/ollama/pull/10081 and released with Ollama 0.6.4 which is currently being built
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6424
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?