github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-14 21:08:04 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #14121] QWEN 0.6B embedding using ollama and llama-cpp shows different embedding results #34973

New Issue
Closed
opened 2026-04-22 19:04:24 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 19:04:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @ehsankf on GitHub (Feb 6, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14121

What is the issue?

This assertion fails

from llama_cpp import Llama
import ollama
llm_cpp = Llama.from_pretrained(repo_id="Qwen/Qwen3-Embedding-0.6B-GGUF",filename="Qwen3-Embedding-0.6B-f16.gguf", embedding=True)
query = "The capital city of France is Paris."
embedding = llm_cpp.create_embedding([query])['data'][0]['embedding']
l2_norm = np.linalg.norm(embedding, ord=2)
normalized_embedding = embedding / l2_norm
ollama_embed = ollama.embed("qwen3-embedding:0.6b", options={"num_ctx": 8192}, input=[query])
assert np.allclose(normalized_embedding, ollama_embed.embeddings[0], atol=1e-5)

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @ehsankf on GitHub (Feb 6, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14121 ### What is the issue? This assertion fails ``` from llama_cpp import Llama import ollama llm_cpp = Llama.from_pretrained(repo_id="Qwen/Qwen3-Embedding-0.6B-GGUF",filename="Qwen3-Embedding-0.6B-f16.gguf", embedding=True) query = "The capital city of France is Paris." embedding = llm_cpp.create_embedding([query])['data'][0]['embedding'] l2_norm = np.linalg.norm(embedding, ord=2) normalized_embedding = embedding / l2_norm ollama_embed = ollama.embed("qwen3-embedding:0.6b", options={"num_ctx": 8192}, input=[query]) assert np.allclose(normalized_embedding, ollama_embed.embeddings[0], atol=1e-5) ``` ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-22 19:04:24 -05:00
GiteaMirror commented 2026-04-22 19:04:26 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 6, 2026):

--- 14121.py.orig	2026-02-06 19:36:42.467197310 +0100
+++ 14121.py	2026-02-06 19:36:19.763198933 +0100
@@ -6,5 +6,5 @@
 embedding = llm_cpp.create_embedding([query])['data'][0]['embedding']
 l2_norm = np.linalg.norm(embedding, ord=2)
 normalized_embedding = embedding / l2_norm
-ollama_embed = ollama.embed("qwen3-embedding:0.6b", options={"num_ctx": 8192}, input=[query])
+ollama_embed = ollama.embed("qwen3-embedding:0.6b-fp16", options={"num_ctx": 8192}, input=[query])
 assert np.allclose(normalized_embedding, ollama_embed.embeddings[0], atol=1e-5)
<!-- gh-comment-id:3862006680 --> @rick-github commented on GitHub (Feb 6, 2026): ```diff --- 14121.py.orig 2026-02-06 19:36:42.467197310 +0100 +++ 14121.py 2026-02-06 19:36:19.763198933 +0100 @@ -6,5 +6,5 @@ embedding = llm_cpp.create_embedding([query])['data'][0]['embedding'] l2_norm = np.linalg.norm(embedding, ord=2) normalized_embedding = embedding / l2_norm -ollama_embed = ollama.embed("qwen3-embedding:0.6b", options={"num_ctx": 8192}, input=[query]) +ollama_embed = ollama.embed("qwen3-embedding:0.6b-fp16", options={"num_ctx": 8192}, input=[query]) assert np.allclose(normalized_embedding, ollama_embed.embeddings[0], atol=1e-5) ```
GiteaMirror commented 2026-04-22 19:04:26 -05:00
Author
Owner
Copy Link
Copy Source

@ehsankf commented on GitHub (Feb 6, 2026):

I set up this test and it show the deviation grows significantly as the context grows

from llama_cpp import Llama
import ollama
llm = Llama.from_pretrained(repo_id="Qwen/Qwen3-Embedding-0.6B-GGUF",filename="Qwen3-Embedding-0.6B-f16.gguf",
                            embedding=True, n_ctx=8192, n_gpu_layers=0, verbose=False)
query_total = ''.join(__import__('random').choices('abcdefghijklmnopqrstuvwxyz ', k=8000))
for i in range(100, 5000, 100):
      query = query_total[:I]
      embedding = llm.create_embedding([query])['data'][0]['embedding']
      l2_norm = np.linalg.norm(embedding, ord=2)
      normalized_embedding = embedding / l2_norm
      response = ollama.embed("qwen3-embedding:0.6b-fp16", options={"num_ctx": 8192}, input=query)
      diff = np.sum(np.abs(np.sign(response.embeddings[0])-np.sign(normalized_embedding))>0)
      results["Length"].append(i)
      results["Sign differences"].append(diff)

Length,Sign differences
100,4
200,2
300,3
400,3
500,3
600,2
700,2
800,1
900,6
1000,418
1100,417
1200,417
1300,415
1400,411
1500,410
1600,410
1700,403
1800,408
1900,410
2000,406
2100,400
2200,402
2300,401
2400,400
2500,396
2600,393
2700,392
2800,403
2900,404
3000,402
3100,401
3200,397
3300,401
3400,398
3500,397
3600,394
3700,396
3800,399
3900,404
4000,401
4100,401
4200,401
4300,400
4400,396
4500,398
4600,397
4700,399
4800,402
4900,398

<!-- gh-comment-id:3862080461 --> @ehsankf commented on GitHub (Feb 6, 2026): I set up this test and it show the deviation grows significantly as the context grows ``` from llama_cpp import Llama import ollama llm = Llama.from_pretrained(repo_id="Qwen/Qwen3-Embedding-0.6B-GGUF",filename="Qwen3-Embedding-0.6B-f16.gguf", embedding=True, n_ctx=8192, n_gpu_layers=0, verbose=False) query_total = ''.join(__import__('random').choices('abcdefghijklmnopqrstuvwxyz ', k=8000)) for i in range(100, 5000, 100): query = query_total[:I] embedding = llm.create_embedding([query])['data'][0]['embedding'] l2_norm = np.linalg.norm(embedding, ord=2) normalized_embedding = embedding / l2_norm response = ollama.embed("qwen3-embedding:0.6b-fp16", options={"num_ctx": 8192}, input=query) diff = np.sum(np.abs(np.sign(response.embeddings[0])-np.sign(normalized_embedding))>0) results["Length"].append(i) results["Sign differences"].append(diff) ``` **Length,Sign differences** 100,4 200,2 300,3 400,3 500,3 600,2 700,2 800,1 900,6 1000,418 1100,417 1200,417 1300,415 1400,411 1500,410 1600,410 1700,403 1800,408 1900,410 2000,406 2100,400 2200,402 2300,401 2400,400 2500,396 2600,393 2700,392 2800,403 2900,404 3000,402 3100,401 3200,397 3300,401 3400,398 3500,397 3600,394 3700,396 3800,399 3900,404 4000,401 4100,401 4200,401 4300,400 4400,396 4500,398 4600,397 4700,399 4800,402 4900,398
GiteaMirror commented 2026-04-22 19:04:27 -05:00
Author
Owner
Copy Link
Copy Source

@ehsankf commented on GitHub (Feb 6, 2026):

I think I found the issue. llama-cpp-python sets n_batch=512. When input length (in tokens) exceeds this limit, the library has to split the prompt into multiple processing chunks. For embedding tasks, if this splitting isn't handled perfectly (specifically regarding positional embeddings/RoPE), the resulting vector is corrupted. So to avoid this we had to n_ctx = n_batch

<!-- gh-comment-id:3862344204 --> @ehsankf commented on GitHub (Feb 6, 2026): I think I found the issue.` llama-cpp-python` sets n_batch=512. When input length (in tokens) exceeds this limit, the library has to split the prompt into multiple processing chunks. For embedding tasks, if this splitting isn't handled perfectly (specifically regarding positional embeddings/RoPE), the resulting vector is corrupted. So to avoid this we had to `n_ctx = n_batch`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#34973