github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7938] KV Cache quants run into issues every couple of messages. #5080

New Issue
Closed
opened 2026-04-12 16:10:31 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-12 16:10:31 -05:00
Owner
Copy Link

Originally created by @SingularityMan on GitHub (Dec 5, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7938

What is the issue?

This is the error message I run into when running my scripts with KV cache q4_0 or q8_0. These are the models:

gemma2:27b-instruct-q4_0 (will be switching to q4_K_S)
minicpm-v-2.6-8b-q8_0

Traceback (most recent call last):

File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 564, in <module>

config.asyncio.run(main())

File "C:\Users\user\.conda\envs\vector_companion\lib\asyncio\runners.py", line 44, in run

return loop.run_until_complete(main)

File "C:\Users\user\.conda\envs\vector_companion\lib\asyncio\base_events.py", line 647, in run_until_complete

return future.result()

File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 520, in main

await queue_agent_responses(

File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 178, in queue_agent_responses

await config.asyncio.gather(process_sentences(), play_audio_queue())

File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 157, in process_sentences

async for sentence in sentence_generator:

File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\config\config.py", line 109, in fetch_stream

for chunk in stream:

File "C:\Users\user\.conda\envs\vector_companion\lib\site-packages\ollama\_client.py", line 90, in _stream

raise ResponseError(e)

ollama._types.ResponseError: an error was encountered while running the model: read tcp 127.0.0.1:34105->127.0.0.1:34102: wsarecv: An existing connection was forcibly closed by the remote host.

So when I look at the server log I see this:

C:\a\ollama\ollama\llama\ggml-cuda\cpy.cu:531: ggml_cuda_cpy: unsupported type combination (q4_0 to f32)

time=2024-12-04T19:38:14.673-05:00 level=DEBUG source=server.go:1092 msg="stopping llama server"
[GIN] 2024/12/04 - 19:38:14 | 200 |     5.073219s |       127.0.0.1 | POST     "/api/chat"
time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:407 msg="context for request finished"
time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\carlo\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc duration=2562047h47m16.854775807s
time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\carlo\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0

This is per the RC uploaded today.

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.4.8 RC

Originally created by @SingularityMan on GitHub (Dec 5, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7938 ### What is the issue? This is the error message I run into when running my scripts with KV cache q4_0 or q8_0. These are the models: ``` gemma2:27b-instruct-q4_0 (will be switching to q4_K_S) minicpm-v-2.6-8b-q8_0 ``` ``` Traceback (most recent call last): File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 564, in <module> config.asyncio.run(main()) File "C:\Users\user\.conda\envs\vector_companion\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\user\.conda\envs\vector_companion\lib\asyncio\base_events.py", line 647, in run_until_complete return future.result() File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 520, in main await queue_agent_responses( File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 178, in queue_agent_responses await config.asyncio.gather(process_sentences(), play_audio_queue()) File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\main.py", line 157, in process_sentences async for sentence in sentence_generator: File "C:\Users\user\PycharmProjects\vector_companion\vector_companion\config\config.py", line 109, in fetch_stream for chunk in stream: File "C:\Users\user\.conda\envs\vector_companion\lib\site-packages\ollama\_client.py", line 90, in _stream raise ResponseError(e) ollama._types.ResponseError: an error was encountered while running the model: read tcp 127.0.0.1:34105->127.0.0.1:34102: wsarecv: An existing connection was forcibly closed by the remote host. ``` So when I look at the server log I see this: ``` C:\a\ollama\ollama\llama\ggml-cuda\cpy.cu:531: ggml_cuda_cpy: unsupported type combination (q4_0 to f32) time=2024-12-04T19:38:14.673-05:00 level=DEBUG source=server.go:1092 msg="stopping llama server" [GIN] 2024/12/04 - 19:38:14 | 200 | 5.073219s | 127.0.0.1 | POST "/api/chat" time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:407 msg="context for request finished" time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\carlo\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc duration=2562047h47m16.854775807s time=2024-12-04T19:38:14.674-05:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\carlo\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0 ``` This is per the RC uploaded today. ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.4.8 RC
GiteaMirror added the bug label 2026-04-12 16:10:31 -05:00
GiteaMirror commented 2026-04-12 16:10:32 -05:00
Author
Owner
Copy Link

@mchiang0610 commented on GitHub (Dec 5, 2024):

Thank you for reporting this!

<!-- gh-comment-id:2518875517 --> @mchiang0610 commented on GitHub (Dec 5, 2024): Thank you for reporting this!
GiteaMirror commented 2026-04-12 16:10:33 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Dec 5, 2024):

It looks like this happens when doing context shifting - unimplemented operation. I see it with q4_0 on CUDA and both q4_0 and q8_0 on Metal.

<!-- gh-comment-id:2519040335 --> @jessegross commented on GitHub (Dec 5, 2024): It looks like this happens when doing context shifting - unimplemented operation. I see it with q4_0 on CUDA and both q4_0 and q8_0 on Metal.
GiteaMirror commented 2026-04-12 16:10:34 -05:00
Author
Owner
Copy Link

@SingularityMan commented on GitHub (Dec 5, 2024):

It looks like this happens when doing context shifting - unimplemented operation. I see it with q4_0 on CUDA and both q4_0 and q8_0 on Metal.

That seems to be the case. Now what?

<!-- gh-comment-id:2519062455 --> @SingularityMan commented on GitHub (Dec 5, 2024): > It looks like this happens when doing context shifting - unimplemented operation. I see it with q4_0 on CUDA and both q4_0 and q8_0 on Metal. That seems to be the case. Now what?
GiteaMirror commented 2026-04-12 16:10:35 -05:00
Author
Owner
Copy Link

@mengqin commented on GitHub (Dec 12, 2024):

Now the llama.cpp doesn't support some type of KV cache quants (in fact only q8_0 support is merged with this PR https://github.com/ggerganov/llama.cpp/pull/9571) with KV shift enabled, this caused by cuda backend cpy kernel doesn't support some specified data type (q4 to F32). And now they have a PR in draft to track this issue (https://github.com/ggerganov/llama.cpp/pull/5653) but it's seems in low priority, and they give out the workaround is disable KV shift (--no-context-shift) if using kv cache. I think ollama doesn't do that caused this problem.
Because ollama only support q4_0, q8_0 and f16 types quants of KV cache, f16 and q8_0 is supported, so they won't cause the problem, only q4_0 will cause the problem, so ollama need wait llama.cpp to fix it by support q4_0.
Or disable the context shift when KV cache quants q4_0, this can be done by ollama side.
In my personal bias, I tested fixing the q4_0 cpy kernel and it works fine on my side, it's pretty easy. But I meet another issue in ggml_compute_forward_dup(also caued by not support KV quants type q4_0/q8_0 to F32) function when tested large context in some model chat (mistral:22b-small-instruct-2409_iq4_XS), I believe it's another issue not related this one. I'm trying to fix it too, both this problem and dup function problem block the test in large context chat.
If some one prefer I can make a PR for this. But seems this should be done by llama.cpp side but not here, right?

<!-- gh-comment-id:2538782632 --> @mengqin commented on GitHub (Dec 12, 2024): Now the llama.cpp doesn't support some type of KV cache quants (in fact only q8_0 support is merged with this PR https://github.com/ggerganov/llama.cpp/pull/9571) with KV shift enabled, this caused by cuda backend cpy kernel doesn't support some specified data type (q4 to F32). And now they have a PR in draft to track this issue (https://github.com/ggerganov/llama.cpp/pull/5653) but it's seems in low priority, and they give out the workaround is disable KV shift (--no-context-shift) if using kv cache. I think ollama doesn't do that caused this problem. Because ollama only support q4_0, q8_0 and f16 types quants of KV cache, f16 and q8_0 is supported, so they won't cause the problem, only q4_0 will cause the problem, so ollama need wait llama.cpp to fix it by support q4_0. Or disable the context shift when KV cache quants q4_0, this can be done by ollama side. In my personal bias, I tested fixing the q4_0 cpy kernel and it works fine on my side, it's pretty easy. But I meet another issue in ggml_compute_forward_dup(also caued by not support KV quants type q4_0/q8_0 to F32) function when tested large context in some model chat (mistral:22b-small-instruct-2409_iq4_XS), I believe it's another issue not related this one. I'm trying to fix it too, both this problem and dup function problem block the test in large context chat. If some one prefer I can make a PR for this. But seems this should be done by llama.cpp side but not here, right?
GiteaMirror commented 2026-04-12 16:10:36 -05:00
Author
Owner
Copy Link

@SingularityMan commented on GitHub (Dec 12, 2024):

Ok so disabling q4_0 seems to have done the trick. This is acceptable for now but I would still like to see this fixed later in its entirety. q8_0 and fp16 work just fine as reported.

<!-- gh-comment-id:2538929061 --> @SingularityMan commented on GitHub (Dec 12, 2024): Ok so disabling q4_0 seems to have done the trick. This is acceptable for now but I would still like to see this fixed later in its entirety. q8_0 and fp16 work just fine as reported.
GiteaMirror commented 2026-04-12 16:10:37 -05:00
Author
Owner
Copy Link

@mengqin commented on GitHub (Dec 12, 2024):

PR created https://github.com/ollama/ollama/pull/8073.

<!-- gh-comment-id:2539455509 --> @mengqin commented on GitHub (Dec 12, 2024): PR created https://github.com/ollama/ollama/pull/8073.
GiteaMirror commented 2026-04-12 16:10:39 -05:00
Author
Owner
Copy Link

@SingularityMan commented on GitHub (Dec 12, 2024):

PR created #8073.

Much appreciated!

<!-- gh-comment-id:2539542946 --> @SingularityMan commented on GitHub (Dec 12, 2024): > PR created #8073. Much appreciated!
GiteaMirror commented 2026-04-12 16:10:41 -05:00
Author
Owner
Copy Link

@cc-umu commented on GitHub (Feb 13, 2025):

This is also present on Mac silicon and fails for q8_0 (and q4 _0 I guess) KV cache quant.

From ollama.log:

ggml_metal_init: found device: Apple M3 Max
.
.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen2.5 14B Instruct
.
.
llama_new_context_with_model: flash_attn = 1
.
.
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 48, can_shift = 1
llama_kv_cache_init: Metal KV buffer size = 816.00 MiB
llama_new_context_with_model: KV self size = 816.00 MiB, K (q8_0): 408.00 MiB, V (q8_0): 408.00 MiB
.

[GIN] 2025/02/13 - 09:25:47 | 200 | 6.422153125s | 127.0.0.1 | POST "/api/chat"
[GIN] 2025/02/13 - 09:27:04 | 200 | 1.571591709s | 127.0.0.1 | POST "/api/chat"
ggml_metal_encode_node: error: unsupported op 'CPY'
ggml-metal_darwin_arm64.m:1292: unsupported op
SIGABRT: abort

<!-- gh-comment-id:2655906938 --> @cc-umu commented on GitHub (Feb 13, 2025): This is also present on Mac silicon and fails for q8_0 (and q4 _0 I guess) KV cache quant. From ollama.log: ggml_metal_init: found device: Apple M3 Max . . llama_model_loader: - kv 0: general.architecture str = qwen2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen2.5 14B Instruct . . llama_new_context_with_model: flash_attn = 1 . . llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 48, can_shift = 1 llama_kv_cache_init: Metal KV buffer size = 816.00 MiB llama_new_context_with_model: KV self size = 816.00 MiB, K (q8_0): 408.00 MiB, V (q8_0): 408.00 MiB . [GIN] 2025/02/13 - 09:25:47 | 200 | 6.422153125s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/02/13 - 09:27:04 | 200 | 1.571591709s | 127.0.0.1 | POST "/api/chat" ggml_metal_encode_node: error: unsupported op 'CPY' ggml-metal_darwin_arm64.m:1292: unsupported op SIGABRT: abort
GiteaMirror commented 2026-04-12 16:10:41 -05:00
Author
Owner
Copy Link

@jbutlerdev commented on GitHub (Mar 3, 2025):

Looks like this has been fixed in llama.cpp

https://github.com/ggml-org/llama.cpp/issues/10976

<!-- gh-comment-id:2693073262 --> @jbutlerdev commented on GitHub (Mar 3, 2025): Looks like this has been fixed in llama.cpp https://github.com/ggml-org/llama.cpp/issues/10976
GiteaMirror commented 2026-04-12 16:10:42 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Mar 3, 2025):

Thanks for pointing that out @jbutlerdev. We bumped our version of llama.cpp not too long ago, so I verified that this is now fixed in the current Ollama codebase.

<!-- gh-comment-id:2695084945 --> @jessegross commented on GitHub (Mar 3, 2025): Thanks for pointing that out @jbutlerdev. We bumped our version of llama.cpp not too long ago, so I verified that this is now fixed in the current Ollama codebase.
GiteaMirror commented 2026-04-12 16:10:42 -05:00
Author
Owner
Copy Link

@jbutlerdev commented on GitHub (Mar 3, 2025):

@jessegross I'm running 0.5.12 and still seeing this issue

<!-- gh-comment-id:2695124340 --> @jbutlerdev commented on GitHub (Mar 3, 2025): @jessegross I'm running 0.5.12 and still seeing this issue
GiteaMirror commented 2026-04-12 16:10:42 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Mar 3, 2025):

It will be in the next release.

<!-- gh-comment-id:2695133810 --> @jessegross commented on GitHub (Mar 3, 2025): It will be in the next release.
GiteaMirror commented 2026-04-12 16:10:43 -05:00
Author
Owner
Copy Link

@jbutlerdev commented on GitHub (Mar 3, 2025):

Great! Thank you!

<!-- gh-comment-id:2695141020 --> @jbutlerdev commented on GitHub (Mar 3, 2025): Great! Thank you!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5080
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?