github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 13:47:08 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8763] K/V Context Quantisation not working for unsloth DeepSeek R1 671B Q1 #5689

New Issue
Closed
opened 2026-04-12 16:58:53 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-04-12 16:58:53 -05:00
Owner
Copy Link

Originally created by @dmatora on GitHub (Feb 1, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8763

K/V Context Quantisation not working for unsloth DeepSeek R1 671B Q1_S

@sammcj have you tried to use the model?
Are you familiar with the issue?

this is what I get with Llama 3.1

Feb 01 19:03:49 ollama ollama[42992]: time=2025-02-01T19:03:49.949Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-a677b4a4b70c45e702b1d600f7905e367733c53898b8be60e3f29272cf334574 --ctx-size 131072 --batch-size 512 --n-gpu-layers 81 --threads 64 --flash-attn --kv-cache-type q8_0 --parallel 1 --tensor-split 7,7,7,7 --port 41091"

Feb 01 19:04:13 ollama ollama[42992]: llama_new_context_with_model: KV self size  = 21760.00 MiB, K (q8_0): 10880.00 MiB, V (q8_0): 10880.00 MiB

this is what I get with DeepSeek (used tutorial to install)

Feb 01 19:06:14 ollama ollama[42992]: time=2025-02-01T19:06:14.971Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-a542caee8df72af41ad48d75b94adacb5fbc61856930460bd599d835400fb3b6 --ctx-size 8192 --batch-size 512 --n-gpu-layers 7 --threads 64 --parallel 1 --port 40561"

Feb 01 19:07:10 ollama ollama[42992]: llama_new_context_with_model: KV self size  = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.7

Originally created by @dmatora on GitHub (Feb 1, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8763 ### K/V Context Quantisation not working for unsloth DeepSeek R1 671B Q1_S @sammcj have you tried to use the [model](https://huggingface.co/unsloth/DeepSeek-R1-GGUF)? Are you familiar with the issue? this is what I get with Llama 3.1 ``` Feb 01 19:03:49 ollama ollama[42992]: time=2025-02-01T19:03:49.949Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-a677b4a4b70c45e702b1d600f7905e367733c53898b8be60e3f29272cf334574 --ctx-size 131072 --batch-size 512 --n-gpu-layers 81 --threads 64 --flash-attn --kv-cache-type q8_0 --parallel 1 --tensor-split 7,7,7,7 --port 41091" Feb 01 19:04:13 ollama ollama[42992]: llama_new_context_with_model: KV self size = 21760.00 MiB, K (q8_0): 10880.00 MiB, V (q8_0): 10880.00 MiB ``` this is what I get with DeepSeek (used [tutorial](https://unsloth.ai/blog/deepseekr1-dynamic) to install) ``` Feb 01 19:06:14 ollama ollama[42992]: time=2025-02-01T19:06:14.971Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-a542caee8df72af41ad48d75b94adacb5fbc61856930460bd599d835400fb3b6 --ctx-size 8192 --batch-size 512 --n-gpu-layers 7 --threads 64 --parallel 1 --port 40561" Feb 01 19:07:10 ollama ollama[42992]: llama_new_context_with_model: KV self size = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.7
GiteaMirror added the bug label 2026-04-12 16:58:53 -05:00
GiteaMirror commented 2026-04-12 16:58:54 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 1, 2025):

Deepseek architecture doesn't support flash attention used for KV quantization: deepseek2.attention.key_length and deepseek2.attention.value_length are different.

<!-- gh-comment-id:2629086759 --> @rick-github commented on GitHub (Feb 1, 2025): Deepseek architecture doesn't support flash attention used for KV quantization: `deepseek2.attention.key_length` and `deepseek2.attention.value_length` are [different](https://github.com/ollama/ollama/blob/475333d533e088538e5d77c0ba65ea60833b2c77/llm/ggml.go#L539).
GiteaMirror commented 2026-04-12 16:58:55 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Feb 1, 2025):

Yep. Came to say the same thing, deepseek is just about the only modern model that doesn't support it.

<!-- gh-comment-id:2629090933 --> @sammcj commented on GitHub (Feb 1, 2025): Yep. Came to say the same thing, deepseek is just about the only modern model that doesn't support it.
GiteaMirror commented 2026-04-12 16:58:55 -05:00
Author
Owner
Copy Link

@dmatora commented on GitHub (Feb 1, 2025):

So this is the issue with original model, not unsloth's work right?
So until Llama4 no O1 on a home server?
So using entire 128K context with DeepSeek 671B requires, how much, 640Gb?

<!-- gh-comment-id:2629093009 --> @dmatora commented on GitHub (Feb 1, 2025): So this is the issue with original model, not unsloth's work right? So until Llama4 no O1 on a home server? So using entire 128K context with DeepSeek 671B requires, how much, 640Gb?
GiteaMirror commented 2026-04-12 16:58:55 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Feb 1, 2025):

Yes this is not an issue with Ollama or Unsloth, it's DeepSeek architecture does not allow for it.

<!-- gh-comment-id:2629100171 --> @sammcj commented on GitHub (Feb 1, 2025): Yes this is not an issue with Ollama or Unsloth, it's DeepSeek architecture does not allow for it.
GiteaMirror commented 2026-04-12 16:58:56 -05:00
Author
Owner
Copy Link

@kth8 commented on GitHub (Feb 1, 2025):

llama.cpp seems to be working on adding flash attention support for Deepseek-R1 https://github.com/ggerganov/llama.cpp/pull/11557

<!-- gh-comment-id:2629104923 --> @kth8 commented on GitHub (Feb 1, 2025): llama.cpp seems to be working on adding flash attention support for Deepseek-R1 https://github.com/ggerganov/llama.cpp/pull/11557
GiteaMirror commented 2026-04-12 16:58:56 -05:00
Author
Owner
Copy Link

@dmatora commented on GitHub (Feb 2, 2025):

@sammcj so once llama.cpp support for flash attention on DeepSeek is working, we'll be able to get context quantisation working, right?

<!-- gh-comment-id:2629237261 --> @dmatora commented on GitHub (Feb 2, 2025): @sammcj so once llama.cpp support for flash attention on DeepSeek is working, we'll be able to get context quantisation working, right?
GiteaMirror commented 2026-04-12 16:58:56 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Feb 2, 2025):

Can't see why not!

<!-- gh-comment-id:2629275571 --> @sammcj commented on GitHub (Feb 2, 2025): Can't see why not!
GiteaMirror commented 2026-04-12 16:58:57 -05:00
Author
Owner
Copy Link

@davidsyoung commented on GitHub (Feb 2, 2025):

Getting FA working with R1 would be incredibly useful at the moment!

<!-- gh-comment-id:2629384489 --> @davidsyoung commented on GitHub (Feb 2, 2025): Getting FA working with R1 would be incredibly useful at the moment!
GiteaMirror commented 2026-04-12 16:58:57 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Feb 2, 2025):

Yes but it's out of scope for Ollama, model support occurs in llama.cpp.

<!-- gh-comment-id:2629543406 --> @sammcj commented on GitHub (Feb 2, 2025): Yes but it's out of scope for Ollama, model support occurs in llama.cpp.
GiteaMirror commented 2026-04-12 16:58:58 -05:00
Author
Owner
Copy Link

@dmatora commented on GitHub (Mar 7, 2025):

https://github.com/ikawrakow/ik_llama.cpp/pull/241
@sammcj now that DeepSeep R1 FlashAttention is supported in llama.cpp, do you have any plans for adding support for than in ollama? :)

<!-- gh-comment-id:2706096760 --> @dmatora commented on GitHub (Mar 7, 2025): https://github.com/ikawrakow/ik_llama.cpp/pull/241 @sammcj now that DeepSeep R1 FlashAttention is supported in llama.cpp, do you have any plans for adding support for than in ollama? :)
GiteaMirror commented 2026-04-12 16:58:58 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Mar 7, 2025):

Hi @dmatora,

If flash attention can be enabled then it should work if Ollama is using an up to date llama.cpp version.

If it does not, please log a new issue with the details.

<!-- gh-comment-id:2706121603 --> @sammcj commented on GitHub (Mar 7, 2025): Hi @dmatora, If flash attention can be enabled then it should work if Ollama is using an up to date llama.cpp version. If it does not, please log a new issue with the details.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5689
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?