github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-18 06:06:18 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8921] Changing the env variable ‘OLLAMA_KV_CACHE_TYPE’ has no effect #31546

New Issue
Closed
opened 2026-04-22 12:05:25 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-22 12:05:25 -05:00
Owner
Copy Link

Originally created by @Root970103 on GitHub (Feb 7, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8921

What is the issue?

The env variable ‘OLLAMA_KV_CACHE_TYPE’ should indicate the dtype of kv cache, but I can't see any difference.
I changed the ~/.bashrc file as below and excute source ~/.bashrc：

export OLLAMA_HOST=127.0.0.1:20010
export OLLAMA_MODELS=/data/shared/cache/ollama/models
export OLLAMA_NUM_PARALLEL=8
export OLLAMA_KV_CACHE_TYPE=q4_0

Relevant log output

The log showed that the kv cache was still fp16

llama_kv_cache_init: kv_size = 16384, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 64, can_shift = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   576.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA2 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA3 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA4 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA5 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA6 KV buffer size =   512.00 MiB
llama_kv_cache_init:      CUDA7 KV buffer size =   448.00 MiB
llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     4.80 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      CUDA0 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA2 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA3 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA4 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA5 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA6 compute buffer size =  1488.01 MiB
llama_new_context_with_model:      CUDA7 compute buffer size =  1488.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   138.02 MiB
llama_new_context_with_model: graph nodes  = 2246
llama_new_context_with_model: graph splits = 9

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.5.7

Originally created by @Root970103 on GitHub (Feb 7, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8921 ### What is the issue? The env variable ‘OLLAMA_KV_CACHE_TYPE’ should indicate the dtype of kv cache, but I can't see any difference. I changed the `~/.bashrc` file as below and excute `source ~/.bashrc`： ```shell export OLLAMA_HOST=127.0.0.1:20010 export OLLAMA_MODELS=/data/shared/cache/ollama/models export OLLAMA_NUM_PARALLEL=8 export OLLAMA_KV_CACHE_TYPE=q4_0 ``` ### Relevant log output ```shell The log showed that the kv cache was still fp16 llama_kv_cache_init: kv_size = 16384, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 64, can_shift = 1 llama_kv_cache_init: CUDA0 KV buffer size = 576.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA2 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA3 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA4 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA5 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA6 KV buffer size = 512.00 MiB llama_kv_cache_init: CUDA7 KV buffer size = 448.00 MiB llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 4.80 MiB llama_new_context_with_model: pipeline parallelism enabled (n_copies=4) llama_new_context_with_model: CUDA0 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA1 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA2 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA3 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA4 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA5 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA6 compute buffer size = 1488.01 MiB llama_new_context_with_model: CUDA7 compute buffer size = 1488.02 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 138.02 MiB llama_new_context_with_model: graph nodes = 2246 llama_new_context_with_model: graph splits = 9 ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.7
GiteaMirror added the bug label 2026-04-22 12:05:25 -05:00
GiteaMirror commented 2026-04-22 12:05:28 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 7, 2025):

You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization.

<!-- gh-comment-id:2642275774 --> @rick-github commented on GitHub (Feb 7, 2025): You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization.
GiteaMirror commented 2026-04-22 12:05:29 -05:00
Author
Owner
Copy Link

@Root970103 commented on GitHub (Feb 8, 2025):

You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization.

Yes, thanks!

<!-- gh-comment-id:2644716781 --> @Root970103 commented on GitHub (Feb 8, 2025): > You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization. Yes, thanks!
GiteaMirror commented 2026-04-22 12:05:30 -05:00
Author
Owner
Copy Link

@Displacer commented on GitHub (Feb 9, 2025):

You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization.

It is not true, inferencing IQ2_XXS DeepSeek r1 model with llama.cpp type_k = 'q4_0' is available. It is also mentioned in https://unsloth.ai/blog/deepseekr1-dynamic Please reopen.

<!-- gh-comment-id:2646157781 --> @Displacer commented on GitHub (Feb 9, 2025): > You don't say which model but based on popularity I guess deepseek. Deepseek doesn't support KV quantization. It is not true, inferencing IQ2_XXS DeepSeek r1 model with llama.cpp type_k = 'q4_0' is available. It is also mentioned in https://unsloth.ai/blog/deepseekr1-dynamic Please reopen.
GiteaMirror commented 2026-04-22 12:05:30 -05:00
Author
Owner
Copy Link

@Displacer commented on GitHub (Feb 10, 2025):

OLLAMA_FLASH_ATTENTION=1 helps for me in case of Ubuntu 22.04 LTS with OLLAMA_KV_CACHE_TYPE

<!-- gh-comment-id:2648936190 --> @Displacer commented on GitHub (Feb 10, 2025): OLLAMA_FLASH_ATTENTION=1 helps for me in case of Ubuntu 22.04 LTS with OLLAMA_KV_CACHE_TYPE
GiteaMirror commented 2026-04-22 12:05:31 -05:00
Author
Owner
Copy Link

@Readon commented on GitHub (Feb 13, 2025):

I have checked the source code of ollama, why does kv_cache_type depends on the flash attention? It is in this code

On my machine, the card did not support flash attention, however it could run the model with parameter --cache-type-k q4_0.

<!-- gh-comment-id:2655826182 --> @Readon commented on GitHub (Feb 13, 2025): I have checked the source code of ollama, why does kv_cache_type depends on the flash attention? It is in [this code](https://github.com/ollama/ollama/blob/82658c3eec0cbb70ba558e5310fe3e68436aa583/llm/server.go#L196) On my machine, the card did not support flash attention, however it could run the model with parameter --cache-type-k q4_0.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#31546
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?