github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 21:51:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6139] error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_ALLOC_FAILED #81513

New Issue
Closed
opened 2026-05-09 11:16:08 -05:00 by GiteaMirror · 22 comments
GiteaMirror commented 2026-05-09 11:16:08 -05:00
Owner
Copy Link

Originally created by @trixtipsfix on GitHub (Aug 2, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6139

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

I'm getting the following error when I try to run the Ollama model:
image

Additionally, sometimes it behaves like this and disappears upon system restart:
image (1)

Error Details:

error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_ALLOC_FAILED
current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:826   
cublasCreate_v2(&cublas_handles[device]) GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: !"CUDA error"

LLM Model: deepseek-coder-v2:latest

Note: It is working with deepseek-coder model.

System Information:

  • CPU - AMD Ryzen 5600
  • OS - Ubuntu 24.04 LTS (GNU/Linux 6.8.0-39-generic x86_64)
  • GPU - Nvidia 3060 Ti 8 GB VRAM
  • CUDA - 12.5
  • RAM - 32 GB
  • Ollama version - 0.3.0

Steps to Reproduce:

  • Start the Ollama model.
    ollama run deepseek-coder-v2
  • Observe the error messages in the logs or terminal.

Expected Behavior:

  • The model should run without any CUDA-related errors.

Actual Behavior:

The model fails to run, showing the above-mentioned CUDA error. When I restart Ollama service, then it works for a while.

~$ systemctl stop ollama
~$ systemctl start ollama

Additional Context:

I've tried restarting the system and reinstalling Ollama, but the issue persists. Any insights or suggestions would be greatly appreciated.

Originally created by @trixtipsfix on GitHub (Aug 2, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6139 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? I'm getting the following error when I try to run the Ollama model: ![image](https://github.com/user-attachments/assets/4b709cd3-fb1a-41aa-995b-0355849ad0ad) Additionally, sometimes it behaves like this and disappears upon system restart: ![image (1)](https://github.com/user-attachments/assets/4830a477-efa6-454c-b76c-5c20570043ef) ### Error Details: ``` error: llama runner process has terminated: CUDA error: CUBLAS_STATUS_ALLOC_FAILED current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:826 cublasCreate_v2(&cublas_handles[device]) GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: !"CUDA error" ``` **LLM Model:** deepseek-coder-v2:latest **Note:** It is working with _deepseek-coder_ model. ### System Information: - **CPU** - AMD Ryzen 5600 - **OS** - Ubuntu 24.04 LTS (GNU/Linux 6.8.0-39-generic x86_64) - **GPU** - Nvidia 3060 Ti 8 GB VRAM - **CUDA** - 12.5 - **RAM** - 32 GB - **Ollama version** - 0.3.0 ### Steps to Reproduce: - Start the Ollama model. `ollama run deepseek-coder-v2` - Observe the error messages in the logs or terminal. ### Expected Behavior: - The model should run without any CUDA-related errors. ### Actual Behavior: The model fails to run, showing the above-mentioned CUDA error. When I restart Ollama service, then it works for a while. ``` ~$ systemctl stop ollama ~$ systemctl start ollama ``` ### Additional Context: I've tried restarting the system and reinstalling Ollama, but the issue persists. Any insights or suggestions would be greatly appreciated.
GiteaMirror added the buglinuxnvidia labels 2026-05-09 11:16:09 -05:00
GiteaMirror commented 2026-05-09 11:16:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 2, 2024):

It's more helpful if you include the full log, because we get to see what happened prior to the error. In this case, given that you have only 8G VRAM and are loading a 8.9G model, I suspect you are running in to an issue where ollama is not properly computing the required resources and llama.cpp is OOMing due to the lack of those resources. There are bugs filed for this so the issue is being looked at, in the meantime you can reduce the number of layers being offloaed to the GPU (https://github.com/ollama/ollama/issues/5913#issuecomment-2248262520) which will reduce the memory pressure.

<!-- gh-comment-id:2265398896 --> @rick-github commented on GitHub (Aug 2, 2024): It's more helpful if you include the full log, because we get to see what happened prior to the error. In this case, given that you have only 8G VRAM and are loading a 8.9G model, I suspect you are running in to an issue where ollama is not properly computing the required resources and llama.cpp is OOMing due to the lack of those resources. There are bugs filed for this so the issue is being looked at, in the meantime you can reduce the number of layers being offloaed to the GPU (https://github.com/ollama/ollama/issues/5913#issuecomment-2248262520) which will reduce the memory pressure.
GiteaMirror commented 2026-05-09 11:16:11 -05:00
Author
Owner
Copy Link

@Siddharth-Latthe-07 commented on GitHub (Aug 2, 2024):

The issue is likely related to GPU resource allocation and compatibility. The error message CUBLAS_STATUS_ALLOC_FAILED indicates that the CUDA BLAS library (cuBLAS) failed to allocate memory.

solutions:-

  1. check gpu memory usage
  2. Reduce Batch Size
  3. Update CUDA and cuBLAS
  4. Restart and upgrade Ollama and Clear GPU Memory.
  5. Test with a simpler or smaller model to verify if the issue is specific to the deepseek-coder-v2 model. If the simpler model works, the issue is likely related to the memory requirements of deepseek-coder-v2.

provide more info related to actual bug,
Hope this helps
Thanks

<!-- gh-comment-id:2265440065 --> @Siddharth-Latthe-07 commented on GitHub (Aug 2, 2024): The issue is likely related to GPU resource allocation and compatibility. The error message CUBLAS_STATUS_ALLOC_FAILED indicates that the CUDA BLAS library (cuBLAS) failed to allocate memory. solutions:- 1. check gpu memory usage 2. Reduce Batch Size 3. Update CUDA and cuBLAS 4. Restart and upgrade Ollama and Clear GPU Memory. 5. Test with a simpler or smaller model to verify if the issue is specific to the deepseek-coder-v2 model. If the simpler model works, the issue is likely related to the memory requirements of deepseek-coder-v2. provide more info related to actual bug, Hope this helps Thanks
GiteaMirror commented 2026-05-09 11:16:11 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 2, 2024):

Before Restart:
image

Ollama Logs:

ollama[72487]: INFO [main] model loaded | tid="123967923728384" timestamp=1722604627
ollama[6601]: time=2024-08-02T18:17:07.355+05:00 level=INFO source=server.go:622 msg="llama runner started in 2.76 seconds"
ollama[6601]: [GIN] 2024/08/02 - 18:17:07 | 200 |  2.868805113s |       127.0.0.1 | POST     "/api/chat"
ollama[6601]: cuda driver library failed to get device context 2time=2024-08-02T18:22:07.441+05:00 level=WARN source=gpu.go:399 msg="error looking up nvidia GPU memory"
ollama[6601]: time=2024-08-02T18:22:12.442+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.086154021 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046
ollama[6601]: time=2024-08-02T18:22:12.691+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.3356719009999996 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046
ollama[6601]: time=2024-08-02T18:22:12.942+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58604744 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046

It works after I restart my system:

image
Ollama Logs:

ollama[2435]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ollama[2435]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ollama[2435]: ggml_cuda_init: found 1 CUDA devices:
 ollama[2435]:   Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes
ollama[2435]: time=2024-08-02T19:11:20.641+05:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server loading model"
ollama[2435]: llm_load_tensors: ggml ctx size =    0.32 MiB
ollama[2435]: llm_load_tensors: offloading 20 repeating layers to GPU
ollama[2435]: llm_load_tensors: offloaded 20/28 layers to GPU
ollama[2435]: llm_load_tensors:        CPU buffer size =  2222.30 MiB
ollama[2435]: llm_load_tensors:      CUDA0 buffer size =  6283.64 MiB
ollama[2435]: llama_new_context_with_model: n_ctx      = 2048
ollama[2435]: llama_new_context_with_model: n_batch    = 512
ollama[2435]: llama_new_context_with_model: n_ubatch   = 512
ollama[2435]: llama_new_context_with_model: flash_attn = 0
ollama[2435]: llama_new_context_with_model: freq_base  = 10000.0
ollama[2435]: llama_new_context_with_model: freq_scale = 0.025
ollama[2435]: llama_kv_cache_init:  CUDA_Host KV buffer size =   140.00 MiB
ollama[2435]: llama_kv_cache_init:      CUDA0 KV buffer size =   400.00 MiB
ollama[2435]: llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
ollama[2435]: llama_new_context_with_model:  CUDA_Host  output buffer size =     0.40 MiB
ollama[2435]: llama_new_context_with_model:      CUDA0 compute buffer size =   376.06 MiB
ollama[2435]: llama_new_context_with_model:  CUDA_Host compute buffer size =    14.01 MiB
ollama[2435]: llama_new_context_with_model: graph nodes  = 1924
ollama[2435]: llama_new_context_with_model: graph splits = 112
ollama[3755]: INFO [main] model loaded | tid="126391020027904" timestamp=1722607889
ollama[2435]: time=2024-08-02T19:11:29.669+05:00 level=INFO source=server.go:622 msg="llama runner started in 9.28 seconds"
ollama[2435]: [GIN] 2024/08/02 - 19:11:29 | 200 |  9.386311926s |       127.0.0.1 | POST     "/api/chat"
ollama[2435]: [GIN] 2024/08/02 - 19:11:37 | 200 |  1.085040502s |       127.0.0.1 | POST     "/api/chat"
<!-- gh-comment-id:2265568573 --> @trixtipsfix commented on GitHub (Aug 2, 2024): **Before Restart:** ![image](https://github.com/user-attachments/assets/46ed6bdb-81b9-406b-bc62-cbdd98b598ee) Ollama Logs: ``` ollama[72487]: INFO [main] model loaded | tid="123967923728384" timestamp=1722604627 ollama[6601]: time=2024-08-02T18:17:07.355+05:00 level=INFO source=server.go:622 msg="llama runner started in 2.76 seconds" ollama[6601]: [GIN] 2024/08/02 - 18:17:07 | 200 | 2.868805113s | 127.0.0.1 | POST "/api/chat" ollama[6601]: cuda driver library failed to get device context 2time=2024-08-02T18:22:07.441+05:00 level=WARN source=gpu.go:399 msg="error looking up nvidia GPU memory" ollama[6601]: time=2024-08-02T18:22:12.442+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.086154021 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 ollama[6601]: time=2024-08-02T18:22:12.691+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.3356719009999996 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 ollama[6601]: time=2024-08-02T18:22:12.942+05:00 level=WARN source=sched.go:634 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58604744 model=/usr/share/ollama/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 ``` **It works after I restart my system:** ![image](https://github.com/user-attachments/assets/3d4236e6-0c1e-4cb5-a87d-f8455fb740d1) Ollama Logs: ``` ollama[2435]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ollama[2435]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ollama[2435]: ggml_cuda_init: found 1 CUDA devices: ollama[2435]: Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes ollama[2435]: time=2024-08-02T19:11:20.641+05:00 level=INFO source=server.go:617 msg="waiting for server to become available" status="llm server loading model" ollama[2435]: llm_load_tensors: ggml ctx size = 0.32 MiB ollama[2435]: llm_load_tensors: offloading 20 repeating layers to GPU ollama[2435]: llm_load_tensors: offloaded 20/28 layers to GPU ollama[2435]: llm_load_tensors: CPU buffer size = 2222.30 MiB ollama[2435]: llm_load_tensors: CUDA0 buffer size = 6283.64 MiB ollama[2435]: llama_new_context_with_model: n_ctx = 2048 ollama[2435]: llama_new_context_with_model: n_batch = 512 ollama[2435]: llama_new_context_with_model: n_ubatch = 512 ollama[2435]: llama_new_context_with_model: flash_attn = 0 ollama[2435]: llama_new_context_with_model: freq_base = 10000.0 ollama[2435]: llama_new_context_with_model: freq_scale = 0.025 ollama[2435]: llama_kv_cache_init: CUDA_Host KV buffer size = 140.00 MiB ollama[2435]: llama_kv_cache_init: CUDA0 KV buffer size = 400.00 MiB ollama[2435]: llama_new_context_with_model: KV self size = 540.00 MiB, K (f16): 324.00 MiB, V (f16): 216.00 MiB ollama[2435]: llama_new_context_with_model: CUDA_Host output buffer size = 0.40 MiB ollama[2435]: llama_new_context_with_model: CUDA0 compute buffer size = 376.06 MiB ollama[2435]: llama_new_context_with_model: CUDA_Host compute buffer size = 14.01 MiB ollama[2435]: llama_new_context_with_model: graph nodes = 1924 ollama[2435]: llama_new_context_with_model: graph splits = 112 ollama[3755]: INFO [main] model loaded | tid="126391020027904" timestamp=1722607889 ollama[2435]: time=2024-08-02T19:11:29.669+05:00 level=INFO source=server.go:622 msg="llama runner started in 9.28 seconds" ollama[2435]: [GIN] 2024/08/02 - 19:11:29 | 200 | 9.386311926s | 127.0.0.1 | POST "/api/chat" ollama[2435]: [GIN] 2024/08/02 - 19:11:37 | 200 | 1.085040502s | 127.0.0.1 | POST "/api/chat" ```
GiteaMirror commented 2026-05-09 11:16:12 -05:00
Author
Owner
Copy Link

@Siddharth-Latthe-07 commented on GitHub (Aug 2, 2024):

@trixtipsfix so, is it working fine?

<!-- gh-comment-id:2265575416 --> @Siddharth-Latthe-07 commented on GitHub (Aug 2, 2024): @trixtipsfix so, is it working fine?
GiteaMirror commented 2026-05-09 11:16:12 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 2, 2024):

@Siddharth-Latthe-07 It will again start failing. It's an intermittent issue.

<!-- gh-comment-id:2265580898 --> @trixtipsfix commented on GitHub (Aug 2, 2024): @Siddharth-Latthe-07 It will again start failing. It's an intermittent issue.
GiteaMirror commented 2026-05-09 11:16:12 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 2, 2024):

I'm trying to trigger it with Ollama API and the error persists.
Library: "github.com/tmc/langchaingo/llms/ollama"

GoLang Code Snippet:

response, err := b.client.Call(ctx, []schema.ChatMessage{
		schema.SystemChatMessage{Content: b.systemPrompt},
		schema.HumanChatMessage{Content: query},
	},
		llms.WithTemperature(b.modelTemperature),
		llms.WithModel(b.modelName),
		llms.WithMaxTokens(b.modelMaxTokens),
		llms.WithSeed(b.modelSeed),
	)

The response screenshots are available in my first comment.

<!-- gh-comment-id:2265602583 --> @trixtipsfix commented on GitHub (Aug 2, 2024): I'm trying to trigger it with Ollama API and the error persists. **Library:** "github.com/tmc/langchaingo/llms/ollama" **GoLang Code Snippet:** ``` response, err := b.client.Call(ctx, []schema.ChatMessage{ schema.SystemChatMessage{Content: b.systemPrompt}, schema.HumanChatMessage{Content: query}, }, llms.WithTemperature(b.modelTemperature), llms.WithModel(b.modelName), llms.WithMaxTokens(b.modelMaxTokens), llms.WithSeed(b.modelSeed), ) ``` The response screenshots are available in my first comment.
GiteaMirror commented 2026-05-09 11:16:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 2, 2024):

Clipping the log removes information that could be used to diagnose the issue. Did you try reducing the number of layers offloaded to the GPU?

<!-- gh-comment-id:2265677665 --> @rick-github commented on GitHub (Aug 2, 2024): Clipping the log removes information that could be used to diagnose the issue. Did you try reducing the number of layers offloaded to the GPU?
GiteaMirror commented 2026-05-09 11:16:14 -05:00
Author
Owner
Copy Link

@mahnoorfirdous commented on GitHub (Aug 2, 2024):

Greetings. I'm from the same team as @trixtipsfix . Here are the full logs.
ollama_logs.txt

<!-- gh-comment-id:2265877498 --> @mahnoorfirdous commented on GitHub (Aug 2, 2024): Greetings. I'm from the same team as @trixtipsfix . Here are the full logs. [ollama_logs.txt](https://github.com/user-attachments/files/16473697/ollama_logs.txt)
GiteaMirror commented 2026-05-09 11:16:15 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 2, 2024):

It might be useful to try some CUDA troubleshooting topics from this doc https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#nvidia-gpu-discovery Perhaps the uvm driver is unloading. We added some hardening for this in our install script, but it's possible that may not have run in your setup.

<!-- gh-comment-id:2266163877 --> @dhiltgen commented on GitHub (Aug 2, 2024): It might be useful to try some CUDA troubleshooting topics from this doc https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#nvidia-gpu-discovery Perhaps the uvm driver is unloading. We added some [hardening ](https://github.com/ollama/ollama/blob/main/scripts/install.sh#L328-L337)for this in our install script, but it's possible that may not have run in your setup.
GiteaMirror commented 2026-05-09 11:16:17 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 2, 2024):

There are several issues in the logs.

GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15147: false && "Deepseek2 does not support K-shift"

llama.cpp doesn't support a K-shift operation for deepseek2 architecture. Currently there's no workaround for this but a couple of PRs in https://github.com/ollama/ollama/issues/5975 may help.

CUDA error: out of memory

OOM errors typically follow when ollama is trying to offload 20 layers to the GPU because ollama thinks it has 7.3 GiB to play with:

Aug 02 14:39:38 mahnoor-budgie ollama[6601]: time=2024-08-02T14:39:38.621+05:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=20 layers.split="" memory.available="[7.3 GiB]" memory.required.full="9.6 GiB" memory.required.partial="7.1 GiB" memory.required.kv="540.0 MiB" memory.required.allocations="[7.1 GiB]" memory.weights.total="8.5 GiB" memory.weights.repeating="8.4 GiB" memory.weights.nonrepeating="164.1 MiB" memory.graph.full="212.0 MiB" memory.graph.partial="376.1 MiB"

At other times, it only offloads 19 layers because it sees less available memory (7.1 GiB), and these times are not followed by an OOM:

Aug 02 16:18:43 mahnoor-budgie ollama[6601]: time=2024-08-02T16:18:43.012+05:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=19 layers.split="" memory.available="[7.1 GiB]" memory.required.full="9.6 GiB" memory.required.partial="6.8 GiB" memory.required.kv="540.0 MiB" memory.required.allocations="[6.8 GiB]" memory.weights.total="8.5 GiB" memory.weights.repeating="8.4 GiB" memory.weights.nonrepeating="164.1 MiB" memory.graph.full="212.0 MiB" memory.graph.partial="376.1 MiB"

The fluctuation in available memory on the GPU is likely due to other processes taking some, you can check this with nvidia-smi. The OOMs may be mitigated by setting the number of layers to offload as described in https://github.com/ollama/ollama/issues/5913#issuecomment-2248262520.

CUDA error: CUBLAS_STATUS_NOT_INITIALIZED

It's not clear why cublas was unable to create a handle. Running the server with OLLAMA_DEBUG=1 may provide more info.

TBH, deepseek2 currently seems more trouble than it's worth. There are several tickets because of this model, perhaps look to other models while the issues are sorted out.

<!-- gh-comment-id:2266236877 --> @rick-github commented on GitHub (Aug 2, 2024): There are several issues in the logs. ``` GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15147: false && "Deepseek2 does not support K-shift" ``` llama.cpp doesn't support a K-shift operation for [deepseek2 architecture](https://github.com/ggerganov/llama.cpp/blob/398ede5efeb07b9adf9fbda7ea63f630d476a792/src/llama.cpp#L15099). Currently there's no workaround for this but a couple of PRs in https://github.com/ollama/ollama/issues/5975 may help. ``` CUDA error: out of memory ``` OOM errors typically follow when ollama is trying to offload 20 layers to the GPU because ollama thinks it has 7.3 GiB to play with: ``` Aug 02 14:39:38 mahnoor-budgie ollama[6601]: time=2024-08-02T14:39:38.621+05:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=20 layers.split="" memory.available="[7.3 GiB]" memory.required.full="9.6 GiB" memory.required.partial="7.1 GiB" memory.required.kv="540.0 MiB" memory.required.allocations="[7.1 GiB]" memory.weights.total="8.5 GiB" memory.weights.repeating="8.4 GiB" memory.weights.nonrepeating="164.1 MiB" memory.graph.full="212.0 MiB" memory.graph.partial="376.1 MiB" ``` At other times, it only offloads 19 layers because it sees less available memory (7.1 GiB), and these times are not followed by an OOM: ``` Aug 02 16:18:43 mahnoor-budgie ollama[6601]: time=2024-08-02T16:18:43.012+05:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=19 layers.split="" memory.available="[7.1 GiB]" memory.required.full="9.6 GiB" memory.required.partial="6.8 GiB" memory.required.kv="540.0 MiB" memory.required.allocations="[6.8 GiB]" memory.weights.total="8.5 GiB" memory.weights.repeating="8.4 GiB" memory.weights.nonrepeating="164.1 MiB" memory.graph.full="212.0 MiB" memory.graph.partial="376.1 MiB" ``` The fluctuation in available memory on the GPU is likely due to other processes taking some, you can check this with `nvidia-smi`. The OOMs may be mitigated by setting the number of layers to offload as described in https://github.com/ollama/ollama/issues/5913#issuecomment-2248262520. ``` CUDA error: CUBLAS_STATUS_NOT_INITIALIZED ``` It's not clear why cublas was unable to create a handle. Running the server with `OLLAMA_DEBUG=1` may provide more info. TBH, deepseek2 currently seems more trouble than it's worth. There are several [tickets](https://github.com/ollama/ollama/issues?q=is%3Aissue+deepseek2) because of this model, perhaps look to other models while the issues are sorted out.
GiteaMirror commented 2026-05-09 11:16:18 -05:00
Author
Owner
Copy Link

@mahnoorfirdous commented on GitHub (Aug 6, 2024):

Is there something to consider if we install ollama and install CUDA later? Does that make a difference? Do we need to recompile anything?

<!-- gh-comment-id:2270955935 --> @mahnoorfirdous commented on GitHub (Aug 6, 2024): Is there something to consider if we install ollama and install CUDA later? Does that make a difference? Do we need to recompile anything?
GiteaMirror commented 2026-05-09 11:16:19 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 6, 2024):

The order of installation shouldn't matter, as long as you restart the ollama server when all packages are done. If you install CUDA after ollama and don't restart, ollama won't be able to use the GPU. Recompiling won't help either, because most of the problems observed in the log are not a function of the compilation.

  1. Unsupported K-shift. May be fixed by #5975, ie code changes.
  2. OOM. May be fixed by adjusting memory calculations, ie code changes. Workaround by limiting layer offloading.
  3. Cublas initialization error. Unknown what will fix this, more information (OLLAMA_DEBUG=1) required.
<!-- gh-comment-id:2271263602 --> @rick-github commented on GitHub (Aug 6, 2024): The order of installation shouldn't matter, as long as you restart the ollama server when all packages are done. If you install CUDA after ollama and don't restart, ollama won't be able to use the GPU. Recompiling won't help either, because most of the problems observed in the log are not a function of the compilation. 1. Unsupported K-shift. May be fixed by [#5975](https://github.com/ollama/ollama/issues/5975), ie code changes. 2. OOM. May be fixed by adjusting memory calculations, ie code changes. Workaround by limiting layer offloading. 3. Cublas initialization error. Unknown what will fix this, more information (`OLLAMA_DEBUG=1`) required.
GiteaMirror commented 2026-05-09 11:16:20 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 6, 2024):

@rick-github Why is that the quality of the response by the model (DeepSeek2) decreases upon each request? Like, the response to first request seems fine but upon further requests, the model doesn't follow the prompt properly.

Upon restarting Ollama, I see the same behaviour i.e. Good response for the initial requests and then the decline in the quality of the response.

Also, Can you suggest a good model for source code analysis based on guidelines (Google C++ Guidelines, MISRA Guidelines, etc.)? As you were saying, "deepseek2 currently seems more trouble than it's worth".

<!-- gh-comment-id:2271340701 --> @trixtipsfix commented on GitHub (Aug 6, 2024): @rick-github Why is that the quality of the response by the model (DeepSeek2) decreases upon each request? Like, the response to first request seems fine but upon further requests, the model doesn't follow the prompt properly. Upon restarting Ollama, I see the same behaviour i.e. Good response for the initial requests and then the decline in the quality of the response. Also, Can you suggest a good model for source code analysis based on guidelines (Google C++ Guidelines, MISRA Guidelines, etc.)? As you were saying, "deepseek2 currently seems more trouble than it's worth".
GiteaMirror commented 2026-05-09 11:16:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 6, 2024):

My guess at the degrading results would be due to the default context window of 2048. If you are dumping a lot of code in to it for analysis, that will quickly fill the window and the responses will lose coherence. You can increase the window either my adding "option":{"num_ctx": 32768} (or whatever length you need) to the API call, or modify the model to have a different default length:

$ ollama show --modelfile deepseek-coder-v2 > Modelfile
# edit the file and add "PARAMETER num_ctx 32768", and modify the FROM line to use the model name, not path
$ ollama create deepseek-coder-v2-c32768 -f Modelfile

Be aware that VRAM usage scales quadratically to the length of the context window, so if you choose a large value you will see a dramtaic increase in memory usage. In most models you can combat this by enabling flash attention (OLLAMA_FLASH_ATTENTION=1) but again, deepseek2 is an architecture that doesn't support this feature.

I don't have any recommendations for a code analysis model since I haven't had a chance to do any evaluations. I did play around with codegemma when it was released and it worked for my limited use case, but I'm sure other users can provide more guidance.

<!-- gh-comment-id:2271406206 --> @rick-github commented on GitHub (Aug 6, 2024): My guess at the degrading results would be due to the default context window of 2048. If you are dumping a lot of code in to it for analysis, that will quickly fill the window and the responses will lose coherence. You can increase the window either my adding `"option":{"num_ctx": 32768}` (or whatever length you need) to the API call, or modify the model to have a different default length: ```sh $ ollama show --modelfile deepseek-coder-v2 > Modelfile # edit the file and add "PARAMETER num_ctx 32768", and modify the FROM line to use the model name, not path $ ollama create deepseek-coder-v2-c32768 -f Modelfile ``` Be aware that VRAM usage scales quadratically to the length of the context window, so if you choose a large value you will see a dramtaic increase in memory usage. In most models you can combat this by enabling flash attention (`OLLAMA_FLASH_ATTENTION=1`) but again, deepseek2 is an architecture that doesn't support this feature. I don't have any recommendations for a code analysis model since I haven't had a chance to do any evaluations. I did play around with codegemma when it was released and it worked for my limited use case, but I'm sure other users can provide more guidance.
GiteaMirror commented 2026-05-09 11:16:21 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 6, 2024):

@rick-github The context size is 4096, and the request I'm sending have almost same token size. I also tried with exact same request multiple time, but the responses was getting poorer.

To investigate further, I set the num_gpu (= 10) and num_batch (= 2) in the request. This time, the response quality is average, but are consistent. But request is taking more time than usual.

Please let me know your thoughts about this?

<!-- gh-comment-id:2271455767 --> @trixtipsfix commented on GitHub (Aug 6, 2024): @rick-github The context size is 4096, and the request I'm sending have almost same token size. I also tried with exact same request multiple time, but the responses was getting poorer. To investigate further, I set the **num_gpu** (= 10) and **num_batch** (= 2) in the request. This time, the response quality is average, but are consistent. But request is taking more time than usual. Please let me know your thoughts about this?
GiteaMirror commented 2026-05-09 11:16:22 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 6, 2024):

Do you have logs from recent attempts? The log that mahnoorfirdous provided shows a context window of 2048, not 4096. How are you calling the model, via API or ollama run?

<!-- gh-comment-id:2271498688 --> @rick-github commented on GitHub (Aug 6, 2024): Do you have logs from recent attempts? The log that mahnoorfirdous provided shows a context window of 2048, not 4096. How are you calling the model, via API or `ollama run`?
GiteaMirror commented 2026-05-09 11:16:22 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 6, 2024):

@rick-github Here are the logs of my recent attempts, via API (using Postman)
ollama_logs.txt

You're right. The previous logs had the default context length, and were generated through ollama run

<!-- gh-comment-id:2271608751 --> @trixtipsfix commented on GitHub (Aug 6, 2024): @rick-github Here are the logs of my recent attempts, via API (using Postman) [ollama_logs.txt](https://github.com/user-attachments/files/16512604/ollama_logs.txt) You're right. The previous logs had the default context length, and were generated through `ollama run`
GiteaMirror commented 2026-05-09 11:16:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 6, 2024):

Nothing jumps out. Could you add OLLAMA_DEBUG=1 to the server environment and then re-run your tests? That will show the prompt that llama.cpp is receiving and some statistics about the token processing that it's doing.

<!-- gh-comment-id:2271671660 --> @rick-github commented on GitHub (Aug 6, 2024): Nothing jumps out. Could you add `OLLAMA_DEBUG=1` to the server environment and then re-run your tests? That will show the prompt that llama.cpp is receiving and some statistics about the token processing that it's doing.
GiteaMirror commented 2026-05-09 11:16:23 -05:00
Author
Owner
Copy Link

@trixtipsfix commented on GitHub (Aug 8, 2024):

@rick-github Here are the Ollama Debug Logs.
ollama_debug_logs.txt

Thank you for your prompt and effective assistance!

<!-- gh-comment-id:2275029360 --> @trixtipsfix commented on GitHub (Aug 8, 2024): @rick-github Here are the Ollama Debug Logs. [ollama_debug_logs.txt](https://github.com/user-attachments/files/16538356/ollama_debug_logs.txt) Thank you for your prompt and effective assistance!
GiteaMirror commented 2026-05-09 11:16:23 -05:00
Author
Owner
Copy Link

@micr0-dev commented on GitHub (Sep 22, 2024):

Bump, I'm having the same exact issue, it's intermittent and only pops up sometimes

<!-- gh-comment-id:2366848464 --> @micr0-dev commented on GitHub (Sep 22, 2024): Bump, I'm having the same exact issue, it's intermittent and only pops up sometimes
GiteaMirror commented 2026-05-09 11:16:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 23, 2024):

Server logs will aid in debugging.

<!-- gh-comment-id:2367074263 --> @rick-github commented on GitHub (Sep 23, 2024): [Server logs](https://github.com/ollama/ollama/blob/ad935f45ac19a8ba090db32580f3a6469e9858bb/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
GiteaMirror commented 2026-05-09 11:16:24 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 31, 2024):

It looks like there may be multiple issues in here. We'll track the "Deepseek2 does not support K-shift" in #5975
If there are other problems that aren't related to deepseek context shifting, please retry on 0.4.0 and if that doesn't clear it up, share an updated log and I'll reopen the issue.

<!-- gh-comment-id:2450544249 --> @dhiltgen commented on GitHub (Oct 31, 2024): It looks like there may be multiple issues in here. We'll track the "Deepseek2 does not support K-shift" in #5975 If there are other problems that aren't related to deepseek context shifting, please retry on 0.4.0 and if that doesn't clear it up, share an updated log and I'll reopen the issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug linux nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#81513
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?