github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9768] Llama runner - exit status 2 #6386

New Issue
Closed
opened 2026-04-12 17:53:48 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 17:53:48 -05:00
Owner
Copy Link

Originally created by @MagnusLarsAndersson on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9768

server.log

What is the issue?

The Discord chat describes an issue that has still not been resolved. Moreover, I have been able to sort out the underlying problem is not as simple as a mismatch between the latest NVdia drivers and Ollama, as when downgrading to earlier drivers still result in that Ollama version >5.7 results in "llama runner process has terminated: exit status 2". However the problem is clearly related to use of GPU as it is possible to load model with num_gpu=0 and it fails with status 2 for num_gpu=1. Interestingly, the problems started when I upgraded to newest Nvidia drivers, that is, newer Ollama ran earlier with driver 552.62, but not any longer. Has also tried to completely uninstall Ollama and re-install with the same result.

Ollama binaries for Windows used

Am I really the only one with this problem? Is there any workaround other than to stick to version 5.7?

Thanks in advance,

Magnus

CPU: AMD Ryzen Threadripper PRO 5975WX
GPU: NVIDIA RTX 6000 Ada (Driver 553.62 (same problem with 570))
Cuda compilation tools, release 12.6, V12.6.77 Build cuda_12.6.r12.6/compiler.34841621_0
Windows: 11 Version 23H2

Relevant log output

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.5.8 - 0.6.0

Originally created by @MagnusLarsAndersson on GitHub (Mar 14, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9768 [server.log](https://github.com/user-attachments/files/19249016/server.log) ### What is the issue? The Discord [chat](https://discord.com/channels/1128867683291627614/1340361644882137241) describes an issue that has still not been resolved. Moreover, I have been able to sort out the underlying problem is not as simple as a mismatch between the latest NVdia drivers and Ollama, as when downgrading to earlier drivers still result in that Ollama version >5.7 results in "llama runner process has terminated: exit status 2". However the problem is clearly related to use of GPU as it is possible to load model with num_gpu=0 and it fails with status 2 for num_gpu=1. Interestingly, the problems started when I upgraded to newest Nvidia drivers, that is, newer Ollama ran earlier with driver 552.62, but not any longer. Has also tried to completely uninstall Ollama and re-install with the same result. Ollama binaries for Windows used Am I really the only one with this problem? Is there any workaround other than to stick to version 5.7? Thanks in advance, Magnus CPU: AMD Ryzen Threadripper PRO 5975WX GPU: NVIDIA RTX 6000 Ada (Driver 553.62 (same problem with 570)) Cuda compilation tools, release 12.6, V12.6.77 Build cuda_12.6.r12.6/compiler.34841621_0 Windows: 11 Version 23H2 ### Relevant log output ```shell ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.8 - 0.6.0
GiteaMirror added the bug label 2026-04-12 17:53:48 -05:00
GiteaMirror commented 2026-04-12 17:53:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 14, 2025):

load_backend: loaded CUDA backend from C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll
ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll
ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-sandybridge.dll
ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-skylakex.dll

The runner was able to load the GPU backend but none of the CPU backends. I don't know if this is contributing the the problem you are seeing, but it's a data point. I've seen a couple of reports of this problem, AFAIK nobody has root-caused it but there is a workaround that resolves it for some users - modify the PATH variable in the server environment by adding C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama to it.

Exception 0xc0000005 0x0 0x21 0x7ff66869b562
PC=0x7ff66869b562
signal arrived during external code execution

This is the problem the runner is having. 0xc0000005 is an access violation, like a segv in Linux - the program is trying to reference a memory location that is not valid in the address space assigned to the process. This usually means trying to dereference a pointer that is 0, but it could also be a corrupted register pointing into the wild blue yonder. It happens during llama_model_load_from_file() which then calls private functions so locating the failure point requires some disassembly or modifying the code and adding some trace statements.

<!-- gh-comment-id:2725073938 --> @rick-github commented on GitHub (Mar 14, 2025): ``` load_backend: loaded CUDA backend from C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-alderlake.dll ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-sandybridge.dll ggml_backend_load_best: failed to load C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-skylakex.dll ``` The runner was able to load the GPU backend but none of the CPU backends. I don't know if this is contributing the the problem you are seeing, but it's a data point. I've seen a couple of reports of this problem, AFAIK nobody has root-caused it but there is a workaround that resolves it for some users - modify the PATH variable in the server environment by adding `C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama` to it. ``` Exception 0xc0000005 0x0 0x21 0x7ff66869b562 PC=0x7ff66869b562 signal arrived during external code execution ``` This is the problem the runner is having. 0xc0000005 is an access violation, like a segv in Linux - the program is trying to reference a memory location that is not valid in the address space assigned to the process. This usually means trying to dereference a pointer that is 0, but it could also be a corrupted register pointing into the wild blue yonder. It happens during [`llama_model_load_from_file()`](https://github.com/ggml-org/llama.cpp/blob/8fcb563613e20a04dd9791f0a9b8a41086428c09/src/llama.cpp#L234) which then calls private functions so locating the failure point requires some disassembly or modifying the code and adding some trace statements.
GiteaMirror commented 2026-04-12 17:53:51 -05:00
Author
Owner
Copy Link

@MagnusLarsAndersson commented on GitHub (Mar 14, 2025):

Rick,

Seems like it solved it!

I had seen the path problem among all incidents, but never imagined it could be related to mine, especially as it happened in connection with updating NVIDIA drivers. Next step will be to see what happens if I update the drivers to the latest version.

/magnus

<!-- gh-comment-id:2725199673 --> @MagnusLarsAndersson commented on GitHub (Mar 14, 2025): Rick, Seems like it solved it! I had seen the path problem among all incidents, but never imagined it could be related to mine, especially as it happened in connection with updating NVIDIA drivers. Next step will be to see what happens if I update the drivers to the latest version. /magnus
GiteaMirror commented 2026-04-12 17:53:52 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Mar 16, 2025):

Thanks @rick-github and thank you @MagnusLarsAndersson for the update. Do you know what driver version you were on before (might be hard to find this) and what you're on now?

<!-- gh-comment-id:2727119122 --> @jmorganca commented on GitHub (Mar 16, 2025): Thanks @rick-github and thank you @MagnusLarsAndersson for the update. Do you know what driver version you were on before (might be hard to find this) and what you're on now?
GiteaMirror commented 2026-04-12 17:53:53 -05:00
Author
Owner
Copy Link

@MagnusLarsAndersson commented on GitHub (Mar 16, 2025):

@jmorganca The interesting background was that I never had any problems with Ollama up to version 5.11 using Nvidia driver R550 U13 (553.62). Then I decided to upgrade to R570 U1 (572.16), resulting in all versions > 5.7 ended up with Error 2 when trying to use GPU, but worked fine with only CPU. However, even more weird is that when I downgraded to 553.62 again and re-installed Ollama, the problem remained until I followed the suggestion by @rick-github. I have no idea how these things are related, and I have not checked what will happen if I upgrade to R570 again, since as far as I can tell it does not an critical changes in my case. Moreover, there have been reports about many problems with Geforce versions on R570, so I will wait before upgrading. Going through the changes that came with upgrading, the only other thing is that it installed the Nvidia App and that option did I not make use of in my downgrade. Finally, I have checked all env variables for paths and there is nothing explains why the fix was needed.

<!-- gh-comment-id:2727296982 --> @MagnusLarsAndersson commented on GitHub (Mar 16, 2025): @jmorganca The interesting background was that I never had any problems with Ollama up to version 5.11 using Nvidia driver R550 U13 (553.62). Then I decided to upgrade to R570 U1 (572.16), resulting in all versions > 5.7 ended up with Error 2 when trying to use GPU, but worked fine with only CPU. However, even more weird is that when I downgraded to 553.62 again and re-installed Ollama, the problem remained until I followed the suggestion by @rick-github. I have no idea how these things are related, and I have not checked what will happen if I upgrade to R570 again, since as far as I can tell it does not an critical changes in my case. Moreover, there have been reports about many problems with Geforce versions on R570, so I will wait before upgrading. Going through the changes that came with upgrading, the only other thing is that it installed the Nvidia App and that option did I not make use of in my downgrade. Finally, I have checked all env variables for paths and there is nothing explains why the fix was needed.
GiteaMirror commented 2026-04-12 17:53:54 -05:00
Author
Owner
Copy Link

@patryksliwinski commented on GitHub (Mar 17, 2025):

Just wanted to say I had the exact same history, it's very weird but since I added C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama to path it works again! What's stranger that the small model worked perfect in my case, but when using ones that couldn't fit on GPU then ollama randomly crashed in middle of creating a message.

<!-- gh-comment-id:2728491627 --> @patryksliwinski commented on GitHub (Mar 17, 2025): Just wanted to say I had the exact same history, it's very weird but since I added `C:\Users\magnu\AppData\Local\Programs\Ollama\lib\ollama` to path it works again! What's stranger that the small model worked perfect in my case, but when using ones that couldn't fit on GPU then ollama randomly crashed in middle of creating a message.
GiteaMirror commented 2026-04-12 17:53:54 -05:00
Author
Owner
Copy Link

@MagnusLarsAndersson commented on GitHub (Mar 17, 2025):

@patryksliwinski I did not try with any models that could not fit in GPU (>48GB), hence cannot confirm/deny if this was the case for me.

<!-- gh-comment-id:2728836670 --> @MagnusLarsAndersson commented on GitHub (Mar 17, 2025): @patryksliwinski I did not try with any models that could not fit in GPU (>48GB), hence cannot confirm/deny if this was the case for me.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6386
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?