github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2029] ggml-cuda.cu: "8792: !" CUDA error #26932

New Issue
Closed
opened 2026-04-22 03:43:27 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-22 03:43:27 -05:00
Owner
Copy Link

Originally created by @hsiehgeorge on GitHub (Jan 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2029

Hi,

We have a Dell XE8545 server with 4 * A100 GPU cards. When we are running "ollama run mixtral", it was fine but few minutes later, it's halt. I got multiple errors from the log:

  1. ggml-cuda.cu: "8792: !" CUDA error
  2. ollama.service: State 'stop-sigterm' timed out. Killing.

I tried to kill ollama process but can't (ollama.service: Processes still around after SIGKILL. Ignoring.), the only solution is reboot it but the same situation happens again.

Please advise how to make it works smoothly.

Thank you.

Originally created by @hsiehgeorge on GitHub (Jan 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2029 Hi, We have a Dell XE8545 server with 4 * A100 GPU cards. When we are running "ollama run mixtral", it was fine but few minutes later, it's halt. I got multiple errors from the log: 1. ggml-cuda.cu: "8792: !" CUDA error 2. ollama.service: State 'stop-sigterm' timed out. Killing. I tried to kill ollama process but can't (ollama.service: Processes still around after SIGKILL. Ignoring.), the only solution is reboot it but the same situation happens again. Please advise how to make it works smoothly. Thank you.
GiteaMirror added the nvidianeeds more info labels 2026-04-22 03:43:27 -05:00
GiteaMirror commented 2026-04-22 03:43:30 -05:00
Author
Owner
Copy Link

@Ca-ressemble-a-du-fake commented on GitHub (Jan 19, 2024):

Hi, same problem here under Debian 12 and latest version (0.1.20). It always appears after suspend. nvidia-smi (drivers 535 and 545) reads 23GiB / 24 GiB (rtx 3090) without any processes running (sudo fuser -v /dev/nvidia* returns nothing). I looks like ollama cannot be killed when the computer suspends while the GPU has a model loaded into memory (so before the ~ 5 minute timeout when ollama unloads the model). In htop the process remains with an X in the S column so it remaining in exit mode. sudo systemctl restart ollama.service or stop do not work (it never returns).

Restarting the server solves the problem until next time!

COuld the systemd ollama.service have a problem ? Because when I let ollama.service run and restart automatically (I did not change the default service given) this problem happens after suspend. On the contrary if I disable this service and run ollama serve from command line, then after suspend, ollama is not active anymore (I have to ollama serve again) and GPU usage is 0 / 24 GiB, so it works perfectly. So maybe the restart option in the service is guilty ?

<!-- gh-comment-id:1900394453 --> @Ca-ressemble-a-du-fake commented on GitHub (Jan 19, 2024): Hi, same problem here under Debian 12 and latest version (0.1.20). It always appears after suspend. nvidia-smi (drivers 535 and 545) reads 23GiB / 24 GiB (rtx 3090) without any processes running (`sudo fuser -v /dev/nvidia*` returns nothing). I looks like ollama cannot be killed when the computer suspends while the GPU has a model loaded into memory (so before the ~ 5 minute timeout when ollama unloads the model). In htop the process remains with an X in the S column so it remaining in exit mode. `sudo systemctl restart ollama.service` or `stop` do not work (it never returns). Restarting the server solves the problem until next time! COuld the systemd ollama.service have a problem ? Because when I let ollama.service run and restart automatically (I did not change the default service given) this problem happens after suspend. On the contrary if I disable this service and run ollama serve from command line, then after suspend, ollama is not active anymore (I have to `ollama serve` again) and GPU usage is 0 / 24 GiB, so it works perfectly. So maybe the restart option in the service is guilty ?
GiteaMirror commented 2026-04-22 03:43:31 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jan 27, 2024):

@hsiehgeorge can you share the server log?

https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues

<!-- gh-comment-id:1912850422 --> @dhiltgen commented on GitHub (Jan 27, 2024): @hsiehgeorge can you share the server log? https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues
GiteaMirror commented 2026-04-22 03:43:31 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Mar 11, 2024):

@hsiehgeorge and @Ca-ressemble-a-du-fake are you still encountering the issue?

<!-- gh-comment-id:1989175972 --> @pdevine commented on GitHub (Mar 11, 2024): @hsiehgeorge and @Ca-ressemble-a-du-fake are you still encountering the issue?
GiteaMirror commented 2026-04-22 03:43:32 -05:00
Author
Owner
Copy Link

@hsiehgeorge commented on GitHub (Mar 12, 2024):

@dhiltgen @pdevine the problem resolved but it's due to NVLink board hardware issue, after we replace NVLink board then it's fine. Thanks for following up.

<!-- gh-comment-id:1989762367 --> @hsiehgeorge commented on GitHub (Mar 12, 2024): @dhiltgen @pdevine the problem resolved but it's due to NVLink board hardware issue, after we replace NVLink board then it's fine. Thanks for following up.
GiteaMirror commented 2026-04-22 03:43:34 -05:00
Author
Owner
Copy Link

@Ca-ressemble-a-du-fake commented on GitHub (Mar 12, 2024):

@pdevine thanks for your following up. I was about to write that the problem persists but I tried and it seems that it works without requiring a reboot. That's great news ! I'll check again to confirm! I am currently using 0.1.28 version.

<!-- gh-comment-id:1991728850 --> @Ca-ressemble-a-du-fake commented on GitHub (Mar 12, 2024): @pdevine thanks for your following up. I was about to write that the problem persists but I tried and it seems that it works without requiring a reboot. That's great news ! I'll check again to confirm! I am currently using 0.1.28 version.
GiteaMirror commented 2026-04-22 03:43:35 -05:00
Author
Owner
Copy Link

@Ca-ressemble-a-du-fake commented on GitHub (Mar 13, 2024):

@pdevine Sadly I still have to reboot when the computer wakes up from suspended state (otherwise it fails back to CPU). That behaviour is OK for me for now (this is not my main computer). When I have time to dig deeper into that behaviour I'll give more details as it may be unrelated to that issue.

<!-- gh-comment-id:1993557245 --> @Ca-ressemble-a-du-fake commented on GitHub (Mar 13, 2024): @pdevine Sadly I still have to reboot when the computer wakes up from suspended state (otherwise it fails back to CPU). That behaviour is OK for me for now (this is not my main computer). When I have time to dig deeper into that behaviour I'll give more details as it may be unrelated to that issue.
GiteaMirror commented 2026-04-22 03:43:36 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Mar 24, 2024):

@Ca-ressemble-a-du-fake I think this is almost certainly a different issue than the original one. Would you be able to create a new issue and include the logs? Preferably w/ the OLLAMA_DEBUG=1 flag set.

<!-- gh-comment-id:2016666525 --> @pdevine commented on GitHub (Mar 24, 2024): @Ca-ressemble-a-du-fake I think this is almost certainly a different issue than the original one. Would you be able to create a new issue and include the logs? Preferably w/ the `OLLAMA_DEBUG=1` flag set.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label needs more info nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#26932
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?