[GH-ISSUE #15016] GPU used with ollama run, but /v1 API forces CPU fallback (same model) #71707

New Issue

GiteaMirror · 2026-05-05T02:23:14-05:00

GiteaMirror commented

2026-05-05 02:23:14 -05:00

Originally created by @LukaszRT on GitHub (Mar 22, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15016

What is the issue?

Title:
GPU works with ollama run, but falls back to CPU when using OpenAI /v1 API (same model)

Description:

When running a model manually using:

ollama run qwen3.5:27b

the model runs correctly on the GPU (100% GPU usage).

However, when using the same model via the OpenAI-compatible /v1 API endpoint, the model falls back to CPU or CPU/GPU hybrid usage.

This results in extremely high CPU load (up to 100%) and poor performance.

Expected behavior:

GPU usage should be consistent across CLI and API usage
No silent fallback to CPU when GPU is available
Same model + same system should produce the same execution behavior

Hypothesis:

The issue may be related to a race condition during model initialization.

When using the OpenAI-compatible /v1 API, requests can arrive while the model is still loading into GPU memory. Instead of waiting for the model to finish initializing, Ollama appears to start a separate execution path that falls back to CPU.

This results in:

GPU initialization happening in parallel
Incoming API requests being handled prematurely
A second (CPU-based) execution being triggered

As a consequence, the system ends up with:

GPU load from model initialization
CPU load from premature inference handling

This suggests that the server does not properly block or queue incoming requests until the model is fully loaded and ready for GPU execution.

A possible fix would be:

Enforcing a strict “model ready” state before handling inference requests
Queuing or delaying incoming /v1 requests until GPU initialization is complete

Actual behavior:

CLI (ollama run) → 100% GPU
/v1 API → CPU or mixed CPU/GPU
CPU spikes to 100%
GPU usage drops significantly or is not used

Example:

ollama ps
qwen3.5:27b 25 GB 12%/88% CPU/GPU

or:

0% GPU / 100% CPU

System:

OS: Windows 11
GPU: (e.g. RTX 3090 24GB)
Ollama version: (run ollama --version)

Configuration:

Environment variables:

OLLAMA_NUM_THREADS=6
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_KEEP_ALIVE=-1

Client configuration:

"contextWindow": 131072
"maxTokens": 8192

Important observations:

Issue occurs only when using /v1 API
Native API (/api/generate) works correctly
CLI works correctly
GPU is functional and properly used outside /v1

Hypothesis:

The issue appears to be related to the OpenAI compatibility layer (/v1):

Different handling of context size or memory allocation
Early context allocation during model load
Possible fallback triggered before GPU execution stabilizes

Steps to reproduce:

Start server:
ollama serve
Run model via CLI:
ollama run qwen3.5:27b

→ GPU works correctly

Run via API:
POST /v1/responses

→ CPU usage spikes

Impact:

High CPU usage
Severe performance degradation
System instability under load
Inconsistent behavior depending on API path

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @LukaszRT on GitHub (Mar 22, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15016 ### What is the issue? Title: GPU works with ollama run, but falls back to CPU when using OpenAI /v1 API (same model) Description: When running a model manually using: ollama run qwen3.5:27b the model runs correctly on the GPU (100% GPU usage). However, when using the same model via the OpenAI-compatible /v1 API endpoint, the model falls back to CPU or CPU/GPU hybrid usage. This results in extremely high CPU load (up to 100%) and poor performance. Expected behavior: GPU usage should be consistent across CLI and API usage No silent fallback to CPU when GPU is available Same model + same system should produce the same execution behavior Hypothesis: The issue may be related to a race condition during model initialization. When using the OpenAI-compatible /v1 API, requests can arrive while the model is still loading into GPU memory. Instead of waiting for the model to finish initializing, Ollama appears to start a separate execution path that falls back to CPU. This results in: GPU initialization happening in parallel Incoming API requests being handled prematurely A second (CPU-based) execution being triggered As a consequence, the system ends up with: GPU load from model initialization CPU load from premature inference handling This suggests that the server does not properly block or queue incoming requests until the model is fully loaded and ready for GPU execution. A possible fix would be: Enforcing a strict “model ready” state before handling inference requests Queuing or delaying incoming /v1 requests until GPU initialization is complete Actual behavior: CLI (ollama run) → 100% GPU /v1 API → CPU or mixed CPU/GPU CPU spikes to 100% GPU usage drops significantly or is not used Example: ollama ps qwen3.5:27b 25 GB 12%/88% CPU/GPU or: 0% GPU / 100% CPU System: OS: Windows 11 GPU: (e.g. RTX 3090 24GB) Ollama version: (run ollama --version) Configuration: Environment variables: OLLAMA_NUM_THREADS=6 OLLAMA_NUM_PARALLEL=1 OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_KEEP_ALIVE=-1 Client configuration: "contextWindow": 131072 "maxTokens": 8192 Important observations: Issue occurs only when using /v1 API Native API (/api/generate) works correctly CLI works correctly GPU is functional and properly used outside /v1 Hypothesis: The issue appears to be related to the OpenAI compatibility layer (/v1): Different handling of context size or memory allocation Early context allocation during model load Possible fallback triggered before GPU execution stabilizes Steps to reproduce: Start server: ollama serve Run model via CLI: ollama run qwen3.5:27b → GPU works correctly Run via API: POST /v1/responses → CPU usage spikes Impact: High CPU usage Severe performance degradation System instability under load Inconsistent behavior depending on API path ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_

GiteaMirror added the bug label 2026-05-05 02:23:14 -05:00

GiteaMirror closed this issue

2026-05-05 02:23:15 -05:00

GiteaMirror commented

2026-05-05 02:23:17 -05:00

@rick-github commented on GitHub (Mar 22, 2026):

Server logs and the full output of ollama ps will aid in debugging.

OLLAMA_NUM_THREADS=6

OLLAMA_NUM_THREADS is not an ollama configuration variable.

@rick-github commented on GitHub (Mar 22, 2026): [Server logs](https://docs.ollama.com/troubleshooting) and the full output of `ollama ps` will aid in debugging. > OLLAMA_NUM_THREADS=6 `OLLAMA_NUM_THREADS` is not an ollama configuration variable.

GiteaMirror commented

2026-05-05 02:23:18 -05:00

@cdll commented on GitHub (Mar 23, 2026):

same issue here

@cdll commented on GitHub (Mar 23, 2026): same issue here

GiteaMirror commented

2026-05-05 02:23:18 -05:00

@rick-github commented on GitHub (Mar 23, 2026):

Server logs will aid in debugging.

@rick-github commented on GitHub (Mar 23, 2026): [Server logs](https://docs.ollama.com/troubleshooting) will aid in debugging.

GiteaMirror commented

2026-05-05 02:23:18 -05:00

@mwhite34 commented on GitHub (Mar 25, 2026):

I was running into this same issue. When I would run ollama serve and then use /v1/chat/completions first, the model loaded 100% onto the CPU.

However, if I used the /api/generate endpoint first, the model loaded correctly onto the GPU and subsequent /v1/chat/completions calls used the GPU as well.

For reference, I was using the Gemma3:27b model on a 12gb card, so only around 39/63 layers load to the GPU. The fact that the model needs to partially load on CPU could be relevant with the v1 startup issue.

@mwhite34 commented on GitHub (Mar 25, 2026): I was running into this same issue. When I would run `ollama serve` and then use `/v1/chat/completions` first, the model loaded 100% onto the CPU. However, if I used the `/api/generate` endpoint first, the model loaded correctly onto the GPU and subsequent `/v1/chat/completions` calls used the GPU as well. For reference, I was using the Gemma3:27b model on a 12gb card, so only around 39/63 layers load to the GPU. The fact that the model needs to partially load on CPU could be relevant with the v1 startup issue.

GiteaMirror commented

2026-05-05 02:23:19 -05:00

@CastelDazur commented on GitHub (Mar 26, 2026):

Saw the same thing with Qwen2.5-32B on a 5090. Ended up sending a dummy /api/generate request at startup to force GPU loading before anything hits /v1.

@CastelDazur commented on GitHub (Mar 26, 2026): Saw the same thing with Qwen2.5-32B on a 5090. Ended up sending a dummy /api/generate request at startup to force GPU loading before anything hits /v1.

GiteaMirror commented

2026-05-05 02:23:19 -05:00

@rick-github commented on GitHub (Mar 26, 2026):

Server logs will aid in debugging.

@rick-github commented on GitHub (Mar 26, 2026): [Server logs](https://docs.ollama.com/troubleshooting) will aid in debugging.

GiteaMirror commented

2026-05-05 02:23:21 -05:00

@LukaszRT commented on GitHub (Mar 26, 2026):

That's precisely the problem. There are no server logs because the server log is actually being handled by a second instance running in the background, which is using the CPU!!!

Therefore, it's a fatal mistake to assume that if the log is a certain state, then the state is a certainty! I've noticed the same bug in the log during processing. The log only continues to run once a process hangs and Ctrl+C is pressed. The server isn't processing data cleanly, which is why almost all problems occur. It's not a "Ollama isn't doing what it's supposed to" problem, but rather a "one process is hanging" problem, and a second process is running in the background because the first one isn't being properly terminated. The second process isn't visible, so it's running in fallback mode. This means the Ollama server doesn't have proper process handling and monitoring!

You can see it only in the screenshots.

Therefore, it's a fatal mistake to say "show me the log" because, in reality, the server log is stuck. The second instance won't be logged until the stuck process is terminated. And that's precisely the main problem with almost all Ollama bugs.
Just so we're clear, I love ollama and I'm not complaining. I just want to help. And the bugs are very simple. The GPU gets stuck, for example, in the runner or in a POST. What does the server do? It opens a new process in the background without handling the hang. What happens? The stuck process blocks the GPU, resulting in a fallback to the CPU, but no log is visible! Right, first check if processes are stuck. If necessary, reset them, then start a new instance.

I can trace almost every Ollama problem back to this. After this realization, Ollama runs in every CUDA version and every driver version. Usually, a process hangs or times out, which then blocks the hardware. It's an architectural problem.

The core issue is not a malfunction of Ollama itself, but improper process handling. When a runner process hangs (e.g., during GPU initialization or inference), it is not properly terminated. This leaves the GPU context locked. Subsequent requests then spawn new processes that cannot access the GPU and silently fall back to CPU execution. This creates the illusion of inconsistent or faulty behavior, while the root cause is a blocked resource due to a stale process.
Why logs are unreliable in this case:
Logs are tied to the active process context. If the primary process hangs, logging may also stall or buffer output. In many cases, logs are only flushed when the process is interrupted (e.g., via Ctrl+C). Meanwhile, secondary fallback processes may run without proper logging visibility. Therefore, relying on logs alone can be misleading, as they do not reflect the actual runtime state of the system when process deadlocks or resource locks occur.

@LukaszRT commented on GitHub (Mar 26, 2026): ![Image](https://github.com/user-attachments/assets/ba010d53-816a-43e3-8d07-f4b4aad8ce1c) That's precisely the problem. There are no server logs because the server log is actually being handled by a second instance running in the background, which is using the CPU!!! Therefore, it's a fatal mistake to assume that if the log is a certain state, then the state is a certainty! I've noticed the same bug in the log during processing. The log only continues to run once a process hangs and Ctrl+C is pressed. The server isn't processing data cleanly, which is why almost all problems occur. It's not a "Ollama isn't doing what it's supposed to" problem, but rather a "one process is hanging" problem, and a second process is running in the background because the first one isn't being properly terminated. The second process isn't visible, so it's running in fallback mode. This means the Ollama server doesn't have proper process handling and monitoring! You can see it only in the screenshots. Therefore, it's a fatal mistake to say "show me the log" because, in reality, the server log is stuck. The second instance won't be logged until the stuck process is terminated. And that's precisely the main problem with almost all Ollama bugs. Just so we're clear, I love ollama and I'm not complaining. I just want to help. And the bugs are very simple. The GPU gets stuck, for example, in the runner or in a POST. What does the server do? It opens a new process in the background without handling the hang. What happens? The stuck process blocks the GPU, resulting in a fallback to the CPU, but no log is visible! Right, first check if processes are stuck. If necessary, reset them, then start a new instance. I can trace almost every Ollama problem back to this. After this realization, Ollama runs in every CUDA version and every driver version. Usually, a process hangs or times out, which then blocks the hardware. It's an architectural problem. The core issue is not a malfunction of Ollama itself, but improper process handling. When a runner process hangs (e.g., during GPU initialization or inference), it is not properly terminated. This leaves the GPU context locked. Subsequent requests then spawn new processes that cannot access the GPU and silently fall back to CPU execution. This creates the illusion of inconsistent or faulty behavior, while the root cause is a blocked resource due to a stale process. Why logs are unreliable in this case: Logs are tied to the active process context. If the primary process hangs, logging may also stall or buffer output. In many cases, logs are only flushed when the process is interrupted (e.g., via Ctrl+C). Meanwhile, secondary fallback processes may run without proper logging visibility. Therefore, relying on logs alone can be misleading, as they do not reflect the actual runtime state of the system when process deadlocks or resource locks occur.

GiteaMirror commented

2026-05-05 02:23:21 -05:00

@rick-github commented on GitHub (Mar 26, 2026):

If there's a problem with GPU discovery, the logs leading up to the problem point will help in debggung. Set OLLAMA_DEBUG=2 to increase the amount of information recorded.

@rick-github commented on GitHub (Mar 26, 2026): If there's a problem with GPU discovery, the logs leading up to the problem point will help in debggung. Set `OLLAMA_DEBUG=2` to increase the amount of information recorded.

GiteaMirror commented

2026-05-05 02:23:22 -05:00

@LukaszRT commented on GitHub (Mar 26, 2026):

I understand your point about increasing log verbosity with OLLAMA_DEBUG=2, and that can certainly help in normal debugging scenarios.

However, the issue I'm describing is different:

When a runner process hangs (e.g., during GPU init or inference), it can block the GPU context without being properly terminated. In this state, the logging pipeline itself may also stall or stop flushing. As a result, the logs leading up to the issue are incomplete or misleading.

Additionally, when a new request is made, a second process may start and fall back to CPU because the GPU is still locked by the hung process. This secondary process may not produce meaningful logs related to the original failure, which makes it appear as if everything is functioning normally.

So the core problem is not insufficient logging detail, but that:

the original process is stuck and not releasing resources
logging may be tied to that stuck process
subsequent fallback behavior is not clearly correlated in the logs

In this scenario, simply increasing log verbosity does not fully capture the actual system state.

(1) Request A
↓
[ Runner #1 ] ────────┐
↓ │
[ GPU ] ←────────────┘ ❌ HANG (blocks GPU + Logging)

      (no Logs from here visible / Logs hanging)

(2) Request B // from hier everthing is invisible
↓
[ Ollama Server ]
↓
[ Runner #2 ]
↓
[ CPU Fallback ] ❗ (GPU blocked)
↓
[ Logs von Runner #2 ]

@LukaszRT commented on GitHub (Mar 26, 2026): I understand your point about increasing log verbosity with OLLAMA_DEBUG=2, and that can certainly help in normal debugging scenarios. However, the issue I'm describing is different: When a runner process hangs (e.g., during GPU init or inference), it can block the GPU context without being properly terminated. In this state, the logging pipeline itself may also stall or stop flushing. As a result, the logs leading up to the issue are incomplete or misleading. Additionally, when a new request is made, a second process may start and fall back to CPU because the GPU is still locked by the hung process. This secondary process may not produce meaningful logs related to the original failure, which makes it appear as if everything is functioning normally. So the core problem is not insufficient logging detail, but that: the original process is stuck and not releasing resources logging may be tied to that stuck process subsequent fallback behavior is not clearly correlated in the logs In this scenario, simply increasing log verbosity does not fully capture the actual system state. (1) Request A ↓ [ Runner #1 ] ────────┐ ↓ │ [ GPU ] ←────────────┘ ❌ HANG (blocks GPU + Logging) (no Logs from here visible / Logs hanging) (2) Request B // from hier everthing is invisible ↓ [ Ollama Server ] ↓ [ Runner #2 ] ↓ [ CPU Fallback ] ❗ (GPU blocked) ↓ [ Logs von Runner #2 ]

GiteaMirror commented

2026-05-05 02:23:23 -05:00

@rick-github commented on GitHub (Mar 26, 2026):

So far all I see is interpretation of the logs with out any logs. No advance can be made without further information. It may not be enough, but it will provide a base level from which the problem can be explored in more depth. No logs, no debugging.

@rick-github commented on GitHub (Mar 26, 2026): So far all I see is interpretation of the logs with out any logs. No advance can be made without further information. It may not be enough, but it will provide a base level from which the problem can be explored in more depth. No logs, no debugging.

GiteaMirror commented

2026-05-05 02:23:25 -05:00

@LukaszRT commented on GitHub (Mar 26, 2026):

You can only see the logs up to the point where the process hangs.

Sometimes the blocked process can be unblocked using CTRL+C and then the fallback is immediately gone, but the log gets lost.

In principle, Ollama is the best software available for this purpose. The problem lies simply in the fact that the blocked processes are not being processed. There is no process in place to handle them. This gives the impression that Ollama isn't doing what it's configured to do. But it's doing exactly what it's supposed to do. It's actually perfect. However, the blocked processes cause fallbacks that are practically invisible.

The reasons why processes freeze vary greatly; often it's due to a timeout that's too short, or a stuck driver that doesn't load in time. But I've found that if you unblock it with Ctrl+C (this often works, but not always), then Ollam runs like the best software in the world and does exactly what it's supposed to. So it's always freezes that are to blame, blocking the hardware. 500-series runners, etc....

However, the process that needs to be unblocked before a new process starts is crucial because a hung process locks the hardware. Therefore, a watchdog and an rtry routine for this are practically non-existent.

The fix would have to be extremely simple. A prolonged hang should trigger the termination of the process, thus eliminating any fallback. only shutdown lockings.

@LukaszRT commented on GitHub (Mar 26, 2026): You can only see the logs up to the point where the process hangs. Sometimes the blocked process can be unblocked using CTRL+C and then the fallback is immediately gone, but the log gets lost. In principle, Ollama is the best software available for this purpose. The problem lies simply in the fact that the blocked processes are not being processed. There is no process in place to handle them. This gives the impression that Ollama isn't doing what it's configured to do. But it's doing exactly what it's supposed to do. It's actually perfect. However, the blocked processes cause fallbacks that are practically invisible. The reasons why processes freeze vary greatly; often it's due to a timeout that's too short, or a stuck driver that doesn't load in time. But I've found that if you unblock it with Ctrl+C (this often works, but not always), then Ollam runs like the best software in the world and does exactly what it's supposed to. So it's always freezes that are to blame, blocking the hardware. 500-series runners, etc.... However, the process that needs to be unblocked before a new process starts is crucial because a hung process locks the hardware. Therefore, a watchdog and an rtry routine for this are practically non-existent. The fix would have to be extremely simple. A prolonged hang should trigger the termination of the process, thus eliminating any fallback. only shutdown lockings.

GiteaMirror commented

2026-05-05 02:23:26 -05:00

@rick-github commented on GitHub (Mar 26, 2026):

A prolonged hang should trigger the termination of the process

It does. Further responses require logs.

@rick-github commented on GitHub (Mar 26, 2026): > A prolonged hang should trigger the termination of the process It does. Further responses require logs.

GiteaMirror commented

2026-05-05 02:23:27 -05:00

@LukaszRT commented on GitHub (Mar 26, 2026):

Here you can clearly see the loader is finished. There's no fallback to the CPU in this example because the GPU wasn't locked. But you can see the CUDA running, then the log stops. There's a backdoor process.

The runner is indeed stuck and will shortly perform the fallback to the CPU next time.

After pressing Ctrl+C, you can see that something was indeed running in the background, but CUDA was dead.

After this phase, pressing Ctrl+C again makes ollama run cleanly in CUDA; the locked process is unblocked.

In the last video, you can see how ollama is stuck due to the 500 error. Ctrl+C unlocked it, but there's no CUDA, and the server is shut down. Model exit. That's what I mean. The problem can usually be solved with Ctrl+C, but not always. It's a problem with stuck processes.

Why does this happen? Because hung processes aren't terminated, and new processes are allowed to start simultaneously. This often leads to a fallback. A simple fix would be to terminate every process; it wouldn't matter if the runner is complete or not, or if it's a 200, 500, or any other type. The server would open a new instance anyway when a process is terminated. But if this isn't done, the hardware can become locked, which will lead to a fallback!

Because under Windows, the hardware in a hung process simply blocks any response. Therefore, the correct approach would be to terminate any process that hangs for a while. This would be a self-healing mode.

Self-healing occurs because with each new request to the server, the runner or a new request would rebuild itself anyway, but no hardware would be blocked.

Therefore, a simple fix would make Ollama very robust and release hung hardware caused by blocked processes. A watchdog that checks after a while whether processes are hanging. Or a trigger on Stdout.

Ultimately, any single process can hang and therefore block the hardware. That's why every single process must be terminated cleanly, at least after a certain period of time. Then the system heals itself. That's the good thing about it. While a few fail requests are possible, the server doesn't fall back and runs smoothly and reliably.

Why did I come to this conclusion? Often the server would run for hours without problems, then suddenly everything would stop working. Not even after restarting the server. The same log would always appear: CPU at 99%, server crashing. Why wouldn't software that's been running all along work? Why can't it simply be restarted? CUDA is fine, it runs stably. Drivers are fine too, it works with any cheap Nvidia driver. The problem is a stalled hardware caused by a stuck process. Even restarting the Ollama server can cause problems; sometimes even a system restart doesn't help because this process can freeze in Windows.

Not so much in the sense that "the process is still alive," but rather that a hang leaves behind states in the driver, CUDA stack, or device reset that a normal restart of Ollama or even a Windows restart doesn't always clean up.

Typical levels:

User process hangs → easily killable
CUDA/Runner hangs in the driver path → more difficult to kill
GPU/Driver state is stuck → can survive a restart
Full hardware reset required → sometimes only after a complete power cycle

That's why your idea of killing hanged processes early is so important:

By ending the hang before it deepens into the hardware/driver state, you prevent precisely these "poisoned states."

Therefore, this problem shouldn't be ignored.

I've found that ollama, if you resolve the stuck processes, functions like a clockwork mechanism, indeed like a masterpiece. The problem lies solely in the architecture.

It's also important to consider that ollama should never be using 100% of the CPU, as this would exacerbate the fallback situation. This could cause further processes to crash, leading to serious errors beyond simply restarting the system, as described above.

Fix Watachdog end kills
Fix Ollama never user 100% cpu

Thanks!

@LukaszRT commented on GitHub (Mar 26, 2026): <img width="1670" height="999" alt="Image" src="https://github.com/user-attachments/assets/a88db1e4-2484-499c-aaa4-e96c94322902" /> Here you can clearly see the loader is finished. There's no fallback to the CPU in this example because the GPU wasn't locked. But you can see the CUDA running, then the log stops. There's a backdoor process. The runner is indeed stuck and will shortly perform the fallback to the CPU next time. After pressing Ctrl+C, you can see that something was indeed running in the background, but CUDA was dead. After this phase, pressing Ctrl+C again makes ollama run cleanly in CUDA; the locked process is unblocked. <img width="1670" height="999" alt="Image" src="https://github.com/user-attachments/assets/11eef455-7b3b-46d6-83f6-b9c793eedb09" /> In the last video, you can see how ollama is stuck due to the 500 error. Ctrl+C unlocked it, but there's no CUDA, and the server is shut down. Model exit. That's what I mean. The problem can usually be solved with Ctrl+C, but not always. It's a problem with stuck processes. <img width="1608" height="756" alt="Image" src="https://github.com/user-attachments/assets/c3d622cf-e7da-41f3-ab89-9573193c7811" /> Why does this happen? Because hung processes aren't terminated, and new processes are allowed to start simultaneously. This often leads to a fallback. A simple fix would be to terminate every process; it wouldn't matter if the runner is complete or not, or if it's a 200, 500, or any other type. The server would open a new instance anyway when a process is terminated. But if this isn't done, the hardware can become locked, which will lead to a fallback! Because under Windows, the hardware in a hung process simply blocks any response. Therefore, the correct approach would be to terminate any process that hangs for a while. This would be a self-healing mode. Self-healing occurs because with each new request to the server, the runner or a new request would rebuild itself anyway, but no hardware would be blocked. Therefore, a simple fix would make Ollama very robust and release hung hardware caused by blocked processes. A watchdog that checks after a while whether processes are hanging. Or a trigger on Stdout. Ultimately, any single process can hang and therefore block the hardware. That's why every single process must be terminated cleanly, at least after a certain period of time. Then the system heals itself. That's the good thing about it. While a few fail requests are possible, the server doesn't fall back and runs smoothly and reliably. Why did I come to this conclusion? Often the server would run for hours without problems, then suddenly everything would stop working. Not even after restarting the server. The same log would always appear: CPU at 99%, server crashing. Why wouldn't software that's been running all along work? Why can't it simply be restarted? CUDA is fine, it runs stably. Drivers are fine too, it works with any cheap Nvidia driver. The problem is a stalled hardware caused by a stuck process. Even restarting the Ollama server can cause problems; sometimes even a system restart doesn't help because this process can freeze in Windows. Not so much in the sense that "the process is still alive," but rather that a hang leaves behind states in the driver, CUDA stack, or device reset that a normal restart of Ollama or even a Windows restart doesn't always clean up. Typical levels: User process hangs → easily killable CUDA/Runner hangs in the driver path → more difficult to kill GPU/Driver state is stuck → can survive a restart Full hardware reset required → sometimes only after a complete power cycle That's why your idea of killing hanged processes early is so important: By ending the hang before it deepens into the hardware/driver state, you prevent precisely these "poisoned states." Therefore, this problem shouldn't be ignored. I've found that ollama, if you resolve the stuck processes, functions like a clockwork mechanism, indeed like a masterpiece. The problem lies solely in the architecture. It's also important to consider that ollama should never be using 100% of the CPU, as this would exacerbate the fallback situation. This could cause further processes to crash, leading to serious errors beyond simply restarting the system, as described above. Fix Watachdog end kills Fix Ollama never user 100% cpu Thanks!

GiteaMirror commented

2026-05-05 02:23:28 -05:00

@rick-github commented on GitHub (Mar 26, 2026):

the log stops. There's a backdoor process.

The log stops because the runner has started and is presumably processing the prompt that caused the model to load. The graph shows GPU activity. If you had set OLLAMA_DEBUG=2 we might even be able to see what was happening. What client are you using?

That's why your idea of killing hanged processes early is so important:

Please do not use an LLM to diagnose issues.

Therefore, this problem shouldn't be ignored.

It's not but you seem to be actively obstructing my attempts at debugging this. Provide a full, unabridged log with OLLAMA_DEBUG=2 and in plain text.

@rick-github commented on GitHub (Mar 26, 2026): > the log stops. There's a backdoor process. The log stops because the runner has started and is presumably processing the prompt that caused the model to load. The graph shows GPU activity. If you had set `OLLAMA_DEBUG=2` we might even be able to see what was happening. What client are you using? > That's why your idea of killing hanged processes early is so important: Please do not use an LLM to diagnose issues. > Therefore, this problem shouldn't be ignored. It's not but you seem to be actively obstructing my attempts at debugging this. Provide a full, unabridged log with `OLLAMA_DEBUG=2` and in plain text.

GiteaMirror commented

2026-05-05 02:23:28 -05:00

@dasnolojy commented on GitHub (Apr 8, 2026):

I ran into a similar problem, except that mine occurred when using any version, not just v1. In the app’s chat, all inference was performed on the GPU, but when using the API, I got inference on the CPU.

The problem was that, following some instructions I found online, I had added the custom environment variable CUDA_VISIBLE_DEVICES=0 and completely forgot about it. The issue only appeared after a reboot, and I didn’t understand what had happened—after all, everything had been working fine just a short while ago.

The solution turned out to be simple: I needed to remove the CUDA_VISIBLE_DEVICES environment variable and, most importantly, reinstall Ollama, since a simple reboot didn’t help—apparently due to some kind of caching.

@dasnolojy commented on GitHub (Apr 8, 2026): I ran into a similar problem, except that mine occurred when using any version, not just v1. In the app’s chat, all inference was performed on the GPU, but when using the API, I got inference on the CPU. The problem was that, following some instructions I found online, I had added the custom environment variable `CUDA_VISIBLE_DEVICES=0` and completely forgot about it. The issue only appeared after a reboot, and I didn’t understand what had happened—after all, everything had been working fine just a short while ago. The solution turned out to be simple: I needed to remove the CUDA_VISIBLE_DEVICES environment variable and, most importantly, reinstall Ollama, since a simple reboot didn’t help—apparently due to some kind of caching.

Sign in to join this conversation.

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#71707