github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #10752] Strange processing after update to 0.7 #7063

New Issue
Closed
opened 2026-04-12 18:58:47 -05:00 by GiteaMirror · 26 comments
GiteaMirror commented 2026-04-12 18:58:47 -05:00
Owner
Copy Link

Originally created by @abes200 on GitHub (May 17, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10752

What is the issue?

So far I have noticed this only with Gemma3, but it's possibly affecting other models.
Before the update everything worked fine. Well, sort of, it was severely underestimating how much VRAM to use so I had to include a manual setting for the num_gpu. But when I sent a message, the GPU usage would spike a little while the CPU usage was fairly low while it processed the system message and conversation, then the GPU usage would drop and the CPU would go to 50% while the AI responded.
Now, soon as I send a message the GPU does nothing, the CPU goes straight to 50% and the AI takes FOREVER to start to generate a response if there is almost ANYTHING in the context window or it has a system message. I am still manually setting the num_gpu and it is using up available VRAM as it should. But it does not seem to be using the GPU for any inference at all anymore.
Once it starts responding it responds at the same speed it was before. But the initial time to wait for a response has increased dramatically.

Also, as I have mentioned before. I will not upload a whole log file as it still contains personal information. Such as my username name for my PC. If certain parts of a log are wanted, I will paste those in after I have gone through them thoroughly to remove identifying details I don't want to share publicly.

Relevant log output

OS

Windows 10

GPU

GTX 1660 6gb

CPU

AMD FX 8350

Ollama version

0.7

Originally created by @abes200 on GitHub (May 17, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10752 ### What is the issue? So far I have noticed this only with Gemma3, but it's possibly affecting other models. Before the update everything worked fine. Well, sort of, it was severely underestimating how much VRAM to use so I had to include a manual setting for the num_gpu. But when I sent a message, the GPU usage would spike a little while the CPU usage was fairly low while it processed the system message and conversation, then the GPU usage would drop and the CPU would go to 50% while the AI responded. Now, soon as I send a message the GPU does nothing, the CPU goes straight to 50% and the AI takes FOREVER to start to generate a response if there is almost ANYTHING in the context window or it has a system message. I am still manually setting the num_gpu and it is using up available VRAM as it should. But it does not seem to be using the GPU for any inference at all anymore. Once it starts responding it responds at the same speed it was before. But the initial time to wait for a response has increased dramatically. Also, as I have mentioned before. I will not upload a whole log file as it still contains personal information. Such as my username name for my PC. If certain parts of a log are wanted, I will paste those in after I have gone through them thoroughly to remove identifying details I don't want to share publicly. ### Relevant log output ```shell ``` ### OS Windows 10 ### GPU GTX 1660 6gb ### CPU AMD FX 8350 ### Ollama version 0.7
GiteaMirror added the bug label 2026-04-12 18:58:47 -05:00
GiteaMirror commented 2026-04-12 18:58:49 -05:00
Author
Owner
Copy Link

@Lantrancy commented on GitHub (May 17, 2025):

same here, using gemma3, first question is running fine, then I ask another, the CPU goes full throttle for a while, then it gives output but GPU load is very low, seems not using GPU at all. It used to be very fast when asking second question cause the model is loaded in RAM already

<!-- gh-comment-id:2888026723 --> @Lantrancy commented on GitHub (May 17, 2025): same here, using gemma3, first question is running fine, then I ask another, the CPU goes full throttle for a while, then it gives output but GPU load is very low, seems not using GPU at all. It used to be very fast when asking second question cause the model is loaded in RAM already
GiteaMirror commented 2026-04-12 18:58:49 -05:00
Author
Owner
Copy Link

@leegimblett commented on GitHub (May 17, 2025):

Same here. GPUs are being ignored with Gemma 3 and QWen3 (the only models I have). GPUs have zero memory and cpu use. Worked fine before upgrade from 0.6.8
OS - Ubuntu 24.04
GPUS - 4060ti and 5070ti

<!-- gh-comment-id:2888037996 --> @leegimblett commented on GitHub (May 17, 2025): Same here. GPUs are being ignored with Gemma 3 and QWen3 (the only models I have). GPUs have zero memory and cpu use. Worked fine before upgrade from 0.6.8 OS - Ubuntu 24.04 GPUS - 4060ti and 5070ti
GiteaMirror commented 2026-04-12 18:58:50 -05:00
Author
Owner
Copy Link

@abes200 commented on GitHub (May 17, 2025):

Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0
Qwen3:
prompt eval rate: 58.29 tokens/s

Gemma3:
prompt eval rate: 4.67 tokens/s

As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.)

<!-- gh-comment-id:2888120615 --> @abes200 commented on GitHub (May 17, 2025): Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0 Qwen3: prompt eval rate: 58.29 tokens/s Gemma3: prompt eval rate: 4.67 tokens/s As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.)
GiteaMirror commented 2026-04-12 18:58:51 -05:00
Author
Owner
Copy Link

@Lantrancy commented on GitHub (May 17, 2025):

Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0 Qwen3: prompt eval rate: 58.29 tokens/s

Gemma3: prompt eval rate: 4.67 tokens/s

As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.)

Hi,
may I ask your config for ollama?
I have a 3900X_64GB+3060ti system, I run Gemma3:12b on ollama 0.6.8 I only get around 8 tps performance

<!-- gh-comment-id:2888229913 --> @Lantrancy commented on GitHub (May 17, 2025): > Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0 Qwen3: prompt eval rate: 58.29 tokens/s > > Gemma3: prompt eval rate: 4.67 tokens/s > > As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.) Hi, may I ask your config for ollama? I have a 3900X_64GB+3060ti system, I run Gemma3:12b on ollama 0.6.8 I only get around 8 tps performance
GiteaMirror commented 2026-04-12 18:58:52 -05:00
Author
Owner
Copy Link

@leegimblett commented on GitHub (May 17, 2025):

Same here. GPUs are being ignored with Gemma 3 and QWen3 (the only models I have). GPUs have zero memory and cpu use. Worked fine before upgrade from 0.6.8 OS - Ubuntu 24.04 GPUS - 4060ti and 5070ti

I reinstalled 0.6.8 and all is functioning normally

Some more information from my experience

Install is a standard OS level using the install.sh script (not docker).

My launch paramters (systemd) are:
[Service] Environment="OLLAMA_ORIGINS=moz-extension://*" Environment="OLLAMA_MODELS=/mnt/Storage/ollama/models" Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Using Ollama 0.6.8

Ollama ps:
NAME ID SIZE PROCESSOR UNTIL
gemma3:27b-it-qat 29eb0b9aeda3 24 GB 100% GPU 4 minutes from now

Image

Using Ollama 0.7.0

Ollama ps:
NAME ID SIZE PROCESSOR UNTIL
gemma3:27b-it-qat 29eb0b9aeda3 29 GB 10%/90% CPU/GPU 4 minutes from now

Oddly the cpu use looks the wrong way around - I am actually seeing low GPU and high CPU Use

Image

<!-- gh-comment-id:2888268256 --> @leegimblett commented on GitHub (May 17, 2025): > Same here. GPUs are being ignored with Gemma 3 and QWen3 (the only models I have). GPUs have zero memory and cpu use. Worked fine before upgrade from 0.6.8 OS - Ubuntu 24.04 GPUS - 4060ti and 5070ti I reinstalled 0.6.8 and all is functioning normally Some more information from my experience Install is a standard OS level using the install.sh script (not docker). My launch paramters (systemd) are: `[Service] Environment="OLLAMA_ORIGINS=moz-extension://*" Environment="OLLAMA_MODELS=/mnt/Storage/ollama/models" Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0"` ### Using Ollama 0.6.8 Ollama ps: NAME ID SIZE PROCESSOR UNTIL gemma3:27b-it-qat 29eb0b9aeda3 24 GB 100% GPU 4 minutes from now ![Image](https://github.com/user-attachments/assets/d69e137a-eb81-4579-b065-33f860cade06) ### Using Ollama 0.7.0 Ollama ps: NAME ID SIZE PROCESSOR UNTIL gemma3:27b-it-qat 29eb0b9aeda3 29 GB 10%/90% CPU/GPU 4 minutes from now Oddly the cpu use looks the wrong way around - I am actually seeing low GPU and high CPU Use ![Image](https://github.com/user-attachments/assets/a630efa7-d422-4ba1-87c3-0f198634363d)
GiteaMirror commented 2026-04-12 18:58:52 -05:00
Author
Owner
Copy Link

@ACheshirov commented on GitHub (May 17, 2025):

Same here... for some reason there is no ollama process in nvidia-smi and everything is handled by CPU/RAM...

<!-- gh-comment-id:2888471925 --> @ACheshirov commented on GitHub (May 17, 2025): Same here... for some reason there is no ollama process in nvidia-smi and everything is handled by CPU/RAM...
GiteaMirror commented 2026-04-12 18:58:53 -05:00
Author
Owner
Copy Link

@marcinm1234 commented on GitHub (May 17, 2025):

Thaks for this topic!! Super confusing!!! Spent 30 mins on workaround... Models explode in size with this version :(((( gemma3:27b-it-qat should be 18GB, but is 25GB!! does not fit anymore on 24 GB VRAM 3090. Please fix ASAP... :)

EDIT: this should be 21 GB, but is:

ollama ps
NAME ID SIZE PROCESSOR UNTIL
qwen2.5vl:32b-q4_K_M 3edc3a52fe98 27 GB 12%/88% CPU/GPU 58 minutes from now

<!-- gh-comment-id:2888538696 --> @marcinm1234 commented on GitHub (May 17, 2025): Thaks for this topic!! Super confusing!!! Spent 30 mins on workaround... **Models explode in size with this version :((((** gemma3:27b-it-qat should be 18GB, but is 25GB!! does not fit anymore on 24 GB VRAM 3090. Please fix ASAP... :) EDIT: this should be 21 GB, but is: > ollama ps NAME ID SIZE PROCESSOR UNTIL qwen2.5vl:32b-q4_K_M 3edc3a52fe98 27 GB 12%/88% CPU/GPU 58 minutes from now
GiteaMirror commented 2026-04-12 18:58:53 -05:00
Author
Owner
Copy Link

@abes200 commented on GitHub (May 17, 2025):

Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0 Qwen3: prompt eval rate: 58.29 tokens/s
Gemma3: prompt eval rate: 4.67 tokens/s
As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.)

Hi, may I ask your config for ollama? I have a 3900X_64GB+3060ti system, I run Gemma3:12b on ollama 0.6.8 I only get around 8 tps performance

Those are the evaluation rates of the prompt, not response speeds. I get the feeling your referring to the actual response tokens/s?
Running Gemma3 12B Q4_K_M, if I just let it load itself, it loads 7/49 layers into GPU. I set it to 30/49. The actual response speed does start to decrease as the context grows.
From Gemma3 I can get up to 3.11 tps. But in reality I get more like 1.7 tps after we have been talking for more than a few messages and it's got a complex system prompt as well.
Qwen3 14b Q4_K_M I can get up to 2.17tps, but that will drop to at most 1 tps with a complex system prompt and several messages.

I haven't got any sort of special config for Ollama, only real difference is that I am assigning num_gpu values as the memory calculations seem to be off when Ollama tries to do it automatically. That applies to a few models as well, calculating how much to offload being wrong. I can usually up it a bit more, but Gemma3 is an exception, without manually setting it to load 30 layers into the GPU it will only load 7. I don't normally see that much of a difference, usually I can only add a few extra layers to GPU. Doing this does dramatically improve it's performance for me.
But it does not fix the extremely slow evaluation speeds on 0.7, just to be clear. I was doing this on earlier versions as well.

If that still seems unusually high for the system I am running, cool. 😄 But afraid I can't explain it anymore than saying I built this PC myself from specially chosen parts. The 1660 super GPU and FX 8350 CPU are old now, but they were beasts in their day.
Also, I am only running 16gb DDR3 memory. So both of those models are maxing out almost all my memory with an 8k context window. 2k there is less being loaded into RAM so they run better, but are crap for conversational context with such a short memory.

<!-- gh-comment-id:2888577471 --> @abes200 commented on GitHub (May 17, 2025): > > Glad to hear it's not just me. Although I don't have the same problem with Qwen3. For comparison, here are the prompt eval rates for Qwen3 14b and Gemma3 12b. Qwen3 is fairly normal for me with my low power hardware. When I was running 0.6.8 Gemma3 was running much faster than Qwen3, but it's not anymore on 0.7.0 Qwen3: prompt eval rate: 58.29 tokens/s > > Gemma3: prompt eval rate: 4.67 tokens/s > > As a note, both have a complex system message, running at 8192 context, both assigned a num_gpu (18 for Qwen, 30 for Gemma, thats the max I can handle for both.) > > Hi, may I ask your config for ollama? I have a 3900X_64GB+3060ti system, I run Gemma3:12b on ollama 0.6.8 I only get around 8 tps performance Those are the evaluation rates of the prompt, not response speeds. I get the feeling your referring to the actual response tokens/s? Running Gemma3 12B Q4_K_M, if I just let it load itself, it loads 7/49 layers into GPU. I set it to 30/49. The actual response speed does start to decrease as the context grows. From Gemma3 I can get up to 3.11 tps. But in reality I get more like 1.7 tps after we have been talking for more than a few messages and it's got a complex system prompt as well. Qwen3 14b Q4_K_M I can get up to 2.17tps, but that will drop to at most 1 tps with a complex system prompt and several messages. I haven't got any sort of special config for Ollama, only real difference is that I am assigning num_gpu values as the memory calculations seem to be off when Ollama tries to do it automatically. That applies to a few models as well, calculating how much to offload being wrong. I can usually up it a bit more, but Gemma3 is an exception, without manually setting it to load 30 layers into the GPU it will only load 7. I don't normally see that much of a difference, usually I can only add a few extra layers to GPU. Doing this does dramatically improve it's performance for me. But it does not fix the extremely slow evaluation speeds on 0.7, just to be clear. I was doing this on earlier versions as well. If that still seems unusually high for the system I am running, cool. 😄 But afraid I can't explain it anymore than saying I built this PC myself from specially chosen parts. The 1660 super GPU and FX 8350 CPU are old now, but they were beasts in their day. Also, I am only running 16gb DDR3 memory. So both of those models are maxing out almost all my memory with an 8k context window. 2k there is less being loaded into RAM so they run better, but are crap for conversational context with such a short memory.
GiteaMirror commented 2026-04-12 18:58:54 -05:00
Author
Owner
Copy Link

@abes200 commented on GitHub (May 17, 2025):

Here's what I should have done. Full verbose output from both models, 8k context on both.

Qwen3 14B Q4_K_M:
total duration: 2m50.0465051s
load duration: 58.1598ms
prompt eval count: 1486 token(s)
prompt eval duration: 25.1483296s
prompt eval rate: 59.09 tokens/s <-- Fine.
eval count: 215 token(s)
eval duration: 2m24.8247451s
eval rate: 1.48 tokens/s <-- response tps

Gemma3 12B Q4_K_M:
total duration: 7m59.3012883s
load duration: 136.2488ms
prompt eval count: 1737 token(s)
prompt eval duration: 6m8.8937726s
prompt eval rate: 4.71 tokens/s <-- Bad.
eval count: 239 token(s)
eval duration: 1m50.2122408s
eval rate: 2.17 tokens/s <-- response tps

<!-- gh-comment-id:2888601865 --> @abes200 commented on GitHub (May 17, 2025): Here's what I should have done. Full verbose output from both models, 8k context on both. Qwen3 14B Q4_K_M: total duration: 2m50.0465051s load duration: 58.1598ms prompt eval count: 1486 token(s) prompt eval duration: 25.1483296s prompt eval rate: 59.09 tokens/s <-- Fine. eval count: 215 token(s) eval duration: 2m24.8247451s eval rate: 1.48 tokens/s <-- response tps Gemma3 12B Q4_K_M: total duration: 7m59.3012883s load duration: 136.2488ms prompt eval count: 1737 token(s) prompt eval duration: 6m8.8937726s prompt eval rate: 4.71 tokens/s <-- Bad. eval count: 239 token(s) eval duration: 1m50.2122408s eval rate: 2.17 tokens/s <-- response tps
GiteaMirror commented 2026-04-12 18:58:54 -05:00
Author
Owner
Copy Link

@eliciel0513 commented on GitHub (May 17, 2025):

Experiencing the same issues with Gemma 3 14B, where all the model is being processed by CPU and load to RAM, not to mention image processing takes like 30minutes, used to take like 1-2 minutes before. Also Qwen 3 works fine sometimes, other times stays stuck in thinking. This Version (0.7.0) is very unstable. i9 14900K, RTX 3090.

<!-- gh-comment-id:2888604046 --> @eliciel0513 commented on GitHub (May 17, 2025): Experiencing the same issues with Gemma 3 14B, where all the model is being processed by CPU and load to RAM, not to mention image processing takes like 30minutes, used to take like 1-2 minutes before. Also Qwen 3 works fine sometimes, other times stays stuck in thinking. This Version (0.7.0) is very unstable. i9 14900K, RTX 3090.
GiteaMirror commented 2026-04-12 18:58:55 -05:00
Author
Owner
Copy Link

@Lantrancy commented on GitHub (May 18, 2025):

Here's what I should have done. Full verbose output from both models, 8k context on both.

Qwen3 14B Q4_K_M: total duration: 2m50.0465051s load duration: 58.1598ms prompt eval count: 1486 token(s) prompt eval duration: 25.1483296s prompt eval rate: 59.09 tokens/s <-- Fine. eval count: 215 token(s) eval duration: 2m24.8247451s eval rate: 1.48 tokens/s <-- response tps

Gemma3 12B Q4_K_M: total duration: 7m59.3012883s load duration: 136.2488ms prompt eval count: 1737 token(s) prompt eval duration: 6m8.8937726s prompt eval rate: 4.71 tokens/s <-- Bad. eval count: 239 token(s) eval duration: 1m50.2122408s eval rate: 2.17 tokens/s <-- response tps

I have got something similar, now I can confirm it's a problem related to ollama, I tried lm studio with Gemma3:12b-it-qat model, if I set GPU layer to 48, then it can fully utilize GPU giving me 28 tokens/s

<!-- gh-comment-id:2888659378 --> @Lantrancy commented on GitHub (May 18, 2025): > Here's what I should have done. Full verbose output from both models, 8k context on both. > > Qwen3 14B Q4_K_M: total duration: 2m50.0465051s load duration: 58.1598ms prompt eval count: 1486 token(s) prompt eval duration: 25.1483296s prompt eval rate: 59.09 tokens/s <-- Fine. eval count: 215 token(s) eval duration: 2m24.8247451s eval rate: 1.48 tokens/s <-- response tps > > Gemma3 12B Q4_K_M: total duration: 7m59.3012883s load duration: 136.2488ms prompt eval count: 1737 token(s) prompt eval duration: 6m8.8937726s prompt eval rate: 4.71 tokens/s <-- Bad. eval count: 239 token(s) eval duration: 1m50.2122408s eval rate: 2.17 tokens/s <-- response tps I have got something similar, now I can confirm it's a problem related to ollama, I tried lm studio with Gemma3:12b-it-qat model, if I set GPU layer to 48, then it can fully utilize GPU giving me 28 tokens/s
GiteaMirror commented 2026-04-12 18:58:56 -05:00
Author
Owner
Copy Link

@Ami-OS commented on GitHub (May 18, 2025):

I have the same problem which upgrade to 0.7

Env

CPU: i7-13700K
GPU: RTX 4080 Super
RAM: 64 GB DDR5
Ollama: 0.7
Model: Gemma3:27B

ollama ps:

  • Size: 24GB
  • Processor: 35%/65% CPU/GPU

Problem

When I asked the second question, the CPU went through a long "pre-processing"? And the CPU usage was only 50% but the temperature reached 95-100 degrees. After CPU stress testing, I determined that my CPU would only soar to nearly 100 degrees when using the FPU (Floating Point Unit).

Based on my observations:

  1. It was very fast when answering the first question, the GPU started working almost immediately, after answering the question the GPU stopped working, the CPU usage rose to 50% and the temperature rose to 95 degrees (obviously the FPU unit started working), when the CPU usage returned to normal, the conversation title was summarized.
  2. Before answering the second question, the CPU usage rate rose to 50% and the temperature rose to 95 degrees. The GPU started to work only after the CPU usage rate returned to normal. The GPU usage rate rose to 40%, and the GPU stopped working after the answer was completed.

So I guess the problem is caused by ollama trying to use the CPU to summarize the conversation and get the conversation title.

<!-- gh-comment-id:2888784796 --> @Ami-OS commented on GitHub (May 18, 2025): I have the same problem which upgrade to 0.7 # Env CPU: i7-13700K GPU: RTX 4080 Super RAM: 64 GB DDR5 Ollama: 0.7 Model: Gemma3:27B ollama ps: - Size: 24GB - Processor: 35%/65% CPU/GPU # Problem When I asked the second question, the CPU went through a long "pre-processing"? And the CPU usage was only 50% but the temperature reached 95-100 degrees. After CPU stress testing, I determined that my CPU would only soar to nearly 100 degrees when using the FPU (Floating Point Unit). Based on my observations: 1. **It was very fast when answering the first question**, the GPU started working almost immediately, after answering the question the GPU stopped working, the CPU usage rose to 50% and the temperature rose to 95 degrees (obviously the FPU unit started working), when the CPU usage returned to normal, the conversation title was summarized. 2. Before answering the second question, the CPU usage rate rose to 50% and the temperature rose to 95 degrees. **The GPU started to work only after the CPU usage rate returned to normal**. The GPU usage rate rose to 40%, and the GPU stopped working after the answer was completed. So I guess the problem is caused by ollama trying to use the CPU to summarize the conversation and get the conversation title.
GiteaMirror commented 2026-04-12 18:58:58 -05:00
Author
Owner
Copy Link

@QTaKs commented on GitHub (May 18, 2025):

My test results (archlinux, ollama-cuda 0.7, 32GB ram, 1650ti 4GB vram):

  • all text only models (qwen2.5-coder, qwen3, embedding models) - splits or completely on GPU
  • llava:7b - splits
  • moondream:1.8b - 100% on gpu
  • gemma3:4b - 100% on cpu
  • gemma3:1b - 100% on gpu
  • qwen2.5vl:7b - 100% on cpu
<!-- gh-comment-id:2888904633 --> @QTaKs commented on GitHub (May 18, 2025): My test results (archlinux, ollama-cuda 0.7, 32GB ram, 1650ti 4GB vram): - all text only models (qwen2.5-coder, qwen3, embedding models) - splits or completely on GPU - llava:7b - splits - moondream:1.8b - 100% on gpu - gemma3:4b - 100% on cpu - gemma3:1b - 100% on gpu - qwen2.5vl:7b - 100% on cpu
GiteaMirror commented 2026-04-12 18:59:00 -05:00
Author
Owner
Copy Link

@MR-444 commented on GitHub (May 18, 2025):

I can confirm this behaviour with Gemma 3 27B, on a RTX 4090, Windows 11.
VRAM use is the same as before, but the workload goes to the CPU mostly.

Setting the num_gpu (Ollama) to 62/63: via the Open WebUI gives me back the old behaviour, but with more VRAM use.

<!-- gh-comment-id:2888985067 --> @MR-444 commented on GitHub (May 18, 2025): I can confirm this behaviour with Gemma 3 27B, on a RTX 4090, Windows 11. VRAM use is the same as before, but the workload goes to the CPU mostly. Setting the num_gpu (Ollama) to 62/63: via the Open WebUI gives me back the old behaviour, but with more VRAM use.
GiteaMirror commented 2026-04-12 18:59:01 -05:00
Author
Owner
Copy Link

@abes200 commented on GitHub (May 18, 2025):

I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8

total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7
load duration: 138.9722ms
prompt eval count: 1737 token(s)
prompt eval duration: 27.0917399s
prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7
eval count: 281 token(s)
eval duration: 2m9.7601859s
eval rate: 2.17 tokens/s <-- Same response speed on both.

I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything.

<!-- gh-comment-id:2889027380 --> @abes200 commented on GitHub (May 18, 2025): I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8 total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7 load duration: 138.9722ms prompt eval count: 1737 token(s) prompt eval duration: 27.0917399s prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7 eval count: 281 token(s) eval duration: 2m9.7601859s eval rate: 2.17 tokens/s <-- Same response speed on both. I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything.
GiteaMirror commented 2026-04-12 18:59:02 -05:00
Author
Owner
Copy Link

@marcinm1234 commented on GitHub (May 18, 2025):

I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8

Yes, can also confirm that downgrade to 0.6.8 fixes the issue (3090, i7-10875h, 64GB RAM), can fully utilize GPU w/32b models loaded fully in VRAM, no CPU/GPU sharing.

<!-- gh-comment-id:2889030604 --> @marcinm1234 commented on GitHub (May 18, 2025): > I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8 Yes, can also confirm that downgrade to 0.6.8 fixes the issue (3090, i7-10875h, 64GB RAM), can fully utilize GPU w/32b models loaded fully in VRAM, no CPU/GPU sharing.
GiteaMirror commented 2026-04-12 18:59:03 -05:00
Author
Owner
Copy Link

@Lantrancy commented on GitHub (May 18, 2025):

I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8

total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7 load duration: 138.9722ms prompt eval count: 1737 token(s) prompt eval duration: 27.0917399s prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7 eval count: 281 token(s) eval duration: 2m9.7601859s eval rate: 2.17 tokens/s <-- Same response speed on both.

I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything.

even 0.6.8 also have problem, compare to other inference solutions like LM Studio and Jan, ollama‘s speed is significantly slower no matter how you jiggle those settings/env variables

<!-- gh-comment-id:2889038962 --> @Lantrancy commented on GitHub (May 18, 2025): > I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8 > > total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7 load duration: 138.9722ms prompt eval count: 1737 token(s) prompt eval duration: 27.0917399s prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7 eval count: 281 token(s) eval duration: 2m9.7601859s eval rate: 2.17 tokens/s <-- Same response speed on both. > > I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything. even 0.6.8 also have problem, compare to other inference solutions like LM Studio and Jan, ollama‘s speed is significantly slower no matter how you jiggle those settings/env variables
GiteaMirror commented 2026-04-12 18:59:03 -05:00
Author
Owner
Copy Link

@HDANILO commented on GitHub (May 18, 2025):

I have same problem, for me it is correctly allocated on the GPU but my 5090 can't even fit qwen2.5-coder:32b, where as before it could fit and still had some room left.

Continue.dev is simply not working either.

<!-- gh-comment-id:2889161776 --> @HDANILO commented on GitHub (May 18, 2025): I have same problem, for me it is correctly allocated on the GPU but my 5090 can't even fit qwen2.5-coder:32b, where as before it could fit and still had some room left. Continue.dev is simply not working either.
GiteaMirror commented 2026-04-12 18:59:04 -05:00
Author
Owner
Copy Link

@abes200 commented on GitHub (May 18, 2025):

I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8
total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7 load duration: 138.9722ms prompt eval count: 1737 token(s) prompt eval duration: 27.0917399s prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7 eval count: 281 token(s) eval duration: 2m9.7601859s eval rate: 2.17 tokens/s <-- Same response speed on both.
I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything.

even 0.6.8 also have problem, compare to other inference solutions like LM Studio and Jan, ollama‘s speed is significantly slower no matter how you jiggle those settings/env variables

Well those speeds are actually quite good for my poor old PC. I can't run LM Studio at all, doesn't support my CPU. As for Jan, I spent a good hour trying to get it working but had errors attempting to run any model. None of the solutions I found worked. Also, I really didn't like it's interface.

<!-- gh-comment-id:2889223216 --> @abes200 commented on GitHub (May 18, 2025): > > I'll also confirm that installing 0.6.8 fixes the problem. Here's the verbose output from the same gemma I mentioned earlier on 0.6.8 > > total duration: 2m37.0533755s <-- compared to 7m59.3012883s on 0.7 load duration: 138.9722ms prompt eval count: 1737 token(s) prompt eval duration: 27.0917399s prompt eval rate: 64.12 tokens/s <-- compared to 4.71 on 0.7 eval count: 281 token(s) eval duration: 2m9.7601859s eval rate: 2.17 tokens/s <-- Same response speed on both. > > I will just add again, on 0.7 it was still loading the model into VRAM. Clear as day in task manager. But it definitely was not utilizing the GPU to process anything. > > even 0.6.8 also have problem, compare to other inference solutions like LM Studio and Jan, ollama‘s speed is significantly slower no matter how you jiggle those settings/env variables Well those speeds are actually quite good for my poor old PC. I can't run LM Studio at all, doesn't support my CPU. As for Jan, I spent a good hour trying to get it working but had errors attempting to run any model. None of the solutions I found worked. Also, I really didn't like it's interface.
GiteaMirror commented 2026-04-12 18:59:05 -05:00
Author
Owner
Copy Link

@ALIENvsROBOT commented on GitHub (May 18, 2025):

Is anyone fixing this ??? I think the problem is from llama.cpp update

<!-- gh-comment-id:2889249515 --> @ALIENvsROBOT commented on GitHub (May 18, 2025): Is anyone fixing this ??? I think the problem is from llama.cpp update
GiteaMirror commented 2026-04-12 18:59:05 -05:00
Author
Owner
Copy Link

@Desslar commented on GitHub (May 19, 2025):

Yes in 0.6.8 Qwen3 32B Q8 32K reports needing 80GB(!) of Vram using ollama ps but only loads the GPU's to 50% VRAM and shuffles the rest to CPU.

<!-- gh-comment-id:2889483221 --> @Desslar commented on GitHub (May 19, 2025): Yes in 0.6.8 Qwen3 32B Q8 32K reports needing 80GB(!) of Vram using ollama ps but only loads the GPU's to 50% VRAM and shuffles the rest to CPU.
GiteaMirror commented 2026-04-12 18:59:06 -05:00
Author
Owner
Copy Link

@makolini commented on GitHub (May 20, 2025):

Had the same issue - Qwen3:32b Q4 running entirely on cpu with 2t/s...
Setting num_gpu = 132, for Qwen3:32b Q4, lets it run entirely in GPU again. (Layers + attention heads, 64 64 and 8)
You can replicate it for any problematic model.
E.g. for Llamaindex in python:
#from llama_index.llms.ollama import Ollama
llm = Ollama(
model=qwen3:32b,
#base_url =
#<(Other params)>
additional_kwargs = {
"num_gpu":132,
"num_ctx":32000 #this parameter will take up extra memory, so be careful
}
)
This fixes everything.

Alternatively you can ollama create a model with custom gguf and modelfile where you specify PARAMETER num_gpu 132 (based on layers)

<!-- gh-comment-id:2894602923 --> @makolini commented on GitHub (May 20, 2025): Had the same issue - Qwen3:32b Q4 running entirely on cpu with 2t/s... Setting num_gpu = 132, for Qwen3:32b Q4, lets it run entirely in GPU again. (Layers + attention heads, 64 64 and 8) You can replicate it for any problematic model. E.g. for Llamaindex in python: #from llama_index.llms.ollama import Ollama llm = Ollama( model=qwen3:32b, #base_url = #<(Other params)> additional_kwargs = { "num_gpu":132, "num_ctx":32000 #this parameter will take up extra memory, so be careful } ) This fixes everything. Alternatively you can ollama create a model with custom gguf and modelfile where you specify PARAMETER num_gpu 132 (based on layers)
GiteaMirror commented 2026-04-12 18:59:07 -05:00
Author
Owner
Copy Link

@ALIENvsROBOT commented on GitHub (May 20, 2025):

Guys I fixed it. give gemma 3 an image and prompt and send it (only for first time). It should fix everything. make sure the model is unloaded from the ram or GPU. once after fixing you can use it with text only.

<!-- gh-comment-id:2895073197 --> @ALIENvsROBOT commented on GitHub (May 20, 2025): Guys I fixed it. give gemma 3 an image and prompt and send it (only for first time). It should fix everything. make sure the model is unloaded from the ram or GPU. once after fixing you can use it with text only.
GiteaMirror commented 2026-04-12 18:59:07 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (May 20, 2025):

Fixed in https://github.com/ollama/ollama/pull/10773

<!-- gh-comment-id:2896080241 --> @jessegross commented on GitHub (May 20, 2025): Fixed in https://github.com/ollama/ollama/pull/10773
GiteaMirror commented 2026-04-12 18:59:08 -05:00
Author
Owner
Copy Link

@marcinm1234 commented on GitHub (May 26, 2025):

For me, the last working version is 0.6.8, both 0.7.0 and 0.7.1 still make gemma3:27b, qwen3:32b and mistral-small3.1:24b EXPLODE to sizes over 24GB VRAM, making it offload to CPU/system RAM, rendering the models unasable....

on 0.7.1 / mistral-small3.1:24b

ollama ps
NAME ID SIZE PROCESSOR UNTIL
mistral-small3.1:24b b9aaf0c2586a 27 GB 38%/62% CPU/GPU 59 minutes from now

qwen/gemma - split CPU/GPU - sloooooooow

0.6.8 - slow, but usable mistral-small3.1:24b / qwen3/Gemma3 100% GPU

ollama ps
NAME ID SIZE PROCESSOR UNTIL
mistral-small3.1:24b b9aaf0c2586a 28 GB 3%/97% CPU/GPU 59 minutes from now

ollama ps
NAME ID SIZE PROCESSOR UNTIL
qwen3:32b e1c9f234c6eb 24 GB 100% GPU 59 minutes from now

ollama ps
NAME ID SIZE PROCESSOR UNTIL
gemma3:27b-it-qat 29eb0b9aeda3 22 GB 100% GPU 59 minutes from now

I need 6K context window (via AnythingLLM).

Is anyone experiencing the same?

<!-- gh-comment-id:2910693205 --> @marcinm1234 commented on GitHub (May 26, 2025): For me, the last working version is 0.6.8, both 0.7.0 and 0.7.1 still make gemma3:27b, qwen3:32b and mistral-small3.1:24b EXPLODE to sizes over 24GB VRAM, making it offload to CPU/system RAM, rendering the models unasable.... **on 0.7.1 / mistral-small3.1:24b** > ollama ps NAME ID SIZE PROCESSOR UNTIL mistral-small3.1:24b b9aaf0c2586a 27 GB 38%/62% CPU/GPU 59 minutes from now qwen/gemma - split CPU/GPU - sloooooooow **0.6.8 - slow, but usable mistral-small3.1:24b / qwen3/Gemma3 100% GPU** > ollama ps NAME ID SIZE PROCESSOR UNTIL mistral-small3.1:24b b9aaf0c2586a 28 GB 3%/97% CPU/GPU 59 minutes from now > ollama ps NAME ID SIZE PROCESSOR UNTIL qwen3:32b e1c9f234c6eb 24 GB 100% GPU 59 minutes from now > ollama ps NAME ID SIZE PROCESSOR UNTIL gemma3:27b-it-qat 29eb0b9aeda3 22 GB 100% GPU 59 minutes from now I need 6K context window (via AnythingLLM). Is anyone experiencing the same?
GiteaMirror commented 2026-04-12 18:59:08 -05:00
Author
Owner
Copy Link

@goldyfruit commented on GitHub (Jul 2, 2025):

Still facing the same issue with version 0.9.4.

Qwen3 and Gemma3 models performs very slowly where all other models (DeepSeekR1, Qwen2.5, Llama3.1) perform very well.

<!-- gh-comment-id:3028551931 --> @goldyfruit commented on GitHub (Jul 2, 2025): Still facing the same issue with version `0.9.4`. Qwen3 and Gemma3 models performs very slowly where all other models _(DeepSeekR1, Qwen2.5, Llama3.1)_ perform very well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7063
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?