github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-24 06:31:50 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4506] Any way to increase performance? And switch to F32? #49335

New Issue
Closed
opened 2026-04-28 11:16:36 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-28 11:16:36 -05:00
Owner
Copy Link

Originally created by @AncientMystic on GitHub (May 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4506

I am using a pascal Tesla P4 8gb gpu and i am looking for a way to increase performance.

Are there any tweaks/environment variables i can apply or things i can install such a pytorch version or something that will boost ollama performance?

I am getting very mixed results, any model bigger than a few gb has a massive performance loss, roughly 2-6x slower generation over a model that is under 2gb, 4-5gb is 2x slower 6-8gb models are about 6+x slower and models over roughly 8-11gb either are so slow they are useless or won't load.

One recent test a response of 521 tokens took 20 minutes on a 8gb model (which is something like 0.4 tokens/s)

It is to be expected to some degree as i do not have adequate vram to run extremely large models but it would be nice if i could somehow get at least slightly faster results on models most if not all of should fit into vram

Also is there a setting to try F32 over F16 precision? Pascals seem to have much higher f32 performance so I figure it is worth a try

I have already set OLLAMA_NUM_PARALLEL & OLLAMA_MAX_LOADED to 1 to achieve lower vram usage (any more tweaks would be very much so appreciated)

System:
OS: proxmox
CPU: i7-6700
Mem: 64GB DDR4 2133mhz
Main drive: 1TB nvme

VM: ubuntu 22.04.4 LTS
Ollama: 0.1.38
vGPU: GRID-P4-2Q 6GB profile
Mem: 32gb

Originally created by @AncientMystic on GitHub (May 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4506 I am using a pascal Tesla P4 8gb gpu and i am looking for a way to increase performance. Are there any tweaks/environment variables i can apply or things i can install such a pytorch version or something that will boost ollama performance? I am getting very mixed results, any model bigger than a few gb has a massive performance loss, roughly 2-6x slower generation over a model that is under 2gb, 4-5gb is 2x slower 6-8gb models are about 6+x slower and models over roughly 8-11gb either are so slow they are useless or won't load. One recent test a response of 521 tokens took 20 minutes on a 8gb model (which is something like 0.4 tokens/s) It is to be expected to some degree as i do not have adequate vram to run extremely large models but it would be nice if i could somehow get at least slightly faster results on models most if not all of should fit into vram Also is there a setting to try F32 over F16 precision? Pascals seem to have much higher f32 performance so I figure it is worth a try I have already set OLLAMA_NUM_PARALLEL & OLLAMA_MAX_LOADED to 1 to achieve lower vram usage (any more tweaks would be very much so appreciated) System: OS: proxmox CPU: i7-6700 Mem: 64GB DDR4 2133mhz Main drive: 1TB nvme VM: ubuntu 22.04.4 LTS Ollama: 0.1.38 vGPU: GRID-P4-2Q 6GB profile Mem: 32gb
GiteaMirror added the question label 2026-04-28 11:16:36 -05:00
GiteaMirror commented 2026-04-28 11:16:41 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 18, 2024):

Hey @AncientMystic , the problem is the P4 doesn't have a lot of vram on it, so almost certainly you're offloading a lot of layers onto the GPU. You can verify this w/ the ollama ps command which was just introduced in 0.1.38.

Increasing from a non-quantized 16 bit model to a non-quantized 32 bit model (which most of the models don't even come in those sizes) would only exacerbate the issue. You can increase the quantization level (i.e. use a 4bit or lower quantized model) which will increase the performance, but you're going to sacrifice the accuracy/quality of the results. You could also decrease the context size which will save more memory

The short answer is either:

  • choose a more quantized model;
  • decrease the context size; or
  • (unfortunately) upgrade to a beefier GPU

I hope that helps! I close out the issue, but feel free to keep commenting.

<!-- gh-comment-id:2119021755 --> @pdevine commented on GitHub (May 18, 2024): Hey @AncientMystic , the problem is the P4 doesn't have a lot of vram on it, so almost certainly you're offloading a lot of layers onto the GPU. You can verify this w/ the `ollama ps` command which was just introduced in `0.1.38`. Increasing from a non-quantized 16 bit model to a non-quantized 32 bit model (which most of the models don't even come in those sizes) would only exacerbate the issue. You can _increase_ the quantization level (i.e. use a 4bit or lower quantized model) which will increase the performance, but you're going to sacrifice the accuracy/quality of the results. You could also decrease the context size which will save more memory The short answer is either: * choose a more quantized model; * decrease the context size; or * (unfortunately) upgrade to a beefier GPU I hope that helps! I close out the issue, but feel free to keep commenting.
GiteaMirror commented 2026-04-28 11:16:44 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (May 19, 2024):

Thank you for your response, i am using lower models than F16 already my favourites are usually the Q5 for performance vs quality but i was referring to ollama itself not the models used, the buffers and everything seem to be using f16 and for the p4 for example half precision performance is 89 GFLOPS where as full precision is 5.7 TFLOPS the card (as many others) just does not perform well with half precision since most gpus aren't made with that in mind, so i was hoping f32 would give some sort of benefits, other tools and AI image generators, etc i was able to switch from f16 to f32 with this card yielded quite a boost in performance, was hoping there was an environment variable to force full precision in ollama itself.

<!-- gh-comment-id:2119034886 --> @AncientMystic commented on GitHub (May 19, 2024): Thank you for your response, i am using lower models than F16 already my favourites are usually the Q5 for performance vs quality but i was referring to ollama itself not the models used, the buffers and everything seem to be using f16 and for the p4 for example half precision performance is 89 GFLOPS where as full precision is 5.7 TFLOPS the card (as many others) just does not perform well with half precision since most gpus aren't made with that in mind, so i was hoping f32 would give some sort of benefits, other tools and AI image generators, etc i was able to switch from f16 to f32 with this card yielded quite a boost in performance, was hoping there was an environment variable to force full precision in ollama itself.
GiteaMirror commented 2026-04-28 11:16:45 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 19, 2024):

@AncientMystic unfortunately not in ollama. Ollama uses llama.cpp as the runner for running inference, and I'm not aware of of any options which will allow you to do that (although it's certainly plausible).

<!-- gh-comment-id:2119097312 --> @pdevine commented on GitHub (May 19, 2024): @AncientMystic unfortunately not in ollama. Ollama uses llama.cpp as the runner for running inference, and I'm not aware of of any options which will allow you to do that (although it's certainly plausible).
GiteaMirror commented 2026-04-28 11:16:47 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (May 19, 2024):

Thank you for your responses i really appreciate it.

I found that llama.cpp supports "--memory-f32" which uses f32 instead of f16 and might be what i am looking for (or at least a step in the right direction possibly) but also doubles the context memory requirement and cached prompt file size, its not recommended and i think it would probably be a bad choice on newer gpus with tensor cores, but it might be beneficial on pascal with such abysmal f16 performance, theres a few commands i kind of want to try from llama.cpp

is there a way to use llama.cpp flags with ollama?

<!-- gh-comment-id:2119240677 --> @AncientMystic commented on GitHub (May 19, 2024): Thank you for your responses i really appreciate it. I found that llama.cpp supports "--memory-f32" which uses f32 instead of f16 and might be what i am looking for (or at least a step in the right direction possibly) but also doubles the context memory requirement and cached prompt file size, its not recommended and i think it would probably be a bad choice on newer gpus with tensor cores, but it might be beneficial on pascal with such abysmal f16 performance, theres a few commands i kind of want to try from llama.cpp is there a way to use llama.cpp flags with ollama?
GiteaMirror commented 2026-04-28 11:16:49 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (May 22, 2024):

I just was playing around with LM Studio and ollama using the same model (7B Q5_k_M)

Ollama the best i can seem to get is 8 tokens/s, averages 4-7 it dips to 1-2 tokens/s at times with ollama ps reporting only 13-18% of the model is on cpu not gpu

LM studio using the same model as ollama and a context of 2048 achieves 19.93 tokens/s (highest Ive seen is 22) context 8192 achieves 13.13 tokens/s and 3.42 tokens/s with cpu only not using the gpu at all 🤔

I have also tried a 13B model which achieves 1.86 tokens/s in ollama and in LM studio it is 2.64 tokens/s this is much lower of a difference than the smaller model.

Wondering what i can do to get closer to the lm studio performance in ollama

<!-- gh-comment-id:2123699148 --> @AncientMystic commented on GitHub (May 22, 2024): I just was playing around with LM Studio and ollama using the same model (7B Q5_k_M) Ollama the best i can seem to get is 8 tokens/s, averages 4-7 it dips to 1-2 tokens/s at times with ollama ps reporting only 13-18% of the model is on cpu not gpu LM studio using the same model as ollama and a context of 2048 achieves 19.93 tokens/s (highest Ive seen is 22) context 8192 achieves 13.13 tokens/s and 3.42 tokens/s with cpu only not using the gpu at all 🤔 I have also tried a 13B model which achieves 1.86 tokens/s in ollama and in LM studio it is 2.64 tokens/s this is much lower of a difference than the smaller model. Wondering what i can do to get closer to the lm studio performance in ollama
GiteaMirror commented 2026-04-28 11:16:51 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jun 9, 2024):

@pdevine just found this llama.cpp fork with a vector kernel specifically for fp32 no tensor core flash attention made to run on pascal gpus that seems to accelerate token/s numbers by almost 2x

https://github.com/JohannesGaessler/llama.cpp/tree/cuda-fa-no-tc-11

And this detailing results and a pull request merging into llama.cpp

https://github.com/ggerganov/llama.cpp/pull/7188

Many users seem to be still using pascal especially with the p40-p100s so this might be a great option to boost speeds if it could be added with an environment veritable to enable it like OLLAMA_FA_FP32 / FA_NOTC or something

<!-- gh-comment-id:2156417077 --> @AncientMystic commented on GitHub (Jun 9, 2024): @pdevine just found this llama.cpp fork with a vector kernel specifically for fp32 no tensor core flash attention made to run on pascal gpus that seems to accelerate token/s numbers by almost 2x https://github.com/JohannesGaessler/llama.cpp/tree/cuda-fa-no-tc-11 And this detailing results and a pull request merging into llama.cpp https://github.com/ggerganov/llama.cpp/pull/7188 Many users seem to be still using pascal especially with the p40-p100s so this might be a great option to boost speeds if it could be added with an environment veritable to enable it like OLLAMA_FA_FP32 / FA_NOTC or something
GiteaMirror commented 2026-04-28 11:16:51 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jul 29, 2024):

@pdevine

I think PR #5894 potentially would solve this, although at the cost of more memory usage, but the other quant options may increase performance aswell, with the benefit of decreasing memory consumption

https://github.com/ollama/ollama/pull/5894

<!-- gh-comment-id:2256794672 --> @AncientMystic commented on GitHub (Jul 29, 2024): @pdevine I think PR #5894 potentially would solve this, although at the cost of more memory usage, but the other quant options may increase performance aswell, with the benefit of decreasing memory consumption https://github.com/ollama/ollama/pull/5894
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#49335
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?