github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5252] How configure ollama to run on Virtual Nvidia GPU #3287

New Issue
Closed
opened 2026-04-12 13:51:02 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-04-12 13:51:02 -05:00
Owner
Copy Link

Originally created by @amdhugo on GitHub (Jun 24, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5252

Originally assigned to: @dhiltgen on GitHub.

I am running ubuntu on VSphere. Nvidia VGPU is running on it. Is there a doc to run ollama on vGPU not on vCPU by default? I think it might be the question has no business with virtual or not virtual GPU. But I don't search out a reference doc on how to configure ollama running on GPU. Anyone who can give me a hand?

Originally created by @amdhugo on GitHub (Jun 24, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5252 Originally assigned to: @dhiltgen on GitHub. I am running ubuntu on VSphere. Nvidia VGPU is running on it. Is there a doc to run ollama on vGPU not on vCPU by default? I think it might be the question has no business with virtual or not virtual GPU. But I don't search out a reference doc on how to configure ollama running on GPU. Anyone who can give me a hand?
GiteaMirror commented 2026-04-12 13:51:03 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jun 24, 2024):

Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU.

I have tried it on both Ubuntu and Windows 10 using vGPU

<!-- gh-comment-id:2186255251 --> @AncientMystic commented on GitHub (Jun 24, 2024): Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU. I have tried it on both Ubuntu and Windows 10 using vGPU
GiteaMirror commented 2026-04-12 13:51:04 -05:00
Author
Owner
Copy Link

@amdhugo commented on GitHub (Jun 25, 2024):

Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU.

I have tried it on both Ubuntu and Windows 10 using vGPU

I will try it. Is there any special configuration on ollama after installing proxmox to enable GPU compute?

<!-- gh-comment-id:2187779432 --> @amdhugo commented on GitHub (Jun 25, 2024): > Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU. > > I have tried it on both Ubuntu and Windows 10 using vGPU I will try it. Is there any special configuration on ollama after installing proxmox to enable GPU compute?
GiteaMirror commented 2026-04-12 13:51:05 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jun 25, 2024):

Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU.

I have tried it on both Ubuntu and Windows 10 using vGPU

I will try it. Is there any special configuration on ollama after installing proxmox to enable GPU compute?

If you already have vsphere with vgpu theres no need for proxmox, proxmox is a hypervisor OS, you should only need to install ollama in your VM and begin using it.

<!-- gh-comment-id:2187907752 --> @AncientMystic commented on GitHub (Jun 25, 2024): > > Personally i run ollama on a Nvidia vGPU using proxmox, it has no problems detecting it just like any other Nvidia GPU. > > > > > > I have tried it on both Ubuntu and Windows 10 using vGPU > > > > I will try it. Is there any special configuration on ollama after installing proxmox to enable GPU compute? If you already have vsphere with vgpu theres no need for proxmox, proxmox is a hypervisor OS, you should only need to install ollama in your VM and begin using it.
GiteaMirror commented 2026-04-12 13:51:07 -05:00
Author
Owner
Copy Link

@amdhugo commented on GitHub (Jun 25, 2024):

I can run ollama on ubuntu(this is a VM in Vsphere) via this command:
ollama run llama3
However, I see the stress is on CPU not GPU(I run nvidia-smi to check it).
I install ollama followed by instruction on 'CLI Reference' here https://github.com/ollama/ollama
Shalll I rebuild ollama in CUDA environment refere to https://github.com/ollama/ollama/blob/main/docs/development.md ?'

<!-- gh-comment-id:2187952646 --> @amdhugo commented on GitHub (Jun 25, 2024): I can run ollama on ubuntu(this is a VM in Vsphere) via this command: ollama run llama3 However, I see the stress is on CPU not GPU(I run nvidia-smi to check it). I install ollama followed by instruction on 'CLI Reference' here https://github.com/ollama/ollama Shalll I rebuild ollama in CUDA environment refere to https://github.com/ollama/ollama/blob/main/docs/development.md ?'
GiteaMirror commented 2026-04-12 13:51:08 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jun 25, 2024):

I can run ollama on ubuntu(this is a VM in Vsphere) via this command:

ollama run llama3

However, I see the stress is on CPU not GPU(I run nvidia-smi to check it).

I install ollama followed by instruction on 'CLI Reference' here https://github.com/ollama/ollama

Shalll I rebuild ollama in CUDA environment refere to https://github.com/ollama/ollama/blob/main/docs/development.md ?'

From my experiences ollama usually just automatically runs on vGPU devices, ive never had it fail, you should check the logs by running

journalctl -e -u ollama

To see if it is detecting your vGPU and using it properly or not. If the vram is under 2gb it will skip the device, that is one reason it could be failing. (You want as much vram as possible for ollama)

Do you have the vgpu drivers and licensing configured properly, etc? (Licensing is only necessary If you are using tesla / compute gpus on the host)

If you post the logs others may be able to provide more help too, it should be using your gpu as it is if everything is setup.

<!-- gh-comment-id:2187967101 --> @AncientMystic commented on GitHub (Jun 25, 2024): > I can run ollama on ubuntu(this is a VM in Vsphere) via this command: > > ollama run llama3 > > However, I see the stress is on CPU not GPU(I run nvidia-smi to check it). > > I install ollama followed by instruction on 'CLI Reference' here https://github.com/ollama/ollama > > Shalll I rebuild ollama in CUDA environment refere to https://github.com/ollama/ollama/blob/main/docs/development.md ?' From my experiences ollama usually just automatically runs on vGPU devices, ive never had it fail, you should check the logs by running journalctl -e -u ollama To see if it is detecting your vGPU and using it properly or not. If the vram is under 2gb it will skip the device, that is one reason it could be failing. (You want as much vram as possible for ollama) Do you have the vgpu drivers and licensing configured properly, etc? (Licensing is only necessary If you are using tesla / compute gpus on the host) If you post the logs others may be able to provide more help too, it should be using your gpu as it is if everything is setup.
GiteaMirror commented 2026-04-12 13:51:09 -05:00
Author
Owner
Copy Link

@amdhugo commented on GitHub (Jun 25, 2024):

Thanks, I think it might be due to 1G VRAM. I would try to assign more vram on it.

<!-- gh-comment-id:2187997541 --> @amdhugo commented on GitHub (Jun 25, 2024): Thanks, I think it might be due to 1G VRAM. I would try to assign more vram on it.
GiteaMirror commented 2026-04-12 13:51:09 -05:00
Author
Owner
Copy Link

@AncientMystic commented on GitHub (Jun 25, 2024):

Thanks, I think it might be due to 1G VRAM. I would try to assign more vram on it.

You're welcome, yeah, that sounds like it's your issue here, give it a minimum of 2gb, (in my experience 4-6gb is basically minimum for a decent experience 6-7gb+ is basically minimum for a fairly good experience)

Also remember to use the command

Ollama ps

To check when a model is loaded to see what ollama reports as loaded on gpu or cpu.

Smaller models with lower quantisations (q5 is usually the best balance) and lower context sizes will help with lower vram, orca mini is a good model for low vram just to start playing with it and make sure it works, the responses aren't great but it works and its small/fast.

<!-- gh-comment-id:2188029236 --> @AncientMystic commented on GitHub (Jun 25, 2024): > Thanks, I think it might be due to 1G VRAM. I would try to assign more vram on it. You're welcome, yeah, that sounds like it's your issue here, give it a minimum of 2gb, (in my experience 4-6gb is basically minimum for a decent experience 6-7gb+ is basically minimum for a fairly good experience) Also remember to use the command Ollama ps To check when a model is loaded to see what ollama reports as loaded on gpu or cpu. Smaller models with lower quantisations (q5 is usually the best balance) and lower context sizes will help with lower vram, orca mini is a good model for low vram just to start playing with it and make sure it works, the responses aren't great but it works and its small/fast.
GiteaMirror commented 2026-04-12 13:51:10 -05:00
Author
Owner
Copy Link

@vimyang commented on GitHub (Jun 25, 2024):

I encountered the same situation using debian. I'll describe the usage scenario.

At first it starts up using the GPU, but after an hour or two of use, the system becomes inaccessible (but a later top command confirms that the system has not rebooted). Once it's accessible, restarting ollama again will use only the CPU and the graphics card status will be normal, except that there is no ollama in the graphics card processes

Environment used

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm)
Codename: bookworm

ollama --version
ollama version is 0.1.39

Before rebooting

Check the logs

image

Checking the status of the process

image

Checking out the graphics card

image

Even when I try to modify it, it doesn't work

>>> /set parameter num_gpu 28
Set parameter 'num_gpu' to '28'

After restarting the computer, it automatically recovers, I've had this happen many times!
image

<!-- gh-comment-id:2188079950 --> @vimyang commented on GitHub (Jun 25, 2024): I encountered the same situation using debian. I'll describe the usage scenario. At first it starts up using the GPU, but after an hour or two of use, the system becomes inaccessible (but a later `top` command confirms that the system has not rebooted). Once it's accessible, restarting ollama again will use only the CPU and the graphics card status will be normal, except that there is no ollama in the graphics card processes Environment used ```Plain Text lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Codename: bookworm ollama --version ollama version is 0.1.39 ``` Before rebooting Check the logs ![image](https://github.com/ollama/ollama/assets/158011334/b1975c24-6cea-43bb-b34f-1a9a1363b9cb) Checking the status of the process ![image](https://github.com/ollama/ollama/assets/158011334/1099f32f-b8b7-4eb5-a4bd-b5e4f796972a) Checking out the graphics card ![image](https://github.com/ollama/ollama/assets/158011334/5d54a95d-530b-4b2a-a7fc-5e3bd7184de1) Even when I try to modify it, it doesn't work ```Plain Text >>> /set parameter num_gpu 28 Set parameter 'num_gpu' to '28' ``` After restarting the computer, it automatically recovers, I've had this happen many times! ![image](https://github.com/ollama/ollama/assets/158011334/9ce885dd-4f14-425a-be21-a874fb3e8769)
GiteaMirror commented 2026-04-12 13:51:11 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 25, 2024):

If you check the server log that should say why we were no longer able to communicate with the GPU. My suspicion is this may be similar to the container scenarios where the UVM driver unloads after a while. Check out https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#container-fails-to-run-on-nvidia-gpu and see if any of those steps help revive the GPU.

<!-- gh-comment-id:2189379054 --> @dhiltgen commented on GitHub (Jun 25, 2024): If you check the server log that should say why we were no longer able to communicate with the GPU. My suspicion is this may be similar to the container scenarios where the UVM driver unloads after a while. Check out https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#container-fails-to-run-on-nvidia-gpu and see if any of those steps help revive the GPU.
GiteaMirror commented 2026-04-12 13:51:11 -05:00
Author
Owner
Copy Link

@vimyang commented on GitHub (Jun 26, 2024):

Thank you for your reply. I followed the instructions in the document (although it is docker), added the CUDA_ERROR_LEVEL=50 parameter and restarted, and there has been no problem so far. I'll sync again after the next problem occurs. Thanks again

OLLAMA_FLASH_ATTENTION=True OLLAMA_HOST=0.0.0.0 CUDA_ERROR_LEVEL=50 ollama serve
<!-- gh-comment-id:2191550218 --> @vimyang commented on GitHub (Jun 26, 2024): Thank you for your reply. I followed the instructions in the document (although it is docker), added the `CUDA_ERROR_LEVEL=50` parameter and restarted, and there has been no problem so far. I'll sync again after the next problem occurs. Thanks again ``` OLLAMA_FLASH_ATTENTION=True OLLAMA_HOST=0.0.0.0 CUDA_ERROR_LEVEL=50 ollama serve ```
GiteaMirror commented 2026-04-12 13:51:12 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 4, 2024):

Happy to hear those troubleshooting steps cleared up the problem.

<!-- gh-comment-id:2207573046 --> @dhiltgen commented on GitHub (Jul 4, 2024): Happy to hear those troubleshooting steps cleared up the problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3287
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?