github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7210] Intel Arc + NVIDIA Docker Setup #4578

New Issue
Closed
opened 2026-04-12 15:30:27 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-12 15:30:27 -05:00
Owner
Copy Link

Originally created by @blunweon on GitHub (Oct 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7210

What is the issue?

Hi all,

I have recently gotten a new Intel Arc A380 and hoped to make it work with Ollama. I have tried reading issues on Github to get a sense on how supported Intel dGPUs are and it seemed to be working for some.

I am currently running it in Docker under Proxmox LXC.

This is my compose.yaml file:

version: "3.3"
services:
  ollama:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    volumes:
      - ollama:/root/.ollama
      - /mnt/nas-hdd0/models:/root/models
    ports:
      - 11434:11434
    environment:
      OLLAMA_ORIGINS: "*"
      OLLAMA_HOST: 0.0.0.0
      OLLAMA_KEEP_ALIVE: 15
      OLLAMA_SCHED_SPREAD: 1
      OLLAMA_DEBUG: 1
      OLLAMA_INTEL_GPU: 1
      NEOReadDebugKeys: 1
      OverrideGpuAddressSpace: 48
    container_name: ollama
    image: ollama/ollama
    restart: always
volumes:
  ollama: {}
networks: {}

My Docker logs:

2024/10/14 16:44:12 routes.go:1158: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:true OLLAMA_KEEP_ALIVE:15s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:true OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-10-14T16:44:12.616Z level=INFO source=images.go:754 msg="total blobs: 74"
time=2024-10-14T16:44:12.618Z level=INFO source=images.go:761 msg="total unused blobs removed: 0"
time=2024-10-14T16:44:12.618Z level=INFO source=routes.go:1205 msg="Listening on [::]:11434 (version 0.3.13)"
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11/ollama_llama_server
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12/ollama_llama_server
time=2024-10-14T16:44:12.619Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12]"
time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-10-14T16:44:12.619Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-10-14T16:44:12.619Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-10-14T16:44:12.620Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.90.07]
CUDA driver version: 12.4
time=2024-10-14T16:44:12.635Z level=DEBUG source=gpu.go:118 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.90.07
[GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] CUDA totalMem 5924 mb
[GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] CUDA freeMem 5851 mb
[GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] Compute Capability 7.5
time=2024-10-14T16:44:12.854Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libze_intel_gpu.so*
time=2024-10-14T16:44:12.854Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]"
time=2024-10-14T16:44:12.855Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2024-10-14T16:44:12.855Z level=DEBUG source=amd_linux.go:376 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2024-10-14T16:44:12.855Z level=INFO source=types.go:107 msg="inference compute" id=GPU-71904de4-2b69-b701-99e0-5d39fa30f86c library=cuda variant=v12 compute=7.5 driver=12.4 name="NVIDIA GeForce GTX 1660 Ti" total="5.8 GiB" available="5.7 GiB"

Based on the above logs, it seems to get a bug(?) after discovering GPU libraries for Intel? I do have Intel drivers installed within the LXC, and other transcoding/ML tasks in Immich works.

I am able to find the inferred GPU library so technically it should work?

> ls /usr/lib/x86_64-linux-gnu/libze_intel*
/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1
/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20

Using the Linux installation script and running OLLAMA_INTEL_GPU=1 ollama serve managed to get both Intel Arc and NVIDIA GPUs detected by Ollama.

However, it seems that there might be some issues with the Docker image as it should still be able to detect the Intel GPU hmmmm

OS

Linux, Docker

GPU

Nvidia, Intel

CPU

Intel

Ollama version

0.3.13

Originally created by @blunweon on GitHub (Oct 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7210 ### What is the issue? Hi all, I have recently gotten a new Intel Arc A380 and hoped to make it work with Ollama. I have tried reading issues on Github to get a sense on how supported Intel dGPUs are and it seemed to be working for some. I am currently running it in Docker under Proxmox LXC. This is my compose.yaml file: ``` version: "3.3" services: ollama: deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: - gpu volumes: - ollama:/root/.ollama - /mnt/nas-hdd0/models:/root/models ports: - 11434:11434 environment: OLLAMA_ORIGINS: "*" OLLAMA_HOST: 0.0.0.0 OLLAMA_KEEP_ALIVE: 15 OLLAMA_SCHED_SPREAD: 1 OLLAMA_DEBUG: 1 OLLAMA_INTEL_GPU: 1 NEOReadDebugKeys: 1 OverrideGpuAddressSpace: 48 container_name: ollama image: ollama/ollama restart: always volumes: ollama: {} networks: {} ``` My Docker logs: ``` 2024/10/14 16:44:12 routes.go:1158: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:true OLLAMA_KEEP_ALIVE:15s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:true OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2024-10-14T16:44:12.616Z level=INFO source=images.go:754 msg="total blobs: 74" time=2024-10-14T16:44:12.618Z level=INFO source=images.go:761 msg="total unused blobs removed: 0" time=2024-10-14T16:44:12.618Z level=INFO source=routes.go:1205 msg="Listening on [::]:11434 (version 0.3.13)" time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11/ollama_llama_server time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12/ollama_llama_server time=2024-10-14T16:44:12.619Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12]" time=2024-10-14T16:44:12.619Z level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-10-14T16:44:12.619Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-10-14T16:44:12.619Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs" time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA" time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so* time=2024-10-14T16:44:12.619Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2024-10-14T16:44:12.620Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.550.90.07] CUDA driver version: 12.4 time=2024-10-14T16:44:12.635Z level=DEBUG source=gpu.go:118 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.90.07 [GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] CUDA totalMem 5924 mb [GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] CUDA freeMem 5851 mb [GPU-71904de4-2b69-b701-99e0-5d39fa30f86c] Compute Capability 7.5 time=2024-10-14T16:44:12.854Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libze_intel_gpu.so* time=2024-10-14T16:44:12.854Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]" time=2024-10-14T16:44:12.855Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[] time=2024-10-14T16:44:12.855Z level=DEBUG source=amd_linux.go:376 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2024-10-14T16:44:12.855Z level=INFO source=types.go:107 msg="inference compute" id=GPU-71904de4-2b69-b701-99e0-5d39fa30f86c library=cuda variant=v12 compute=7.5 driver=12.4 name="NVIDIA GeForce GTX 1660 Ti" total="5.8 GiB" available="5.7 GiB" ``` Based on the above logs, it seems to get a bug(?) after discovering GPU libraries for Intel? I do have Intel drivers installed within the LXC, and other transcoding/ML tasks in Immich works. I am able to find the inferred GPU library so technically it should work? ``` > ls /usr/lib/x86_64-linux-gnu/libze_intel* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20 ``` Using the Linux installation script and running `OLLAMA_INTEL_GPU=1 ollama serve` managed to get both Intel Arc and NVIDIA GPUs detected by Ollama. However, it seems that there might be some issues with the Docker image as it should still be able to detect the Intel GPU hmmmm ### OS Linux, Docker ### GPU Nvidia, Intel ### CPU Intel ### Ollama version 0.3.13
GiteaMirror added the bug label 2026-04-12 15:30:27 -05:00
GiteaMirror commented 2026-04-12 15:30:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Oct 15, 2024):

Did you run the ls command on the host system or inside the container? I had a look in the 0.3.13 docker image and it looks like the libraries are not shipped with the image.

<!-- gh-comment-id:2413990493 --> @rick-github commented on GitHub (Oct 15, 2024): Did you run the `ls` command on the host system or inside the container? I had a look in the 0.3.13 docker image and it looks like the libraries are not shipped with the image.
GiteaMirror commented 2026-04-12 15:30:29 -05:00
Author
Owner
Copy Link

@blunweon commented on GitHub (Oct 15, 2024):

Good question, it seems that I ran the ls command on the host instead of within the container... Though that discovery would mean that there is missing libraries within the Docker image yes?

Would I be able to mount the missing libraries through the compose file from the host system or it would not be able to work for discovery?

EDIT: This line in particular intrigued me that prompted me to seek help regarding the missing library/path: time=2024-10-14T16:44:12.855Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]

<!-- gh-comment-id:2414033451 --> @blunweon commented on GitHub (Oct 15, 2024): Good question, it seems that I ran the `ls` command on the host instead of within the container... Though that discovery would mean that there is missing libraries within the Docker image yes? Would I be able to mount the missing libraries through the compose file from the host system or it would not be able to work for discovery? EDIT: This line in particular intrigued me that prompted me to seek help regarding the missing library/path: `time=2024-10-14T16:44:12.855Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]`
GiteaMirror commented 2026-04-12 15:30:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Oct 15, 2024):

Mounting the library should work, provided the library on the host system is not many versions away from what ollama was built with. The Dockerfile has commands for copying library files for cuda, CPU, rocm, etc, I wonder if it's just missing the commands for oneAPI libs.

<!-- gh-comment-id:2414140747 --> @rick-github commented on GitHub (Oct 15, 2024): Mounting the library should work, provided the library on the host system is not many versions away from what ollama was built with. The Dockerfile has commands for copying library files for cuda, CPU, rocm, etc, I wonder if it's just missing the commands for oneAPI libs.
GiteaMirror commented 2026-04-12 15:30:30 -05:00
Author
Owner
Copy Link

@blunweon commented on GitHub (Oct 15, 2024):

Mounting the library did work, however I run into this new error:

time=2024-10-15T14:46:27.096Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]"
time=2024-10-15T14:46:27.098Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20]
wiring Level-Zero management library functions in /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20
dlsym: zesInit
dlsym: zesDriverGet
dlsym: zesDeviceGet
dlsym: zesDeviceGetProperties
dlsym: zesDeviceEnumMemoryModules
dlsym: zesMemoryGetProperties
dlsym: zesMemoryGetState
calling zesInit
zesInit err: 78000001
releasing oneapi library
time=2024-10-15T14:46:27.115Z level=DEBUG source=gpu.go:605 msg="Unable to load oneAPI management library" library=/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20 error="oneapi vram init failure: 78000001"

It seems possible that the OneAPI library might be too new and not understood by the current Docker image?

<!-- gh-comment-id:2414156028 --> @blunweon commented on GitHub (Oct 15, 2024): Mounting the library did work, however I run into this new error: ``` time=2024-10-15T14:46:27.096Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]" time=2024-10-15T14:46:27.098Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20] wiring Level-Zero management library functions in /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20 dlsym: zesInit dlsym: zesDriverGet dlsym: zesDeviceGet dlsym: zesDeviceGetProperties dlsym: zesDeviceEnumMemoryModules dlsym: zesMemoryGetProperties dlsym: zesMemoryGetState calling zesInit zesInit err: 78000001 releasing oneapi library time=2024-10-15T14:46:27.115Z level=DEBUG source=gpu.go:605 msg="Unable to load oneAPI management library" library=/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20 error="oneapi vram init failure: 78000001" ``` It seems possible that the OneAPI library might be too new and not understood by the current Docker image?
GiteaMirror commented 2026-04-12 15:30:30 -05:00
Author
Owner
Copy Link

@blunweon commented on GitHub (Oct 15, 2024):

After much trial and error, I got it to work!
My docker logs:

time=2024-10-15T15:34:37.663Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libze_intel_gpu.so*
time=2024-10-15T15:34:37.664Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]"
time=2024-10-15T15:34:37.665Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20]
wiring Level-Zero management library functions in /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20
dlsym: zesInit
dlsym: zesDriverGet
dlsym: zesDeviceGet
dlsym: zesDeviceGetProperties
dlsym: zesDeviceEnumMemoryModules
dlsym: zesMemoryGetProperties
dlsym: zesMemoryGetState
calling zesInit
calling zesDriverGet
oneapi driver count: 1
calling zesDeviceGet count 0: 0x7ca1fd55d6d0
[0:0] oneAPI device name: Intel(R) Arc(TM) A380 Graphics
[0:0] oneAPI brand: unknown
[0:0] oneAPI vendor: Intel(R) Corporation
[0:0] oneAPI S/N: unknown
[0:0] oneAPI board number: unknown
discovered 1 Level-Zero memory modules
time=2024-10-15T15:34:37.691Z level=DEBUG source=amd_linux.go:376 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
releasing oneapi library
time=2024-10-15T15:34:37.694Z level=INFO source=types.go:107 msg="inference compute" id=GPU-71904de4-2b69-b701-99e0-5d39fa30f86c library=cuda variant=v12 compute=7.5 driver=12.4 name="NVIDIA GeForce GTX 1660 Ti" total="5.8 GiB" available="3.7 GiB"
time=2024-10-15T15:34:37.694Z level=INFO source=types.go:107 msg="inference compute" id=0 library=oneapi variant="" compute="" driver=0.0 name="Intel(R) Arc(TM) A380 Graphics" total="5.9 GiB" available="5.6 GiB"

My compose.yaml:

version: "3.3"
services:
  ollama:
    deploy:
      resources:
        limits:
          # Number of CPU threads
          cpus: "16.00"
          # Gigabytes of memory
          memory: 128G
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    devices:
        - /dev/dri:/dev/dri
    volumes:
      - ollama:/root/.ollama
      - /mnt/nas-hdd0/models:/root/models
      - /usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu
    ports:
      - 11434:11434
    environment:
      OLLAMA_ORIGINS: "*"
      OLLAMA_HOST: 0.0.0.0
      OLLAMA_KEEP_ALIVE: 15
      OLLAMA_SCHED_SPREAD: 1
      OLLAMA_DEBUG: 1
      OLLAMA_INTEL_GPU: 1
      NEOReadDebugKeys: 1
      OverrideGpuAddressSpace: 48
    container_name: ollama
    image: ollama/ollama
    restart: always
volumes:
  ollama: {}
networks: {}

So it seemed I simply needed to pass my /dev/dri devices and the /usr/lib/x86_64-linux-gnu directory through to the Ollama container within Docker as well as enable the environment flags in order for it to discover the GPU.

EDIT: In hindsight this seems like a quick and dirty solution. The Docker image might still need updating to include the libraries? (I'm not sure as I'm not experienced with it, just giving my inputs 😄)

I'm leaving my solution here in case of random Google searches.

<!-- gh-comment-id:2414377146 --> @blunweon commented on GitHub (Oct 15, 2024): After much trial and error, I got it to work! My docker logs: ``` time=2024-10-15T15:34:37.663Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libze_intel_gpu.so* time=2024-10-15T15:34:37.664Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libze_intel_gpu.so* /usr/local/nvidia/lib/libze_intel_gpu.so* /usr/local/nvidia/lib64/libze_intel_gpu.so* /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so* /usr/lib*/libze_intel_gpu.so*]" time=2024-10-15T15:34:37.665Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20] wiring Level-Zero management library functions in /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1.3.29735.20 dlsym: zesInit dlsym: zesDriverGet dlsym: zesDeviceGet dlsym: zesDeviceGetProperties dlsym: zesDeviceEnumMemoryModules dlsym: zesMemoryGetProperties dlsym: zesMemoryGetState calling zesInit calling zesDriverGet oneapi driver count: 1 calling zesDeviceGet count 0: 0x7ca1fd55d6d0 [0:0] oneAPI device name: Intel(R) Arc(TM) A380 Graphics [0:0] oneAPI brand: unknown [0:0] oneAPI vendor: Intel(R) Corporation [0:0] oneAPI S/N: unknown [0:0] oneAPI board number: unknown discovered 1 Level-Zero memory modules time=2024-10-15T15:34:37.691Z level=DEBUG source=amd_linux.go:376 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library releasing oneapi library time=2024-10-15T15:34:37.694Z level=INFO source=types.go:107 msg="inference compute" id=GPU-71904de4-2b69-b701-99e0-5d39fa30f86c library=cuda variant=v12 compute=7.5 driver=12.4 name="NVIDIA GeForce GTX 1660 Ti" total="5.8 GiB" available="3.7 GiB" time=2024-10-15T15:34:37.694Z level=INFO source=types.go:107 msg="inference compute" id=0 library=oneapi variant="" compute="" driver=0.0 name="Intel(R) Arc(TM) A380 Graphics" total="5.9 GiB" available="5.6 GiB" ``` My compose.yaml: ``` version: "3.3" services: ollama: deploy: resources: limits: # Number of CPU threads cpus: "16.00" # Gigabytes of memory memory: 128G reservations: devices: - driver: nvidia count: all capabilities: - gpu devices: - /dev/dri:/dev/dri volumes: - ollama:/root/.ollama - /mnt/nas-hdd0/models:/root/models - /usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ports: - 11434:11434 environment: OLLAMA_ORIGINS: "*" OLLAMA_HOST: 0.0.0.0 OLLAMA_KEEP_ALIVE: 15 OLLAMA_SCHED_SPREAD: 1 OLLAMA_DEBUG: 1 OLLAMA_INTEL_GPU: 1 NEOReadDebugKeys: 1 OverrideGpuAddressSpace: 48 container_name: ollama image: ollama/ollama restart: always volumes: ollama: {} networks: {} ``` So it seemed I simply needed to pass my `/dev/dri` devices and the `/usr/lib/x86_64-linux-gnu` directory through to the Ollama container within Docker as well as enable the environment flags in order for it to discover the GPU. EDIT: In hindsight this seems like a quick and dirty solution. The Docker image might still need updating to include the libraries? (I'm not sure as I'm not experienced with it, just giving my inputs 😄) I'm leaving my solution here in case of random Google searches.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4578
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?