github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #3761] GPU not detected in Kubernetes. #64358

New Issue
Closed
opened 2026-05-03 17:17:02 -05:00 by GiteaMirror · 17 comments
GiteaMirror commented 2026-05-03 17:17:02 -05:00
Owner
Copy Link

Originally created by @dylanbstorey on GitHub (Apr 19, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3761

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

When deploying into kubernetes the container is complaining about being unable to load the cudart library. (Or maybe its out of date)

Based on the documentation and provided examples I expect it to detect and utilize the GPU in container.

Every test I can think of (which is limited) seems to indicate this should be working but I'll bet I'm missing some nuance in the stack here - any advice would be appreciated.

Host Configuration :

uname -a                                                                                                                                                                                               
Linux overseer 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

nvcc

vcc --version                                                                                                                                                                                          
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Docker Run outputs :

docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama ollama/ollama   docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama ollama/ollama   
docker logs ollama 
time=2024-04-19T17:59:48.712Z level=INFO source=images.go:817 msg="total blobs: 0"
time=2024-04-19T17:59:48.712Z level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-19T17:59:48.712Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-04-19T17:59:48.712Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama4206527122/runners
time=2024-04-19T17:59:50.712Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11 rocm_v60002 cpu]"
time=2024-04-19T17:59:50.712Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-19T17:59:50.712Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-04-19T17:59:50.713Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama4206527122/runners/cuda_v11/libcudart.so.11.0]"
time=2024-04-19T17:59:50.746Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-19T17:59:50.746Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-19T17:59:50.855Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
[GIN] 2024/04/19 - 18:00:35 | 404 |     154.523µs |      172.17.0.1 | POST     "/api/generate"

Deployment Configuration:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  runtimeClassName: nvidia
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama
          resources:
            limits:
              nvidia.com/gpu: 1
          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule
          env:
            - name: PATH
              value: /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            - name: LD_LIBRARY_PATH
              value: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: compute,utility
            - name: OLLAMA_DEBUG
              value: "1"
          ports:
            - name: http
              containerPort: 11434
              protocol: TCP

Deployment Logs:

│                                                                                                                           Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:On                                                                                                                            │
│ Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.                                                                                                                                                                                                                                                 │
│ Your new public key is:                                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                                                       │
│ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMp1XONYlspAEBzMEJNATgAMm39ctFUiN3XZxLwlzVMB                                                                                                                                                                                                                                      │
│                                                                                                                                                                                                                                                                                                                       │
│ time=2024-04-19T17:27:12.252Z level=INFO source=images.go:817 msg="total blobs: 0"                                                                                                                                                                                                                                    │
│ time=2024-04-19T17:27:12.252Z level=INFO source=images.go:824 msg="total unused blobs removed: 0"                                                                                                                                                                                                                     │
│ time=2024-04-19T17:27:12.253Z level=INFO source=routes.go:1143 msg="Listening on :11434 (version 0.1.32)"                                                                                                                                                                                                             │
│ time=2024-04-19T17:27:12.253Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama2307533246/runners                                                                                                                                                                                       │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz                                                                                                                                                                     │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz                                                                                                                                                             │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz                                                                                                                                                           │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz                                                                                                                                                               │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz                                                                                                                                                             │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz                                                                                                                                                             │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz                                                                                                                                                           │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz                                                                                                                                                                │
│ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz                                                                                                                                                     │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu                                                                                                                                                                                  │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu_avx                                                                                                                                                                              │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu_avx2                                                                                                                                                                             │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cuda_v11                                                                                                                                                                             │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/rocm_v60002                                                                                                                                                                          │
│ time=2024-04-19T17:27:14.217Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"                                                                                                                                                                                 │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"                                                                                                                                                                                           │
│ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:121 msg="Detecting GPU type"                                                                                                                                                                                                                                   │
│ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"                                                                                                                                                                                                   │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama2307533246/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /u │
│ sr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* / │
│ usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"                                                                                                                                                                                                                                          │
│ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0]"                                                                                                                                                                 │
│ wiring cudart library functions in /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0                                                                                                                                                                                                                           │
│ dlsym: cudaSetDevice                                                                                                                                                                                                                                                                                                  │
│ dlsym: cudaDeviceSynchronize                                                                                                                                                                                                                                                                                          │
│ dlsym: cudaDeviceReset                                                                                                                                                                                                                                                                                                │
│ dlsym: cudaMemGetInfo                                                                                                                                                                                                                                                                                                 │
│ dlsym: cudaGetDeviceCount                                                                                                                                                                                                                                                                                             │
│ dlsym: cudaDeviceGetAttribute                                                                                                                                                                                                                                                                                         │
│ dlsym: cudaDriverGetVersion                                                                                                                                                                                                                                                                                           │
│ cudaSetDevice err: 35                                                                                                                                                                                                                                                                                                 │
│ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0: your nvidia driver is too old or missing, please upgrade to run ollama"                                                                       │
│ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libnvidia-ml.so"                                                                                                                                                                                                 │
│ time=2024-04-19T17:27:14.218Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/usr/local/cuda/lib64/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/libnvidia-ml.so* /usr/lib/wsl/lib/libnvidia-ml.so* /usr/lib/wsl/drivers/*/libnvidia-ml.so*  │
│ /opt/cuda/lib64/libnvidia-ml.so* /usr/lib*/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/libnvidia-ml.so* /usr/local/lib*/libnvidia-ml.so* /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so* /usr/local/nvidia/lib/libnvidia-ml.so* /usr/local/nvidi │
│ a/lib64/libnvidia-ml.so*]"                                                                                                                                                                                                                                                                                            │
│ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []"                                                                                                                                                                                                                         │
│ time=2024-04-19T17:27:14.218Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"                                                                                                                                                                                                                                   │
│ time=2024-04-19T17:27:14.218Z level=DEBUG source=amd_linux.go:280 msg="amdgpu driver not detected /sys/module/amdgpu"                                                                                                                                                                                                 │
│ time=2024-04-19T17:27:14.218Z level=INFO source=routes.go:1164 msg="no GPU detected"

Kubernetes Based nbody run :

 cat <<EOF | kubectl create -f -                                                                                                                                                                                                                                                                                   
apiVersion: v1              
kind: Pod
metadata:
  name: nbody-gpu-benchmark
  namespace: default
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-container
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    resources:
      limits:
        nvidia.com/gpu: 1
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all
EOF

nbody container logs

                                                                                                                           Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                                                                                                           │
│ Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.                                                                                                                                                                                                                                               │
│     -fullscreen       (run n-body simulation in fullscreen mode)                                                                                                                                                                                                                                                      │
│     -fp64             (use double precision floating point values for simulation)                                                                                                                                                                                                                                     │
│     -hostmem          (stores simulation data in host memory)                                                                                                                                                                                                                                                         │
│     -benchmark        (run benchmark to measure performance)                                                                                                                                                                                                                                                          │
│     -numbodies=<N>    (number of bodies (>= 1) to run in simulation)                                                                                                                                                                                                                                                  │
│     -device=<d>       (where d=0,1,2.... for the CUDA device to use)                                                                                                                                                                                                                                                  │
│     -numdevices=<i>   (where i=(number of CUDA devices > 0) to use for simulation)                                                                                                                                                                                                                                    │
│     -compare          (compares simulation results running once on the default GPU and once on the CPU)                                                                                                                                                                                                               │
│     -cpu              (run n-body simulation on the CPU)                                                                                                                                                                                                                                                              │
│     -tipsy=<file.bin> (load a tipsy model file for simulation)                                                                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                                                       │
│ NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                                                       │
│ > Windowed mode                                                                                                                                                                                                                                                                                                       │
│ > Simulation data stored in video memory                                                                                                                                                                                                                                                                              │
│ > Single precision floating point simulation                                                                                                                                                                                                                                                                          │
│ > 1 Devices used for simulation                                                                                                                                                                                                                                                                                       │
│ GPU Device 0: "Ampere" with compute capability 8.6                                                                                                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                                                       │
│ > Compute 8.6 CUDA device: [NVIDIA GeForce RTX 3060]                                                                                                                                                                                                                                                                  │
│ 28672 bodies, total time for 10 iterations: 22.067 ms                                                                                                                                                                                                                                                                 │
│ = 372.538 billion interactions per second                                                                                                                                                                                                                                                                             │
│ = 7450.761 single-precision GFLOP/s at 20 flops per interaction

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

latest

Originally created by @dylanbstorey on GitHub (Apr 19, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3761 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? When deploying into kubernetes the container is complaining about being unable to load the cudart library. (Or maybe its out of date) Based on the documentation and provided examples I expect it to detect and utilize the GPU in container. Every test I can think of (which is limited) seems to indicate this should be working but I'll bet I'm missing some nuance in the stack here - any advice would be appreciated. Host Configuration : ```bash uname -a Linux overseer 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux ``` nvcc ```bash vcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Mar_28_02:18:24_PDT_2024 Cuda compilation tools, release 12.4, V12.4.131 Build cuda_12.4.r12.4/compiler.34097967_0 ``` Docker Run outputs : ```bash docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama ollama/ollama docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama ollama/ollama docker logs ollama time=2024-04-19T17:59:48.712Z level=INFO source=images.go:817 msg="total blobs: 0" time=2024-04-19T17:59:48.712Z level=INFO source=images.go:824 msg="total unused blobs removed: 0" time=2024-04-19T17:59:48.712Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)" time=2024-04-19T17:59:48.712Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama4206527122/runners time=2024-04-19T17:59:50.712Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11 rocm_v60002 cpu]" time=2024-04-19T17:59:50.712Z level=INFO source=gpu.go:121 msg="Detecting GPU type" time=2024-04-19T17:59:50.712Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" time=2024-04-19T17:59:50.713Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama4206527122/runners/cuda_v11/libcudart.so.11.0]" time=2024-04-19T17:59:50.746Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart" time=2024-04-19T17:59:50.746Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-04-19T17:59:50.855Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6" [GIN] 2024/04/19 - 18:00:35 | 404 | 154.523µs | 172.17.0.1 | POST "/api/generate" ``` Deployment Configuration: ```bash --- apiVersion: apps/v1 kind: Deployment metadata: name: ollama namespace: ollama spec: runtimeClassName: nvidia selector: matchLabels: name: ollama template: metadata: labels: name: ollama spec: containers: - name: ollama image: ollama/ollama resources: limits: nvidia.com/gpu: 1 tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule env: - name: PATH value: /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - name: LD_LIBRARY_PATH value: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 - name: NVIDIA_VISIBLE_DEVICES value: all - name: NVIDIA_DRIVER_CAPABILITIES value: compute,utility - name: OLLAMA_DEBUG value: "1" ports: - name: http containerPort: 11434 protocol: TCP ``` Deployment Logs: ```bash │ Autoscroll:On FullScreen:Off Timestamps:Off Wrap:On │ │ Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. │ │ Your new public key is: │ │ │ │ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMp1XONYlspAEBzMEJNATgAMm39ctFUiN3XZxLwlzVMB │ │ │ │ time=2024-04-19T17:27:12.252Z level=INFO source=images.go:817 msg="total blobs: 0" │ │ time=2024-04-19T17:27:12.252Z level=INFO source=images.go:824 msg="total unused blobs removed: 0" │ │ time=2024-04-19T17:27:12.253Z level=INFO source=routes.go:1143 msg="Listening on :11434 (version 0.1.32)" │ │ time=2024-04-19T17:27:12.253Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama2307533246/runners │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz │ │ time=2024-04-19T17:27:12.253Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu_avx │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cpu_avx2 │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/cuda_v11 │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama2307533246/runners/rocm_v60002 │ │ time=2024-04-19T17:27:14.217Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]" │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" │ │ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:121 msg="Detecting GPU type" │ │ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama2307533246/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /u │ │ sr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* / │ │ usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" │ │ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0]" │ │ wiring cudart library functions in /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0 │ │ dlsym: cudaSetDevice │ │ dlsym: cudaDeviceSynchronize │ │ dlsym: cudaDeviceReset │ │ dlsym: cudaMemGetInfo │ │ dlsym: cudaGetDeviceCount │ │ dlsym: cudaDeviceGetAttribute │ │ dlsym: cudaDriverGetVersion │ │ cudaSetDevice err: 35 │ │ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0: your nvidia driver is too old or missing, please upgrade to run ollama" │ │ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libnvidia-ml.so" │ │ time=2024-04-19T17:27:14.218Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/usr/local/cuda/lib64/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/libnvidia-ml.so* /usr/lib/wsl/lib/libnvidia-ml.so* /usr/lib/wsl/drivers/*/libnvidia-ml.so* │ │ /opt/cuda/lib64/libnvidia-ml.so* /usr/lib*/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/libnvidia-ml.so* /usr/local/lib*/libnvidia-ml.so* /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so* /usr/local/nvidia/lib/libnvidia-ml.so* /usr/local/nvidi │ │ a/lib64/libnvidia-ml.so*]" │ │ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []" │ │ time=2024-04-19T17:27:14.218Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" │ │ time=2024-04-19T17:27:14.218Z level=DEBUG source=amd_linux.go:280 msg="amdgpu driver not detected /sys/module/amdgpu" │ │ time=2024-04-19T17:27:14.218Z level=INFO source=routes.go:1164 msg="no GPU detected" ``` Kubernetes Based nbody run : ```bash cat <<EOF | kubectl create -f - apiVersion: v1 kind: Pod metadata: name: nbody-gpu-benchmark namespace: default spec: restartPolicy: OnFailure runtimeClassName: nvidia containers: - name: cuda-container image: nvcr.io/nvidia/k8s/cuda-sample:nbody args: ["nbody", "-gpu", "-benchmark"] resources: limits: nvidia.com/gpu: 1 env: - name: NVIDIA_VISIBLE_DEVICES value: all - name: NVIDIA_DRIVER_CAPABILITIES value: all EOF ``` nbody container logs ``` Autoscroll:On FullScreen:Off Timestamps:Off Wrap:Off │ │ Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance. │ │ -fullscreen (run n-body simulation in fullscreen mode) │ │ -fp64 (use double precision floating point values for simulation) │ │ -hostmem (stores simulation data in host memory) │ │ -benchmark (run benchmark to measure performance) │ │ -numbodies=<N> (number of bodies (>= 1) to run in simulation) │ │ -device=<d> (where d=0,1,2.... for the CUDA device to use) │ │ -numdevices=<i> (where i=(number of CUDA devices > 0) to use for simulation) │ │ -compare (compares simulation results running once on the default GPU and once on the CPU) │ │ -cpu (run n-body simulation on the CPU) │ │ -tipsy=<file.bin> (load a tipsy model file for simulation) │ │ │ │ NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled. │ │ │ │ > Windowed mode │ │ > Simulation data stored in video memory │ │ > Single precision floating point simulation │ │ > 1 Devices used for simulation │ │ GPU Device 0: "Ampere" with compute capability 8.6 │ │ │ │ > Compute 8.6 CUDA device: [NVIDIA GeForce RTX 3060] │ │ 28672 bodies, total time for 10 iterations: 22.067 ms │ │ = 372.538 billion interactions per second │ │ = 7450.761 single-precision GFLOP/s at 20 flops per interaction ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version latest
GiteaMirror added the bugnvidia labels 2026-05-03 17:17:03 -05:00
GiteaMirror commented 2026-05-03 17:17:08 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 19, 2024):

Can you share what driver version you have on the host? nvidia-smi on the host should report it.

<!-- gh-comment-id:2067256370 --> @dhiltgen commented on GitHub (Apr 19, 2024): Can you share what driver version you have on the host? `nvidia-smi` on the host should report it.
GiteaMirror commented 2026-05-03 17:17:11 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (Apr 19, 2024):

Sure thing !

Fri Apr 19 16:41:58 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:0B:00.0  On |                  N/A |
|  0%   45C    P8             20W /  170W |    6167MiB /  12288MiB |     18%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1351      G   /usr/lib/xorg/Xorg                            420MiB |
|    0   N/A  N/A      1710      G   /usr/bin/gnome-shell                           50MiB |
|    0   N/A  N/A      5747      G   ...ures=SpareRendererForSitePerProcess        241MiB |
|    0   N/A  N/A     20562      G   ...yOnDemand --variations-seed-version        117MiB |
|    0   N/A  N/A    127243      G   ...seed-version=20240419-050138.465000        187MiB |
|    0   N/A  N/A    206026      C   ...unners/cuda_v11/ollama_llama_server       5114MiB |
+-----------------------------------------------------------------------------------------+
<!-- gh-comment-id:2067258525 --> @dylanbstorey commented on GitHub (Apr 19, 2024): Sure thing ! ``` Fri Apr 19 16:41:58 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 Off | 00000000:0B:00.0 On | N/A | | 0% 45C P8 20W / 170W | 6167MiB / 12288MiB | 18% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 1351 G /usr/lib/xorg/Xorg 420MiB | | 0 N/A N/A 1710 G /usr/bin/gnome-shell 50MiB | | 0 N/A N/A 5747 G ...ures=SpareRendererForSitePerProcess 241MiB | | 0 N/A N/A 20562 G ...yOnDemand --variations-seed-version 117MiB | | 0 N/A N/A 127243 G ...seed-version=20240419-050138.465000 187MiB | | 0 N/A N/A 206026 C ...unners/cuda_v11/ollama_llama_server 5114MiB | +-----------------------------------------------------------------------------------------+ ```
GiteaMirror commented 2026-05-03 17:17:15 -05:00
Author
Owner
Copy Link

@chrisnurse commented on GitHub (Apr 20, 2024):

I am hosting locally and running ollama with my RTX4090.

I had an issue running the new Llama3 model, so thought I'd update my local image/container.

Since pulling the new version, I get the error shown in the screen clip, that my nvidia driver is too old:
image

Coincidentally, there was an nvidia driver update available, but applying this update has made no difference.
My current nvidia driver version is 552.22 released 16th April 2024.

<!-- gh-comment-id:2067504717 --> @chrisnurse commented on GitHub (Apr 20, 2024): I am hosting locally and running ollama with my RTX4090. I had an issue running the new Llama3 model, so thought I'd update my local image/container. Since pulling the new version, I get the error shown in the screen clip, that my nvidia driver is too old: ![image](https://github.com/ollama/ollama/assets/45703746/ed771aad-4706-47b3-9b53-d0ec4bcd7472) Coincidentally, there was an nvidia driver update available, but applying this update has made no difference. My current nvidia driver version is 552.22 released 16th April 2024.
GiteaMirror commented 2026-05-03 17:17:17 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 22, 2024):

Hmm... I have a very similar test setup and haven't been able to reproduce yet. (Ubuntu 22.04 kernel 5.15.0-105-generic)

% nvidia-smi
Mon Apr 22 23:00:49 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   45C    P8             14W /  170W |    1872MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3060        Off |   00000000:05:00.0 Off |                  N/A |
|  0%   48C    P8             13W /  170W |    1792MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      3159      C   ...unners/cuda_v11/ollama_llama_server       1866MiB |
|    1   N/A  N/A      3159      C   ...unners/cuda_v11/ollama_llama_server       1786MiB |
+-----------------------------------------------------------------------------------------+

% docker run --rm -it --gpus=all -e OLLAMA_DEBUG=1  ollama/ollama
Unable to find image 'ollama/ollama:latest' locally
latest: Pulling from ollama/ollama
bccd10f490ab: Pull complete
50f1619ca0b1: Pull complete
7a344f274044: Pull complete
Digest: sha256:c5018bf71b27a38f50da37d86fa0067105eea488cdcc258ace6d222dde632f75
Status: Downloaded newer image for ollama/ollama:latest
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIC3jXTsRoQI5yH6n0hhiHtRbJoD6gYOd9C81TRfk+Ck

time=2024-04-22T22:59:11.746Z level=INFO source=images.go:817 msg="total blobs: 0"
time=2024-04-22T22:59:11.746Z level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-22T22:59:11.746Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-04-22T22:59:11.746Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama1759042201/runners
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz
time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu_avx
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu_avx2
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cuda_v11
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/rocm_v60002
time=2024-04-22T22:59:13.484Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11 rocm_v60002 cpu cpu_avx]"
time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-04-22T22:59:13.484Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama1759042201/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama1759042201/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama1759042201/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-4
time=2024-04-22T22:59:13.525Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-22T22:59:13.525Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 12622168064
[0] CUDA freeMem 12508921856
[1] CUDA totalMem 12622168064
[1] CUDA freeMem 12508921856
time=2024-04-22T22:59:13.621Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
releasing cudart library
[GIN] 2024/04/22 - 22:59:38 | 200 |      48.977µs |       127.0.0.1 | HEAD     "/"

Exec'ing into the container shows fast token rate (logs also show running on GPU)

% docker exec -it stupefied_bhabha ollama run --verbose orca-mini hello
 Hello there! How can I assist you today?

total duration:       153.15311ms
load duration:        318.736µs
prompt eval duration: 20.014ms
prompt eval rate:     0.00 tokens/s
eval count:           11 token(s)
eval duration:        86.036ms
eval rate:            127.85 tokens/s

Is it possible the nvidia container toolkit is out of rev? If not that, perhaps the kernel version being newer has an impact.

<!-- gh-comment-id:2071101015 --> @dhiltgen commented on GitHub (Apr 22, 2024): Hmm... I have a very similar test setup and haven't been able to reproduce yet. (Ubuntu 22.04 kernel 5.15.0-105-generic) ``` % nvidia-smi Mon Apr 22 23:00:49 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 Off | 00000000:01:00.0 Off | N/A | | 0% 45C P8 14W / 170W | 1872MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3060 Off | 00000000:05:00.0 Off | N/A | | 0% 48C P8 13W / 170W | 1792MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 3159 C ...unners/cuda_v11/ollama_llama_server 1866MiB | | 1 N/A N/A 3159 C ...unners/cuda_v11/ollama_llama_server 1786MiB | +-----------------------------------------------------------------------------------------+ ``` ``` % docker run --rm -it --gpus=all -e OLLAMA_DEBUG=1 ollama/ollama Unable to find image 'ollama/ollama:latest' locally latest: Pulling from ollama/ollama bccd10f490ab: Pull complete 50f1619ca0b1: Pull complete 7a344f274044: Pull complete Digest: sha256:c5018bf71b27a38f50da37d86fa0067105eea488cdcc258ace6d222dde632f75 Status: Downloaded newer image for ollama/ollama:latest Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIC3jXTsRoQI5yH6n0hhiHtRbJoD6gYOd9C81TRfk+Ck time=2024-04-22T22:59:11.746Z level=INFO source=images.go:817 msg="total blobs: 0" time=2024-04-22T22:59:11.746Z level=INFO source=images.go:824 msg="total unused blobs removed: 0" time=2024-04-22T22:59:11.746Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)" time=2024-04-22T22:59:11.746Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama1759042201/runners time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz time=2024-04-22T22:59:11.746Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu_avx time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cpu_avx2 time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/cuda_v11 time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama1759042201/runners/rocm_v60002 time=2024-04-22T22:59:13.484Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11 rocm_v60002 cpu cpu_avx]" time=2024-04-22T22:59:13.484Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:121 msg="Detecting GPU type" time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" time=2024-04-22T22:59:13.484Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama1759042201/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" time=2024-04-22T22:59:13.484Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama1759042201/runners/cuda_v11/libcudart.so.11.0]" wiring cudart library functions in /tmp/ollama1759042201/runners/cuda_v11/libcudart.so.11.0 dlsym: cudaSetDevice dlsym: cudaDeviceSynchronize dlsym: cudaDeviceReset dlsym: cudaMemGetInfo dlsym: cudaGetDeviceCount dlsym: cudaDeviceGetAttribute dlsym: cudaDriverGetVersion CUDA driver version: 12-4 time=2024-04-22T22:59:13.525Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart" time=2024-04-22T22:59:13.525Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" [0] CUDA totalMem 12622168064 [0] CUDA freeMem 12508921856 [1] CUDA totalMem 12622168064 [1] CUDA freeMem 12508921856 time=2024-04-22T22:59:13.621Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6" releasing cudart library [GIN] 2024/04/22 - 22:59:38 | 200 | 48.977µs | 127.0.0.1 | HEAD "/" ``` Exec'ing into the container shows fast token rate (logs also show running on GPU) ``` % docker exec -it stupefied_bhabha ollama run --verbose orca-mini hello Hello there! How can I assist you today? total duration: 153.15311ms load duration: 318.736µs prompt eval duration: 20.014ms prompt eval rate: 0.00 tokens/s eval count: 11 token(s) eval duration: 86.036ms eval rate: 127.85 tokens/s ``` Is it possible the nvidia container toolkit is out of rev? If not that, perhaps the kernel version being newer has an impact.
GiteaMirror commented 2026-05-03 17:17:18 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (Apr 23, 2024):

The docker container is working fine - gpu getting used as expected, I'm only seeing the problem within kubernetes.

The log that I think indicates what is going on is here is at :

│ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"                                                                                                                                                                                                   │
│ time=2024-04-19T17:27:14.217Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama2307533246/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /u │
│ sr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* / │
│ usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"                                                                                                                                                                                                                                          │
│ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0]"                                                                                                                                                                 │
│ wiring cudart library functions in /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0                                                                                                                                                                                                                           │
│ dlsym: cudaSetDevice                                                                                                                                                                                                                                                                                                  │
│ dlsym: cudaDeviceSynchronize                                                                                                                                                                                                                                                                                          │
│ dlsym: cudaDeviceReset                                                                                                                                                                                                                                                                                                │
│ dlsym: cudaMemGetInfo                                                                                                                                                                                                                                                                                                 │
│ dlsym: cudaGetDeviceCount                                                                                                                                                                                                                                                                                             │
│ dlsym: cudaDeviceGetAttribute                                                                                                                                                                                                                                                                                         │
│ dlsym: cudaDriverGetVersion                                                                                                                                                                                                                                                                                           │
│ cudaSetDevice err: 35                                                                                                                                                                                                                                                                                                 │
│ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0: your nvidia driver is too old or missing, please upgrade to run ollama"

What confuses me is that i'm able to load other cuda libraries (i think) when using the nbody program so I know the GPU is available in cluster as part of the run time.

<!-- gh-comment-id:2071173034 --> @dylanbstorey commented on GitHub (Apr 23, 2024): The docker container is working fine - gpu getting used as expected, I'm only seeing the problem within kubernetes. The log that I think indicates what is going on is here is at : ```bash │ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" │ │ time=2024-04-19T17:27:14.217Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama2307533246/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /u │ │ sr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* / │ │ usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" │ │ time=2024-04-19T17:27:14.217Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0]" │ │ wiring cudart library functions in /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0 │ │ dlsym: cudaSetDevice │ │ dlsym: cudaDeviceSynchronize │ │ dlsym: cudaDeviceReset │ │ dlsym: cudaMemGetInfo │ │ dlsym: cudaGetDeviceCount │ │ dlsym: cudaDeviceGetAttribute │ │ dlsym: cudaDriverGetVersion │ │ cudaSetDevice err: 35 │ │ time=2024-04-19T17:27:14.218Z level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library /tmp/ollama2307533246/runners/cuda_v11/libcudart.so.11.0: your nvidia driver is too old or missing, please upgrade to run ollama" ``` What confuses me is that i'm able to load other cuda libraries (i think) when using the nbody program so I know the GPU is available in cluster as part of the run time.
GiteaMirror commented 2026-05-03 17:17:19 -05:00
Author
Owner
Copy Link

@chrisnurse commented on GitHub (Apr 23, 2024):

@dhiltgen quick note of appreciation for the response.
Mine may have been a short term issue whilst all the different moving parts aligned around Llama3. Today everything looks good with latest version of Ollama and AnythingLLM (my current goto environment for learning).

Apologies if this was off topic.

<!-- gh-comment-id:2071276148 --> @chrisnurse commented on GitHub (Apr 23, 2024): @dhiltgen quick note of appreciation for the response. Mine may have been a short term issue whilst all the different moving parts aligned around Llama3. Today everything looks good with latest version of Ollama and AnythingLLM (my current goto environment for learning). Apologies if this was off topic.
GiteaMirror commented 2026-05-03 17:17:21 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 24, 2024):

I wonder if this might be a dup of #1500 and the bundled cudart library we bake into the image doesn't support MIG?

<!-- gh-comment-id:2075510073 --> @dhiltgen commented on GitHub (Apr 24, 2024): I wonder if this might be a dup of #1500 and the bundled cudart library we bake into the image doesn't support MIG?
GiteaMirror commented 2026-05-03 17:17:22 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (Apr 24, 2024):

Oh good eye, I'll pull that PR and see if that fixes the issue and then patiently wait for the merge to come through if so.

<!-- gh-comment-id:2075782262 --> @dylanbstorey commented on GitHub (Apr 24, 2024): Oh good eye, I'll pull that PR and see if that fixes the issue and then patiently wait for the merge to come through if so.
GiteaMirror commented 2026-05-03 17:17:25 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (Apr 24, 2024):

This doesn't appear to be related. 😞 I'll keep looking - if i get an answer i'll make sure to post it. Thanks for providing the support.

Maybe useful - i ran nvidia-smi within the container and got an NVML: Unknown error.

<!-- gh-comment-id:2075957809 --> @dylanbstorey commented on GitHub (Apr 24, 2024): This doesn't appear to be related. :disappointed: I'll keep looking - if i get an answer i'll make sure to post it. Thanks for providing the support. Maybe useful - i ran `nvidia-smi` within the container and got an NVML: Unknown error.
GiteaMirror commented 2026-05-03 17:17:26 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 1, 2024):

I've been working on a change to how we do GPU discovery that might resolve this issue. I've pushed test images up to Docker Hub for folks to try out.

dhiltgen/ollama:0.1.33-rc5-24-g089daae
dhiltgen/ollama:0.1.33-rc5-24-g089daae-rocm
<!-- gh-comment-id:2088697294 --> @dhiltgen commented on GitHub (May 1, 2024): I've been working on a [change](https://github.com/ollama/ollama/pull/4067) to how we do GPU discovery that might resolve this issue. I've pushed test images up to Docker Hub for folks to try out. ``` dhiltgen/ollama:0.1.33-rc5-24-g089daae dhiltgen/ollama:0.1.33-rc5-24-g089daae-rocm ```
GiteaMirror commented 2026-05-03 17:17:30 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (May 2, 2024):

Still getting a 35 error.

<!-- gh-comment-id:2089376721 --> @dylanbstorey commented on GitHub (May 2, 2024): Still getting a 35 error.
GiteaMirror commented 2026-05-03 17:17:35 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 2, 2024):

@dylanbstorey can you share the server log with -e OLLAMA_DEBUG=1 set so I can see which API is getting that error?

<!-- gh-comment-id:2090943752 --> @dhiltgen commented on GitHub (May 2, 2024): @dylanbstorey can you share the server log with `-e OLLAMA_DEBUG=1` set so I can see which API is getting that error?
GiteaMirror commented 2026-05-03 17:17:40 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (May 3, 2024):

│ Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.                                                                                                                   │
│ Your new public key is:                                                                                                                                                                 │
│                                                                                                                                                                                         │
│ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGToQkRNU4xkWVE/tqEac0rtLgjiOPiF58sO7iSlbYJK                                                                                                        │
│                                                                                                                                                                                         │
│ time=2024-05-02T01:16:39.059Z level=INFO source=images.go:828 msg="total blobs: 0"                                                                                                      │
│ time=2024-05-02T01:16:39.062Z level=INFO source=images.go:835 msg="total unused blobs removed: 0"                                                                                       │
│ time=2024-05-02T01:16:39.062Z level=INFO source=routes.go:1074 msg="Listening on :11434 (version 0.1.33-rc5-24-g089daae)"                                                               │
│ time=2024-05-02T01:16:39.063Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3280993898/runners                                                         │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz                                       │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz                               │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz                             │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz                                 │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz                               │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz                               │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz                             │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz                                  │
│ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz                       │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu                                                    │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu_avx                                                │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu_avx2                                               │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cuda_v11                                               │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/rocm_v60002                                            │
│ time=2024-05-02T01:16:41.203Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"                                                   │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"                                                             │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"                                                                                              │
│ time=2024-05-02T01:16:41.203Z level=INFO source=gpu.go:121 msg="Detecting GPUs"                                                                                                         │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:247 msg="Searching for GPU library" name=libcuda.so*                                                                            │
│ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/t │
│ argets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib │
│ */libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"                                                                                      │
│ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[]                                                                                     │
│ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:247 msg="Searching for GPU library" name=libcudart.so*                                                                          │
│ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so** /tmp/ollama328 │
│ 0993898/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib │
│ /libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/li │
│ bcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"                                         │
│ time=2024-05-02T01:16:41.207Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[/tmp/ollama3280993898/runners/cuda_v11/libcudart.so.11.0]                             │
│ cudaSetDevice err: 35                                                                                                                                                                   │
│ time=2024-05-02T01:16:41.211Z level=DEBUG source=gpu.go:306 msg="Unable to load cudart" library=/tmp/ollama3280993898/runners/cuda_v11/libcudart.so.11.0 error="your nvidia driver is t │
│ oo old or missing.  If you have a CUDA GPU please upgrade to run ollama"                                                                                                                │
│ time=2024-05-02T01:16:41.211Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"                                                                                                     │
│ time=2024-05-02T01:16:41.211Z level=DEBUG source=amd_linux.go:297 msg="amdgpu driver not detected /sys/module/amdgpu"                                                                   │
│
<!-- gh-comment-id:2092838516 --> @dylanbstorey commented on GitHub (May 3, 2024): ```bash │ Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. │ │ Your new public key is: │ │ │ │ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGToQkRNU4xkWVE/tqEac0rtLgjiOPiF58sO7iSlbYJK │ │ │ │ time=2024-05-02T01:16:39.059Z level=INFO source=images.go:828 msg="total blobs: 0" │ │ time=2024-05-02T01:16:39.062Z level=INFO source=images.go:835 msg="total unused blobs removed: 0" │ │ time=2024-05-02T01:16:39.062Z level=INFO source=routes.go:1074 msg="Listening on :11434 (version 0.1.33-rc5-24-g089daae)" │ │ time=2024-05-02T01:16:39.063Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3280993898/runners │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz │ │ time=2024-05-02T01:16:39.063Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu_avx │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cpu_avx2 │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/cuda_v11 │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3280993898/runners/rocm_v60002 │ │ time=2024-05-02T01:16:41.203Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]" │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" │ │ time=2024-05-02T01:16:41.203Z level=INFO source=gpu.go:121 msg="Detecting GPUs" │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:247 msg="Searching for GPU library" name=libcuda.so* │ │ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/t │ │ argets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib │ │ */libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" │ │ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[] │ │ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:247 msg="Searching for GPU library" name=libcudart.so* │ │ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so** /tmp/ollama328 │ │ 0993898/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib │ │ /libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/li │ │ bcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" │ │ time=2024-05-02T01:16:41.207Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[/tmp/ollama3280993898/runners/cuda_v11/libcudart.so.11.0] │ │ cudaSetDevice err: 35 │ │ time=2024-05-02T01:16:41.211Z level=DEBUG source=gpu.go:306 msg="Unable to load cudart" library=/tmp/ollama3280993898/runners/cuda_v11/libcudart.so.11.0 error="your nvidia driver is t │ │ oo old or missing. If you have a CUDA GPU please upgrade to run ollama" │ │ time=2024-05-02T01:16:41.211Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" │ │ time=2024-05-02T01:16:41.211Z level=DEBUG source=amd_linux.go:297 msg="amdgpu driver not detected /sys/module/amdgpu" │ │ ```
GiteaMirror commented 2026-05-03 17:17:44 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 3, 2024):

│ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/t │
│ argets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib │
│ */libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"                                                                                      │
│ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[]

This indicates we're not finding a Driver API library but falling back to the runtime API. Perhaps I don't have enough search paths wired up for all the different container runtime permutations. (On my test system with local Docker and the nvidia container toolkit it finds the library) Can you try exec'ing into the pod and running a find / -name libcuda.so\* and see if it's located someplace else?

<!-- gh-comment-id:2093389504 --> @dhiltgen commented on GitHub (May 3, 2024): ``` │ time=2024-05-02T01:16:41.203Z level=DEBUG source=gpu.go:266 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/t │ │ argets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib │ │ */libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" │ │ time=2024-05-02T01:16:41.205Z level=DEBUG source=gpu.go:294 msg="discovered GPU libraries" paths=[] ``` This indicates we're not finding a Driver API library but falling back to the runtime API. Perhaps I don't have enough search paths wired up for all the different container runtime permutations. (On my test system with local Docker and the nvidia container toolkit it finds the library) Can you try exec'ing into the pod and running a `find / -name libcuda.so\*` and see if it's located someplace else?
GiteaMirror commented 2026-05-03 17:17:48 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (May 8, 2024):

giphy

Ok - somewhere along the line (I'll have to dig through git blame to figure out when), I removed the runtimeClass setting on the pod... now that I've set it correctly to nvidia this container is working well for me.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  runtimeClassName: nvidia
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      runtimeClassName: nvidia
      containers:
        - name: ollama
          image: dhiltgen/ollama:0.1.33-rc5-24-g089daae
          resources:
            limits:
              nvidia.com/gpu: 1
          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule
          env:
            - name: PATH
              value: /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local
            - name: LD_LIBRARY_PATH
              value: /usr/local/nvidia/lib:/usr/local/nvidia/lib64/:/usr/local/
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: compute,utility
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
            - name: OLLAMA_DEBUG
              value: "1"
          ports:
            - name: http
              containerPort: 11434
              protocol: TCP

With this container it is mounting correctly now and working as intended. I'll try to find some time to go back and see if I can figure out if other builds were also working I was just being a dolt about the pod definition.

I want to just say thank you for your patience and work on this project.

<!-- gh-comment-id:2100471738 --> @dylanbstorey commented on GitHub (May 8, 2024): ![giphy](https://github.com/ollama/ollama/assets/6005970/e1d1be81-60dc-44d3-bd00-048aae15ecbb) Ok - somewhere along the line (I'll have to dig through git blame to figure out when), I removed the runtimeClass setting on the pod... now that I've set it correctly to `nvidia` this container is working well for me. ```bash --- apiVersion: apps/v1 kind: Deployment metadata: name: ollama namespace: ollama spec: runtimeClassName: nvidia selector: matchLabels: name: ollama template: metadata: labels: name: ollama spec: runtimeClassName: nvidia containers: - name: ollama image: dhiltgen/ollama:0.1.33-rc5-24-g089daae resources: limits: nvidia.com/gpu: 1 tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule env: - name: PATH value: /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local - name: LD_LIBRARY_PATH value: /usr/local/nvidia/lib:/usr/local/nvidia/lib64/:/usr/local/ - name: NVIDIA_DRIVER_CAPABILITIES value: compute,utility - name: NVIDIA_VISIBLE_DEVICES value: all - name: OLLAMA_DEBUG value: "1" ports: - name: http containerPort: 11434 protocol: TCP ``` With this container it is mounting correctly now and working as intended. I'll try to find some time to go back and see if I can figure out if other builds were also working I was just being a dolt about the pod definition. I want to just say thank you for your patience and work on this project.
GiteaMirror commented 2026-05-03 17:17:50 -05:00
Author
Owner
Copy Link

@dims commented on GitHub (May 8, 2024):

glad you found it :)

<!-- gh-comment-id:2100477441 --> @dims commented on GitHub (May 8, 2024): glad you found it :)
GiteaMirror commented 2026-05-03 17:17:52 -05:00
Author
Owner
Copy Link

@dylanbstorey commented on GitHub (May 10, 2024):

confirmed working fine with 0.1.34 image now. Assuming its fine with 0.1.33 as well. Thanks again.

<!-- gh-comment-id:2104500781 --> @dylanbstorey commented on GitHub (May 10, 2024): confirmed working fine with 0.1.34 image now. Assuming its fine with 0.1.33 as well. Thanks again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#64358
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?