github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6673] Ollama-rocm on Kubernetes with shared AMD GPU seems to have problems allocating vram #29960

New Issue
Closed
opened 2026-04-22 09:19:33 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-22 09:19:33 -05:00
Owner
Copy Link

Originally created by @kubax on GitHub (Sep 6, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6673

What is the issue?

Hi,

i'm pretty new to Ollama, and recently replaced my RX580 with a RX7600 to be able to use Ollama in Kubernetes with ROCm.

When i run Ollama on Arch directly with ROCm support everything works great and is realy snappy.

But when i run Ollama through Kubernetes with the AMD Plugin to share GPUs, it seems to not be able to allocate vram.

Here is my Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ollama
  name: ollama
  namespace: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - env:
            - name: OLLAMA_HOST
              value: 0.0.0.0
            - name: OLLAMA_KEEP_ALIVE
              value: 24h
            - name: HSA_OVERRIDE_GFX_VERSION
              value: "10.3.0"
            - name: AMD_SERIALIZE_KERNEL
              value: "3"
            - name: OLLAMA_LLM_LIBRARY
              value: "rocm_v60002"
          # image: ollama/ollama:latest
          image: ollama/ollama:rocm
          # image: bergutman/ollama-rocm
          name: ollama
          ports:
            - containerPort: 11434
              hostPort: 7869
              protocol: TCP
          tty: true
          volumeMounts:
            - mountPath: /code
              name: ollama-claim0
            - mountPath: /root/.ollama
              name: ollama-claim1
          resources:
            limits:
              amd.com/gpu: 1 # requesting a GPU
      restartPolicy: Always
      volumes:
        - name: ollama-claim0
          persistentVolumeClaim:
            claimName: ollama-claim0
        - name: ollama-claim1
          persistentVolumeClaim:
            claimName: ollama-claim1

When Ollama tries to load the modle into vram nvtop does report 0byte vram usage and Ollama crashes with a segfault.

Here are the last lines wich seem to be interessting to me.

time=2024-09-06T10:51:05.040+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.27 MiB
time=2024-09-06T10:51:46.399+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-09-06T10:51:47.553+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
[GIN] 2024/09/06 - 10:51:47 | 500 | 43.035446402s |       127.0.0.1 | POST     "/api/chat"
time=2024-09-06T10:51:52.553+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000619052 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T10:51:52.803+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250422078 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T10:51:53.054+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.500667507 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
[GIN] 2024/09/06 - 11:13:11 | 200 |      46.378µs |       127.0.0.1 | GET      "/api/version"

Is there something i can try to fix this?

OS

Linux, Docker

GPU

AMD

CPU

AMD

Ollama version

0.3.9

Originally created by @kubax on GitHub (Sep 6, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6673 ### What is the issue? Hi, i'm pretty new to Ollama, and recently replaced my RX580 with a RX7600 to be able to use Ollama in Kubernetes with ROCm. When i run Ollama on Arch directly with ROCm support everything works great and is realy snappy. But when i run Ollama through Kubernetes with the AMD Plugin to share GPUs, it seems to not be able to allocate vram. Here is my Kubernetes deployment: ``` apiVersion: apps/v1 kind: Deployment metadata: labels: app: ollama name: ollama namespace: ollama spec: replicas: 1 selector: matchLabels: app: ollama strategy: type: Recreate template: metadata: labels: app: ollama spec: containers: - env: - name: OLLAMA_HOST value: 0.0.0.0 - name: OLLAMA_KEEP_ALIVE value: 24h - name: HSA_OVERRIDE_GFX_VERSION value: "10.3.0" - name: AMD_SERIALIZE_KERNEL value: "3" - name: OLLAMA_LLM_LIBRARY value: "rocm_v60002" # image: ollama/ollama:latest image: ollama/ollama:rocm # image: bergutman/ollama-rocm name: ollama ports: - containerPort: 11434 hostPort: 7869 protocol: TCP tty: true volumeMounts: - mountPath: /code name: ollama-claim0 - mountPath: /root/.ollama name: ollama-claim1 resources: limits: amd.com/gpu: 1 # requesting a GPU restartPolicy: Always volumes: - name: ollama-claim0 persistentVolumeClaim: claimName: ollama-claim0 - name: ollama-claim1 persistentVolumeClaim: claimName: ollama-claim1 ``` When Ollama tries to load the modle into vram nvtop does report 0byte vram usage and Ollama crashes with a segfault. Here are the last lines wich seem to be interessting to me. ``` time=2024-09-06T10:51:05.040+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.27 MiB time=2024-09-06T10:51:46.399+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2024-09-06T10:51:47.553+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)" [GIN] 2024/09/06 - 10:51:47 | 500 | 43.035446402s | 127.0.0.1 | POST "/api/chat" time=2024-09-06T10:51:52.553+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000619052 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T10:51:52.803+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250422078 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T10:51:53.054+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.500667507 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 [GIN] 2024/09/06 - 11:13:11 | 200 | 46.378µs | 127.0.0.1 | GET "/api/version" ``` Is there something i can try to fix this? ### OS Linux, Docker ### GPU AMD ### CPU AMD ### Ollama version 0.3.9
GiteaMirror added the bug label 2026-04-22 09:19:33 -05:00
GiteaMirror commented 2026-04-22 09:19:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 6, 2024):

If you add

            - name: OLLAMA_DEBUG
              value: 1

to your .env stanza it will add some info from the runner about what it's doing. There's also logging done earlier about device detection and memory calculations, it would be good if you could add the entire log.

<!-- gh-comment-id:2333676349 --> @rick-github commented on GitHub (Sep 6, 2024): If you add ``` - name: OLLAMA_DEBUG value: 1 ``` to your `.env` stanza it will add some info from the runner about what it's doing. There's also logging done earlier about device detection and memory calculations, it would be good if you could add the entire log.
GiteaMirror commented 2026-04-22 09:19:37 -05:00
Author
Owner
Copy Link

@kubax commented on GitHub (Sep 6, 2024):

Here is the log with debugging enabled:

2024/09/06 11:43:25 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:10.3.0 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_LLM_LIBRARY:rocm_v60002 OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-09-06T11:43:25.732+02:00 level=INFO source=images.go:753 msg="total blobs: 16"
time=2024-09-06T11:43:25.740+02:00 level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-09-06T11:43:25.740+02:00 level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.9)"
time=2024-09-06T11:43:25.741+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2302962439/runners
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/ollama_llama_server.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libggml.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libllama.so.gz
time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/ollama_llama_server.gz
time=2024-09-06T11:43:34.069+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server
time=2024-09-06T11:43:34.070+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]"
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-09-06T11:43:34.070+02:00 level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
time=2024-09-06T11:43:34.075+02:00 level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-09-06T11:43:34.075+02:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-09-06T11:43:34.076+02:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /opt/rocm/lib/libcuda.so* /usr/local/lib/libcuda.so* /opt/rh/devtoolset-7/root/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so*
time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /opt/rocm/lib/libcudart.so* /usr/local/lib/libcudart.so* /opt/rh/devtoolset-7/root/libcudart.so* /tmp/ollama2302962439/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2024-09-06T11:43:34.092+02:00 level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:102 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:127 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:102 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:217 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29824 unique_id=0
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:251 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:283 msg="amdgpu memory" gpu=0 total="16.0 GiB"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:284 msg="amdgpu memory" gpu=0 available="15.7 GiB"
time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama"
time=2024-09-06T11:43:34.093+02:00 level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama"
time=2024-09-06T11:43:34.093+02:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-09-06T11:43:34.093+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1102 driver=0.0 name=1002:7480 total="16.0 GiB" available="15.7 GiB"
[GIN] 2024/09/06 - 11:44:21 | 200 |      24.346µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/09/06 - 11:44:21 | 200 |   34.878325ms |       127.0.0.1 | POST     "/api/show"
time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x816740 gpu_count=1
time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]"
time=2024-09-06T11:44:21.734+02:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 gpu=0 parallel=4 available=16814800896 required="8.7 GiB"
time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=server.go:101 msg="system memory" total="29.8 GiB" free="17.0 GiB" free_swap="13.6 GiB"
time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]"
time=2024-09-06T11:44:21.734+02:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.7 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.4 GiB" memory.weights.repeating="7.3 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="681.0 MiB"
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server
time=2024-09-06T11:44:21.735+02:00 level=INFO source=server.go:172 msg="Invalid OLLAMA_LLM_LIBRARY rocm_v60002 - not found"
time=2024-09-06T11:44:21.736+02:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207"
time=2024-09-06T11:44:21.736+02:00 level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/opt/rh/devtoolset-7/root/usr/bin:/opt/rocm/hcc/bin:/opt/rocm/hip/bin:/opt/rocm/bin:/opt/rocm/hcc/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama:/tmp/ollama2302962439/runners/rocm_v60102:/opt/rocm/lib:/usr/local/lib:/opt/rh/devtoolset-7/root HSA_OVERRIDE_GFX_VERSION=10.3.0 HIP_VISIBLE_DEVICES=0]"
time=2024-09-06T11:44:21.760+02:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2024-09-06T11:44:21.760+02:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2024-09-06T11:44:21.760+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="1e6f655" tid="123594537734976" timestamp=1725615863
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123594537734976" timestamp=1725615863 total_threads=16
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="36207" tid="123594537734976" timestamp=1725615863
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 3
llm_load_vocab: token to piece cache size = 0.1684 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) 
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_print_meta: max token length = 48
time=2024-09-06T11:44:24.016+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.27 MiB
time=2024-09-06T11:45:11.360+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-09-06T11:45:12.512+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
[GIN] 2024/09/06 - 11:45:12 | 500 | 50.796323994s |       127.0.0.1 | POST     "/api/chat"
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=server.go:1048 msg="stopping llama server"
time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:12.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:12.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:13.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:13.013+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:13.264+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:13.264+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:13.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:13.513+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:13.764+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:13.764+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:14.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:14.013+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:14.264+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:14.264+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:14.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:14.513+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:14.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:14.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:15.014+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.6 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:15.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:15.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.6 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:15.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:15.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.7 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:15.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:15.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.7 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:15.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:16.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:16.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:16.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:16.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:16.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:16.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:16.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:16.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:17.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:17.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:17.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:17.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:17.513+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001331081 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests"
time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:17.763+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250768463 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
time=2024-09-06T11:45:17.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB"
time=2024-09-06T11:45:17.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB"
time=2024-09-06T11:45:18.013+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.501278564 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246
[GIN] 2024/09/06 - 11:46:11 | 200 |   10.286873ms |       10.42.3.0 | GET      "/api/tags"

Hopefully it's ok to use inline code blocks for such long logs :)

<!-- gh-comment-id:2333689528 --> @kubax commented on GitHub (Sep 6, 2024): Here is the log with debugging enabled: ``` 2024/09/06 11:43:25 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:10.3.0 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_LLM_LIBRARY:rocm_v60002 OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-09-06T11:43:25.732+02:00 level=INFO source=images.go:753 msg="total blobs: 16" time=2024-09-06T11:43:25.740+02:00 level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-09-06T11:43:25.740+02:00 level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.9)" time=2024-09-06T11:43:25.741+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2302962439/runners time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/ollama_llama_server.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libggml.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libllama.so.gz time=2024-09-06T11:43:25.741+02:00 level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/ollama_llama_server.gz time=2024-09-06T11:43:34.069+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server time=2024-09-06T11:43:34.070+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]" time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-09-06T11:43:34.070+02:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-09-06T11:43:34.070+02:00 level=INFO source=gpu.go:200 msg="looking for compatible GPUs" time=2024-09-06T11:43:34.075+02:00 level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA" time=2024-09-06T11:43:34.075+02:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so* time=2024-09-06T11:43:34.076+02:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /opt/rocm/lib/libcuda.so* /usr/local/lib/libcuda.so* /opt/rh/devtoolset-7/root/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[] time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so* time=2024-09-06T11:43:34.090+02:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /opt/rocm/lib/libcudart.so* /usr/local/lib/libcudart.so* /opt/rh/devtoolset-7/root/libcudart.so* /tmp/ollama2302962439/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[] time=2024-09-06T11:43:34.092+02:00 level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:102 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:127 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:102 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:217 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29824 unique_id=0 time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:251 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:283 msg="amdgpu memory" gpu=0 total="16.0 GiB" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_linux.go:284 msg="amdgpu memory" gpu=0 available="15.7 GiB" time=2024-09-06T11:43:34.092+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama" time=2024-09-06T11:43:34.093+02:00 level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama" time=2024-09-06T11:43:34.093+02:00 level=INFO source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-09-06T11:43:34.093+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1102 driver=0.0 name=1002:7480 total="16.0 GiB" available="15.7 GiB" [GIN] 2024/09/06 - 11:44:21 | 200 | 24.346µs | 127.0.0.1 | HEAD  "/" [GIN] 2024/09/06 - 11:44:21 | 200 | 34.878325ms | 127.0.0.1 | POST  "/api/show" time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:44:21.722+02:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x816740 gpu_count=1 time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]" time=2024-09-06T11:44:21.734+02:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 gpu=0 parallel=4 available=16814800896 required="8.7 GiB" time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=server.go:101 msg="system memory" total="29.8 GiB" free="17.0 GiB" free_swap="13.6 GiB" time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]" time=2024-09-06T11:44:21.734+02:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.7 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.4 GiB" memory.weights.repeating="7.3 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="681.0 MiB" time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cpu_avx2/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v11/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/cuda_v12/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server time=2024-09-06T11:44:21.735+02:00 level=INFO source=server.go:172 msg="Invalid OLLAMA_LLM_LIBRARY rocm_v60002 - not found" time=2024-09-06T11:44:21.736+02:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207" time=2024-09-06T11:44:21.736+02:00 level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/opt/rh/devtoolset-7/root/usr/bin:/opt/rocm/hcc/bin:/opt/rocm/hip/bin:/opt/rocm/bin:/opt/rocm/hcc/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama:/tmp/ollama2302962439/runners/rocm_v60102:/opt/rocm/lib:/usr/local/lib:/opt/rh/devtoolset-7/root HSA_OVERRIDE_GFX_VERSION=10.3.0 HIP_VISIBLE_DEVICES=0]" time=2024-09-06T11:44:21.760+02:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2024-09-06T11:44:21.760+02:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2024-09-06T11:44:21.760+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="1e6f655" tid="123594537734976" timestamp=1725615863 INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123594537734976" timestamp=1725615863 total_threads=16 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="36207" tid="123594537734976" timestamp=1725615863 llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = LLaMA v2 llama_model_loader: - kv 2: llama.context_length u32 = 4096 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 2 llama_model_loader: - kv 11: tokenizer.ggml.model str = llama llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 15: tokenizer.ggml.merges arr[str,61249] = ["▁ t", "e r", "i n", "▁ a", "e n... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 21: tokenizer.chat_template str = {% if messages[0]['role'] == 'system'... llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_0: 225 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 3 llm_load_vocab: token to piece cache size = 0.1684 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 4096 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 4096 llm_load_print_meta: n_embd_v_gqa = 4096 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 11008 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 6.74 B llm_load_print_meta: model size = 3.56 GiB (4.54 BPW) llm_load_print_meta: general.name = LLaMA v2 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: max token length = 48 time=2024-09-06T11:44:24.016+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.27 MiB time=2024-09-06T11:45:11.360+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2024-09-06T11:45:12.512+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)" time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 [GIN] 2024/09/06 - 11:45:12 | 500 | 50.796323994s | 127.0.0.1 | POST  "/api/chat" time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=server.go:1048 msg="stopping llama server" time=2024-09-06T11:45:12.512+02:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:12.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:12.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:13.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:13.013+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:13.264+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:13.264+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:13.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.8 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:13.513+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:13.764+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.8 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:13.764+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:14.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:14.013+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:14.264+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:14.264+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:14.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:14.513+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:14.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:14.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:15.014+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.6 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:15.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:15.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.6 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:15.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:15.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.7 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:15.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:15.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.7 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:15.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:16.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:16.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:16.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:16.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:16.513+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:16.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:16.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:16.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:17.013+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:17.014+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:17.263+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="16.9 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:17.263+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:17.513+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001331081 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests" time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="16.9 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:17.514+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:17.763+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250768463 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 time=2024-09-06T11:45:17.763+02:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="29.8 GiB" before.free="17.0 GiB" before.free_swap="13.6 GiB" now.total="29.8 GiB" now.free="17.0 GiB" now.free_swap="13.6 GiB" time=2024-09-06T11:45:17.763+02:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:7480 before="15.7 GiB" now="15.7 GiB" time=2024-09-06T11:45:18.013+02:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.501278564 model=/root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 [GIN] 2024/09/06 - 11:46:11 | 200 | 10.286873ms | 10.42.3.0 | GET  "/api/tags" ``` Hopefully it's ok to use inline code blocks for such long logs :)
GiteaMirror commented 2026-04-22 09:19:37 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 6, 2024):

You have an OLLAMA_LLM_LIBRARY override of "rocm_v60002" but that's not a recognized library so ollama falls back to rocm_v60102.

time=2024-09-06T11:44:21.735+02:00 level=INFO source=server.go:172 msg="Invalid OLLAMA_LLM_LIBRARY rocm_v60002 - not found"

ollama detects a GPU with 16G RAM and starts a runner

time=2024-09-06T11:43:34.093+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1102 driver=0.0 name=1002:7480 total="16.0 GiB" available="15.7 GiB"
time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]"
time=2024-09-06T11:44:21.736+02:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207"

The runner detects the GPU and is about to start allocating memory

  Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.27 MiB

and then dies without leaving a trace

time=2024-09-06T11:45:12.512+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"

Usually at that point the runner is capable of catching errors related to memory issues, so that it died without logging anything indicates an unusual failure (duh, segfault).

If you can get a shell in the ollama container, it would be interesting to see what happens if you start the runner manually:

/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207
<!-- gh-comment-id:2333785827 --> @rick-github commented on GitHub (Sep 6, 2024): You have an OLLAMA_LLM_LIBRARY override of "rocm_v60002" but that's not a recognized library so ollama falls back to rocm_v60102. ``` time=2024-09-06T11:44:21.735+02:00 level=INFO source=server.go:172 msg="Invalid OLLAMA_LLM_LIBRARY rocm_v60002 - not found" ``` ollama detects a GPU with 16G RAM and starts a runner ``` time=2024-09-06T11:43:34.093+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1102 driver=0.0 name=1002:7480 total="16.0 GiB" available="15.7 GiB" time=2024-09-06T11:44:21.734+02:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[15.7 GiB]" time=2024-09-06T11:44:21.736+02:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207" ``` The runner detects the GPU and is about to start allocating memory ``` Device 0: AMD Radeon™ RX 7600 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.27 MiB ``` and then dies without leaving a trace ``` time=2024-09-06T11:45:12.512+02:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)" ``` Usually at that point the runner is capable of catching errors related to memory issues, so that it died without logging anything indicates an unusual failure (duh, segfault). If you can get a shell in the ollama container, it would be interesting to see what happens if you start the runner manually: ``` /tmp/ollama2302962439/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 36207 ```
GiteaMirror commented 2026-04-22 09:19:37 -05:00
Author
Owner
Copy Link

@kubax commented on GitHub (Sep 6, 2024):

oh boy... i'm realy sorry!!!

When you mentioned the overrides i went pale white -.-

I added the overrides when i tried to make use of my old RX580, because the GPU was incompatible with ROCm. When i remove those 3 lines:

            - name: HSA_OVERRIDE_GFX_VERSION
              value: "10.3.0"
            - name: AMD_SERIALIZE_KERNEL
              value: "3"
            - name: OLLAMA_LLM_LIBRARY
              value: "rocm_v60002"

everything works as expected...

I apologize immensely for any inconvenience I may have caused.

<!-- gh-comment-id:2333805658 --> @kubax commented on GitHub (Sep 6, 2024): oh boy... i'm realy sorry!!! When you mentioned the overrides i went pale white -.- I added the overrides when i tried to make use of my old RX580, because the GPU was incompatible with ROCm. When i remove those 3 lines: ``` - name: HSA_OVERRIDE_GFX_VERSION value: "10.3.0" - name: AMD_SERIALIZE_KERNEL value: "3" - name: OLLAMA_LLM_LIBRARY value: "rocm_v60002" ``` everything works as expected... I apologize immensely for any inconvenience I may have caused.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#29960
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?