github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #6802] Potential Bug in AMD GPU Detection leading to rocBLAS Initialization Failure #4292

New Issue
Closed
opened 2026-04-12 15:13:02 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 15:13:02 -05:00
Owner
Copy Link

Originally created by @Bihan on GitHub (Sep 14, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6802

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

This issue happens randomly—sometimes the AMD GPU detection works, but at other times, it fails, resulting in the following error

rocBLAS error: Could not initialize Tensile host: No devices found.

When the issue occurs, it seems that certain GPU nodes in the /sys/class/kfd/kfd/topology/nodes/ directory may not be accessible due to permission errors. When these nodes are inaccessible, it might be leading to incorrect GPU indexing.

The issue might be related to the AMDGetGPUInfo() function. This function skips nodes if permission errors occur when reading the properties file see here, which may cause some GPU nodes to be missed and result in incorrect GPU indexing.

Below are the output samples for when the GPU detection does not work (non-working case) and when it works as expected (working case).

Non-Working Output (Failure Case):

Here’s the output when the rocBLAS error occurs:

time=2024-09-09T08:39:19.349Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
...
time=2024-09-09T08:39:19.362Z level=INFO source=amd_linux.go:345 msg="amdgpu is supported" gpu=3 gpu_type=gfx942
...
time=2024-09-09T08:39:19.362Z level=INFO source=types.go:107 msg="inference compute" id=3 library=rocm variant="" compute=gfx942 driver=6.7 name=1002:74a1 total="192.0 GiB" available="191.7 GiB"

Notice, gpu=3 gpu_type=gfx942

Working Output (Success Case):

Here’s the output when serving works:

2024-09-09T13:55:25.972961384Z time=2024-09-09T13:55:25.972Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
2024-09-09T13:55:25.979418720Z time=2024-09-09T13:55:25.979Z level=INFO source=amd_linux.go:345 msg="amdgpu is supported" gpu=0 gpu_type=gfx942
...
...
2024-09-09T13:55:25.979551298Z time=2024-09-09T13:55:25.979Z level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx942 driver=6.7 name=1002:74a1 total="192.0 GiB" available="191.7 GiB"

Notice, gpu=0 gpu_type=gfx942

Important information about AMD GPU Drivers:

The AMD GPU driver always exposes the host CPU(s) first, but HIP libraries expects zero is the first GPU, not the CPU

Steps To Reproduce:

  1. Run a Docker container using ollama/ollama:rocm on a machine with a single MI300X
  2. Inside the container, run ollama run llama3.1:70B

Related Discussion:See

OS

Linux

GPU

AMD

CPU

AMD

Ollama version

0.3.9

Originally created by @Bihan on GitHub (Sep 14, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6802 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? This issue happens **randomly**—sometimes the AMD GPU detection works, but at other times, it fails, resulting in the following error `rocBLAS error: Could not initialize Tensile host: No devices found.` When the issue occurs, it seems that certain GPU nodes in the `/sys/class/kfd/kfd/topology/nodes/` directory may not be accessible due to `permission errors.` When these nodes are inaccessible, it might be leading to incorrect GPU indexing. The issue might be related to the [AMDGetGPUInfo()](https://github.com/ollama/ollama/blob/56b9af336a049dae37f7cc62246121ac9d18576e/gpu/amd_linux.go#L84) function. This function skips nodes if permission errors occur when reading the properties file see [here](https://github.com/ollama/ollama/blob/56b9af336a049dae37f7cc62246121ac9d18576e/gpu/amd_linux.go#L104), which may cause some GPU nodes to be missed and result in incorrect GPU indexing. Below are the output samples for when the GPU detection does not work (non-working case) and when it works as expected (working case). **Non-Working Output (Failure Case):** Here’s the output when the rocBLAS error occurs: ``` time=2024-09-09T08:39:19.349Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs" ... time=2024-09-09T08:39:19.362Z level=INFO source=amd_linux.go:345 msg="amdgpu is supported" gpu=3 gpu_type=gfx942 ... time=2024-09-09T08:39:19.362Z level=INFO source=types.go:107 msg="inference compute" id=3 library=rocm variant="" compute=gfx942 driver=6.7 name=1002:74a1 total="192.0 GiB" available="191.7 GiB" ``` **Notice**, `gpu=3 gpu_type=gfx942` **Working Output (Success Case):** Here’s the output when serving works: ``` 2024-09-09T13:55:25.972961384Z time=2024-09-09T13:55:25.972Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs" 2024-09-09T13:55:25.979418720Z time=2024-09-09T13:55:25.979Z level=INFO source=amd_linux.go:345 msg="amdgpu is supported" gpu=0 gpu_type=gfx942 ... ... 2024-09-09T13:55:25.979551298Z time=2024-09-09T13:55:25.979Z level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx942 driver=6.7 name=1002:74a1 total="192.0 GiB" available="191.7 GiB" ``` **Notice**, `gpu=0 gpu_type=gfx942` **Important information about AMD GPU Drivers:** The AMD GPU driver always exposes the host CPU(s) first, but HIP libraries expects zero is the first GPU, not the CPU **Steps To Reproduce:** 1. Run a Docker container using `ollama/ollama:rocm` on a machine with a single `MI300X` 2. Inside the container, run ollama run llama3.1:70B Related Discussion:[See](https://github.com/ollama/ollama/issues/6423) ### OS Linux ### GPU AMD ### CPU AMD ### Ollama version 0.3.9
GiteaMirror added the bugamd labels 2026-04-12 15:13:02 -05:00
GiteaMirror commented 2026-04-12 15:13:03 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Sep 25, 2024):

It's odd that permissions are changing on the fly intermittently.

How reproducible is this in your setup? Are there any correlations to load on the system, other workload running, etc, that might help explain the dynamic behavior?

Can you try running with -e OLLAMA_DEBUG=1 set so we can get some more verbose logging during GPU discovery?

<!-- gh-comment-id:2375281586 --> @dhiltgen commented on GitHub (Sep 25, 2024): It's odd that permissions are changing on the fly intermittently. How reproducible is this in your setup? Are there any correlations to load on the system, other workload running, etc, that might help explain the dynamic behavior? Can you try running with `-e OLLAMA_DEBUG=1` set so we can get some more verbose logging during GPU discovery?
GiteaMirror commented 2026-04-12 15:13:03 -05:00
Author
Owner
Copy Link

@blackandcold commented on GitHub (Oct 25, 2024):

I have the same error but not in Docker.
EndevourOS, 5900X, RX 6800

It detects memory, card and so on well but then suddenly fails with

rocBLAS error: Could not initialize Tensile host: No devices found

rocBLAS is installed, rocm to, I also downloaded and extractet the additional archive (idk why this is in the docs... I mean this https://github.com/ollama/ollama/blob/main/docs/linux.md#amd-gpu-install ) and it worked fine months ago with just running the ollama install script.

rocm-smi --showdriverversion --showproductname --showhw

============================ ROCm System Management Interface ============================
================================= Concise Hardware Info ==================================
GPU DID DREV GFX RAS SDMA RAS UMC RAS VBIOS BUS
0 73bf c3 N/A N/A N/A 115-D414S00-101 0000:0C:00.0

============================== Version of System Component ===============================
Driver version: 6.11.5-arch1-1

====================================== Product Info ======================================
GPU[0] : Card series: Navi 21 [Radeon RX 6800/6800 XT / 6900 XT]
GPU[0] : Card model: 0x04f6
GPU[0] : Card vendor: Advanced Micro Devices, Inc. [AMD/ATI]
GPU[0] : Card SKU: D414S00

================================== End of ROCm SMI Log ===================================

Really no clue here.

OLLAMA_DEBUG=1 ollama serve

2024/10/25 22:20:08 routes.go:1158: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/bac/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-10-25T22:20:08.823+02:00 level=INFO source=images.go:754 msg="total blobs: 22"
time=2024-10-25T22:20:08.824+02:00 level=INFO source=images.go:761 msg="total unused blobs removed: 0"
time=2024-10-25T22:20:08.824+02:00 level=INFO source=routes.go:1205 msg="Listening on 127.0.0.1:11434 (version 0.3.14)"
time=2024-10-25T22:20:08.824+02:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama1123413816/runners
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libggml.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libllama.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/ollama_llama_server.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libggml.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libllama.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/ollama_llama_server.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libggml.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libllama.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/ollama_llama_server.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libggml.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libllama.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/ollama_llama_server.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libggml.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libllama.so.gz
time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/ollama_llama_server.gz
time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libggml.so.gz
time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libllama.so.gz
time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/ollama_llama_server.gz
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server
time=2024-10-25T22:20:18.953+02:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102 cpu]"
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-10-25T22:20:18.953+02:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:94 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=libcuda.so
time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /home/bac/libcuda.so* /usr/local/cuda*/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-10-25T22:20:18.977+02:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths="[/usr/lib/libcuda.so.560.35.03 /usr/lib64/libcuda.so.560.35.03]"
cuInit err: 100
time=2024-10-25T22:20:18.998+02:00 level=INFO source=gpu.go:606 msg="no nvidia devices detected by library /usr/lib/libcuda.so.560.35.03"
cuInit err: 100
time=2024-10-25T22:20:19.019+02:00 level=INFO source=gpu.go:606 msg="no nvidia devices detected by library /usr/lib64/libcuda.so.560.35.03"
time=2024-10-25T22:20:19.019+02:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=libcudart.so*
time=2024-10-25T22:20:19.019+02:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /home/bac/libcudart.so* /usr/local/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2024-10-25T22:20:19.027+02:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.4.99 /usr/local/lib/ollama/libcudart.so.11.3.109 /opt/cuda/lib64/libcudart.so.12.6.77]"
cudaSetDevice err: 100
time=2024-10-25T22:20:19.048+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.4.99: cudart init failure: 100"
cudaSetDevice err: 100
time=2024-10-25T22:20:19.069+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.11.3.109: cudart init failure: 100"
cudaSetDevice err: 100
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /opt/cuda/lib64/libcudart.so.12.6.77: cudart init failure: 100"
time=2024-10-25T22:20:19.090+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:104 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:129 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:104 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:215 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29631 unique_id=15226005257930310577
time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:249 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device
time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_linux.go:315 msg="amdgpu memory" gpu=0 total="16.0 GiB"
time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_linux.go:316 msg="amdgpu memory" gpu=0 available="12.1 GiB"
time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/local/lib/ollama"
time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /opt/rocm/lib"
time=2024-10-25T22:20:19.092+02:00 level=DEBUG source=amd_linux.go:368 msg="rocm supported GPUs" types="[gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942]"
time=2024-10-25T22:20:19.092+02:00 level=INFO source=amd_linux.go:383 msg="amdgpu is supported" gpu=0 gpu_type=gfx1030
time=2024-10-25T22:20:19.092+02:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=rocm variant="" compute=gfx1030 driver=0.0 name=1002:73bf total="16.0 GiB" available="12.1 GiB"
[GIN] 2024/10/25 - 22:20:19 | 200 | 38.871µs | 127.0.0.1 | HEAD "/"
[GIN] 2024/10/25 - 22:20:19 | 200 | 16.109382ms | 127.0.0.1 | POST "/api/show"
time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.2 GiB" now.free_swap="440.8 MiB"
time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB"
time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x80fd00 gpu_count=1
time=2024-10-25T22:20:19.651+02:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:19.651+02:00 level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[12.1 GiB]"
time=2024-10-25T22:20:19.651+02:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=13005623296 required="6.2 GiB"
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.2 GiB" now.free_swap="440.8 MiB"
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB"
time=2024-10-25T22:20:19.652+02:00 level=INFO source=server.go:105 msg="system memory" total="125.7 GiB" free="98.2 GiB" free_swap="440.8 MiB"
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[12.1 GiB]"
time=2024-10-25T22:20:19.652+02:00 level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[12.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/ollama_llama_server.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/ollama_llama_server.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/ollama_llama_server.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/ollama_llama_server.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/ollama_llama_server.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libggml.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libllama.so.gz
time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/ollama_llama_server.gz
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server
time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server
time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server --model /home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --n-gpu-layers 33 --verbose --threads 12 --parallel 4 --port 45373"
time=2024-10-25T22:20:19.654+02:00 level=DEBUG source=server.go:405 msg=subprocess environment="[PATH=/opt/rocm/bin:/home/bac/bin:/usr/local/bin:/opt/rocm/bin:/home/bac/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/usr/lib/rustup/bin CUDA_PATH=/opt/cuda LD_LIBRARY_PATH=/opt/rocm/lib:/tmp/ollama1123413816/runners/rocm_v60102 HIP_VISIBLE_DEVICES=0]"
time=2024-10-25T22:20:19.654+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
INFO [main] starting c++ runner | tid="127421724951616" timestamp=1729887619
INFO [main] build info | build=10 commit="8f7ec4a" tid="127421724951616" timestamp=1729887619
INFO [main] system info | n_threads=12 n_threads_batch=12 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="127421724951616" timestamp=1729887619 total_threads=24
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="45373" tid="127421724951616" timestamp=1729887619
llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = Meta-Llama-3-8B-Instruct
llama_model_loader: - kv 2: llama.block_count u32 = 32
llama_model_loader: - kv 3: llama.context_length u32 = 8192
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 2
llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009
llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
llama_model_loader: - kv 21: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_0: 225 tensors
llama_model_loader: - type q6_K: 1 tensors
time=2024-10-25T22:20:19.905+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.8000 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 8192
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 8192
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.33 GiB (4.64 BPW)
llm_load_print_meta: general.name = Meta-Llama-3-8B-Instruct
llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: EOG token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256

rocBLAS error: Could not initialize Tensile host: No devices found
time=2024-10-25T22:20:20.356+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding"
time=2024-10-25T22:20:21.048+02:00 level=DEBUG source=server.go:428 msg="llama runner terminated" error="signal: aborted (core dumped)"
time=2024-10-25T22:20:21.058+02:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
[GIN] 2024/10/25 - 22:20:21 | 500 | 1.448243751s | 127.0.0.1 | POST "/api/generate"
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.1 GiB" now.free_swap="440.8 MiB"
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB"
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=server.go:1086 msg="stopping llama server"
time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:380 msg="runner released" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.1 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.1 GiB" now.free_swap="440.8 MiB"
time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB"
time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:659 msg="gpu VRAM free memory converged after 0.25 seconds" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests"
^Ctime=2024-10-25T22:21:48.632+02:00 level=DEBUG source=common.go:73 msg="cleaning up" dir=/tmp/ollama1123413816
time=2024-10-25T22:21:48.632+02:00 level=DEBUG source=sched.go:119 msg="shutting down scheduler pending loop"
time=2024-10-25T22:21:48.632+02:00 level=DEBUG source=sched.go:318 msg="shutting down scheduler completed loop"

Edit: maybe something to do with https://github.com/ROCm/Tensile/issues/1757

I also installed pytorch-pytorch-rocm which has rocBLAS as dependency. Probably picked up instead of a packaged one?

Edit 2: I uninstalled everything (https://github.com/ollama/ollama/blob/main/docs/linux.md#uninstall because there was a permission error in the journal!)
Also

yay -R rocblas hipblas hipblaslt hipsolver miopen-hip rocalution rocm-hip-libraries rocm-hip-sdk rocsolver magma-hip python-pytorch-rocm
then
curl -fsSL https://ollama.com/install.sh | sh

and now it works.

Seems like a permission / user / update issue on my end leading to the lib not able to access the device.

Now Docker sounds more reasonable since it does not alter shared resources or their permissions!

<!-- gh-comment-id:2438737440 --> @blackandcold commented on GitHub (Oct 25, 2024): I have the same error but not in Docker. EndevourOS, 5900X, RX 6800 It detects memory, card and so on well but then suddenly fails with > rocBLAS error: Could not initialize Tensile host: No devices found rocBLAS is installed, rocm to, I also downloaded and extractet the additional archive (idk why this is in the docs... I mean this https://github.com/ollama/ollama/blob/main/docs/linux.md#amd-gpu-install ) and it worked fine months ago with just running the ollama install script. <details><summary>rocm-smi --showdriverversion --showproductname --showhw </summary> <p> ============================ ROCm System Management Interface ============================ ================================= Concise Hardware Info ================================== GPU DID DREV GFX RAS SDMA RAS UMC RAS VBIOS BUS 0 73bf c3 N/A N/A N/A 115-D414S00-101 0000:0C:00.0 ========================================================================================== ============================== Version of System Component =============================== Driver version: 6.11.5-arch1-1 ========================================================================================== ====================================== Product Info ====================================== GPU[0] : Card series: Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] GPU[0] : Card model: 0x04f6 GPU[0] : Card vendor: Advanced Micro Devices, Inc. [AMD/ATI] GPU[0] : Card SKU: D414S00 ========================================================================================== ================================== End of ROCm SMI Log =================================== </p> </details> Really no clue here. <details><summary>OLLAMA_DEBUG=1 ollama serve </summary> <p> 2024/10/25 22:20:08 routes.go:1158: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/bac/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2024-10-25T22:20:08.823+02:00 level=INFO source=images.go:754 msg="total blobs: 22" time=2024-10-25T22:20:08.824+02:00 level=INFO source=images.go:761 msg="total unused blobs removed: 0" time=2024-10-25T22:20:08.824+02:00 level=INFO source=routes.go:1205 msg="Listening on 127.0.0.1:11434 (version 0.3.14)" time=2024-10-25T22:20:08.824+02:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama1123413816/runners time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libggml.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libllama.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/ollama_llama_server.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libggml.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libllama.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/ollama_llama_server.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libggml.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libllama.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/ollama_llama_server.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libggml.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libllama.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/ollama_llama_server.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libggml.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libllama.so.gz time=2024-10-25T22:20:08.824+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/ollama_llama_server.gz time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libggml.so.gz time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libllama.so.gz time=2024-10-25T22:20:08.825+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/ollama_llama_server.gz time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server time=2024-10-25T22:20:18.953+02:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102 cpu]" time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-10-25T22:20:18.953+02:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-10-25T22:20:18.953+02:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs" time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:94 msg="searching for GPU discovery libraries for NVIDIA" time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=libcuda.so* time=2024-10-25T22:20:18.954+02:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /home/bac/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2024-10-25T22:20:18.977+02:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths="[/usr/lib/libcuda.so.560.35.03 /usr/lib64/libcuda.so.560.35.03]" cuInit err: 100 time=2024-10-25T22:20:18.998+02:00 level=INFO source=gpu.go:606 msg="no nvidia devices detected by library /usr/lib/libcuda.so.560.35.03" cuInit err: 100 time=2024-10-25T22:20:19.019+02:00 level=INFO source=gpu.go:606 msg="no nvidia devices detected by library /usr/lib64/libcuda.so.560.35.03" time=2024-10-25T22:20:19.019+02:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=libcudart.so* time=2024-10-25T22:20:19.019+02:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /home/bac/libcudart.so* /usr/local/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2024-10-25T22:20:19.027+02:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.4.99 /usr/local/lib/ollama/libcudart.so.11.3.109 /opt/cuda/lib64/libcudart.so.12.6.77]" cudaSetDevice err: 100 time=2024-10-25T22:20:19.048+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.4.99: cudart init failure: 100" cudaSetDevice err: 100 time=2024-10-25T22:20:19.069+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.11.3.109: cudart init failure: 100" cudaSetDevice err: 100 time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /opt/cuda/lib64/libcudart.so.12.6.77: cudart init failure: 100" time=2024-10-25T22:20:19.090+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:104 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:129 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:104 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties" time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:215 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29631 unique_id=15226005257930310577 time=2024-10-25T22:20:19.090+02:00 level=DEBUG source=amd_linux.go:249 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_linux.go:315 msg="amdgpu memory" gpu=0 total="16.0 GiB" time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_linux.go:316 msg="amdgpu memory" gpu=0 available="12.1 GiB" time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/local/lib/ollama" time=2024-10-25T22:20:19.091+02:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /opt/rocm/lib" time=2024-10-25T22:20:19.092+02:00 level=DEBUG source=amd_linux.go:368 msg="rocm supported GPUs" types="[gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942]" time=2024-10-25T22:20:19.092+02:00 level=INFO source=amd_linux.go:383 msg="amdgpu is supported" gpu=0 gpu_type=gfx1030 time=2024-10-25T22:20:19.092+02:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=rocm variant="" compute=gfx1030 driver=0.0 name=1002:73bf total="16.0 GiB" available="12.1 GiB" [GIN] 2024/10/25 - 22:20:19 | 200 | 38.871µs | 127.0.0.1 | HEAD "/" [GIN] 2024/10/25 - 22:20:19 | 200 | 16.109382ms | 127.0.0.1 | POST "/api/show" time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.2 GiB" now.free_swap="440.8 MiB" time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB" time=2024-10-25T22:20:19.624+02:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x80fd00 gpu_count=1 time=2024-10-25T22:20:19.651+02:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:19.651+02:00 level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[12.1 GiB]" time=2024-10-25T22:20:19.651+02:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=13005623296 required="6.2 GiB" time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.2 GiB" now.free_swap="440.8 MiB" time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB" time=2024-10-25T22:20:19.652+02:00 level=INFO source=server.go:105 msg="system memory" total="125.7 GiB" free="98.2 GiB" free_swap="440.8 MiB" time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[12.1 GiB]" time=2024-10-25T22:20:19.652+02:00 level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[12.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/ollama_llama_server.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/ollama_llama_server.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/ollama_llama_server.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v11 payload=linux/amd64/cuda_v11/ollama_llama_server.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=cuda_v12 payload=linux/amd64/cuda_v12/ollama_llama_server.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libggml.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/libllama.so.gz time=2024-10-25T22:20:19.652+02:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm_v60102 payload=linux/amd64/rocm_v60102/ollama_llama_server.gz time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cpu_avx2/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v11/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/cuda_v12/ollama_llama_server time=2024-10-25T22:20:19.653+02:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama1123413816/runners/rocm_v60102/ollama_llama_server --model /home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --n-gpu-layers 33 --verbose --threads 12 --parallel 4 --port 45373" time=2024-10-25T22:20:19.654+02:00 level=DEBUG source=server.go:405 msg=subprocess environment="[PATH=/opt/rocm/bin:/home/bac/bin:/usr/local/bin:/opt/rocm/bin:/home/bac/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin:/usr/lib/rustup/bin CUDA_PATH=/opt/cuda LD_LIBRARY_PATH=/opt/rocm/lib:/tmp/ollama1123413816/runners/rocm_v60102 HIP_VISIBLE_DEVICES=0]" time=2024-10-25T22:20:19.654+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-10-25T22:20:19.654+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] starting c++ runner | tid="127421724951616" timestamp=1729887619 INFO [main] build info | build=10 commit="8f7ec4a" tid="127421724951616" timestamp=1729887619 INFO [main] system info | n_threads=12 n_threads_batch=12 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="127421724951616" timestamp=1729887619 total_threads=24 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="45373" tid="127421724951616" timestamp=1729887619 llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = Meta-Llama-3-8B-Instruct llama_model_loader: - kv 2: llama.block_count u32 = 32 llama_model_loader: - kv 3: llama.context_length u32 = 8192 llama_model_loader: - kv 4: llama.embedding_length u32 = 4096 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.attention.head_count u32 = 32 llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 2 llama_model_loader: - kv 11: llama.vocab_size u32 = 128256 llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ... llama_model_loader: - kv 21: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_0: 225 tensors llama_model_loader: - type q6_K: 1 tensors time=2024-10-25T22:20:19.905+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.8000 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 8192 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 8192 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 8.03 B llm_load_print_meta: model size = 4.33 GiB (4.64 BPW) llm_load_print_meta: general.name = Meta-Llama-3-8B-Instruct llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>' llm_load_print_meta: EOS token = 128009 '<|eot_id|>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: EOG token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 rocBLAS error: Could not initialize Tensile host: No devices found time=2024-10-25T22:20:20.356+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding" time=2024-10-25T22:20:21.048+02:00 level=DEBUG source=server.go:428 msg="llama runner terminated" error="signal: aborted (core dumped)" time=2024-10-25T22:20:21.058+02:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found" time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa [GIN] 2024/10/25 - 22:20:21 | 500 | 1.448243751s | 127.0.0.1 | POST "/api/generate" time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.2 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.1 GiB" now.free_swap="440.8 MiB" time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB" time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=server.go:1086 msg="stopping llama server" time=2024-10-25T22:20:21.058+02:00 level=DEBUG source=sched.go:380 msg="runner released" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=gpu.go:396 msg="updating system memory data" before.total="125.7 GiB" before.free="98.1 GiB" before.free_swap="440.8 MiB" now.total="125.7 GiB" now.free="98.1 GiB" now.free_swap="440.8 MiB" time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=amd_linux.go:485 msg="updating rocm free memory" gpu=0 name=1002:73bf before="12.1 GiB" now="12.1 GiB" time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:659 msg="gpu VRAM free memory converged after 0.25 seconds" model=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/home/bac/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa time=2024-10-25T22:20:21.309+02:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests" ^Ctime=2024-10-25T22:21:48.632+02:00 level=DEBUG source=common.go:73 msg="cleaning up" dir=/tmp/ollama1123413816 time=2024-10-25T22:21:48.632+02:00 level=DEBUG source=sched.go:119 msg="shutting down scheduler pending loop" time=2024-10-25T22:21:48.632+02:00 level=DEBUG source=sched.go:318 msg="shutting down scheduler completed loop" </p> </details> Edit: maybe something to do with https://github.com/ROCm/Tensile/issues/1757 I also installed pytorch-pytorch-rocm which has rocBLAS as dependency. Probably picked up instead of a packaged one? Edit 2: I uninstalled everything (https://github.com/ollama/ollama/blob/main/docs/linux.md#uninstall because there was a permission error in the journal!) Also > yay -R rocblas hipblas hipblaslt hipsolver miopen-hip rocalution rocm-hip-libraries rocm-hip-sdk rocsolver magma-hip python-pytorch-rocm then > curl -fsSL https://ollama.com/install.sh | sh and now it works. Seems like a permission / user / update issue on my end leading to the lib not able to access the device. Now Docker sounds more reasonable since it does not alter shared resources or their permissions!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label amd bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4292
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?