github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 21:51:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #15762] failure during GPU discovery with ROCm (crash) in ubuntu 26.04 and docker container #72104

New Issue
Open
opened 2026-05-05 03:29:14 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-05-05 03:29:14 -05:00
Owner
Copy Link

Originally created by @3v1n0 on GitHub (Apr 23, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15762

What is the issue?

Running ROCm in an LXD container installed from official ubuntu repositories for 26.04 (apt install rocm) and the GPU gets properly recognized.

Launching ollama (installed with curl) fails with:

Relevant log output

time=2026-04-23T01:57:05.334Z level=INFO source=runner.go:464 msg="failure during GPU discovery" OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]" error="runner crashed"

It's unclear what the crash is

OS

Linux

GPU

AMD

CPU

AMD

Ollama version

0.21.1

Originally created by @3v1n0 on GitHub (Apr 23, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15762 ### What is the issue? Running ROCm in an LXD container installed from official ubuntu repositories for 26.04 (`apt install rocm`) and the GPU gets properly recognized. Launching ollama (installed with curl) fails with: ### Relevant log output ```shell time=2026-04-23T01:57:05.334Z level=INFO source=runner.go:464 msg="failure during GPU discovery" OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]" error="runner crashed" ``` It's unclear what the crash is ### OS Linux ### GPU AMD ### CPU AMD ### Ollama version 0.21.1
GiteaMirror added the bug label 2026-05-05 03:29:14 -05:00
GiteaMirror commented 2026-05-05 03:29:15 -05:00
Author
Owner
Copy Link

@3v1n0 commented on GitHub (Apr 23, 2026):

Note that the same happens also in the official docker container:

❯ docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
Unable to find image 'ollama/ollama:rocm' locally
rocm: Pulling from ollama/ollama
b40150c1c271: Pull complete 
2d0e9b7d523c: Pull complete 
38c1d9600047: Pull complete 
bd051fc166cf: Pull complete 
Digest: sha256:e2e4157a138fd3bcebe5288fa2d2bbfa1dbf7e9290154e15eb8e9d0d76825dbc
Status: Downloaded newer image for ollama/ollama:rocm
559c23fbaa1fb11e3c156a418a0cf3b81a3640593bf88f2ec3a620c329f7a27b

tricky /tmp/ubuntu-release-notes 26.04-updates 30s                                                        04:08:23
❯ docker ps
CONTAINER ID   IMAGE                COMMAND               CREATED         STATUS         PORTS                                           NAMES
559c23fbaa1f   ollama/ollama:rocm   "/bin/ollama serve"   5 seconds ago   Up 4 seconds   0.0.0.0:11434->11434/tcp, :::11434->11434/tcp   ollama

tricky /tmp/ubuntu-release-notes 26.04-updates                                                            04:08:27
❯ docker logs ollama                    
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDTeqlPHL47dxke6SWtMNww8w7EnapxY1xN/9QILbOH4

time=2026-04-23T02:08:23.739Z level=INFO source=routes.go:1752 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2026-04-23T02:08:23.740Z level=INFO source=routes.go:1754 msg="Ollama cloud disabled: false"
time=2026-04-23T02:08:23.740Z level=INFO source=images.go:517 msg="total blobs: 0"
time=2026-04-23T02:08:23.740Z level=INFO source=images.go:524 msg="total unused blobs removed: 0"
time=2026-04-23T02:08:23.740Z level=INFO source=routes.go:1810 msg="Listening on [::]:11434 (version 0.21.1)"
time=2026-04-23T02:08:23.741Z level=INFO source=runner.go:67 msg="discovering available GPUs..."
time=2026-04-23T02:08:23.741Z level=INFO source=server.go:444 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --port 37145"
time=2026-04-23T02:08:23.831Z level=INFO source=server.go:444 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --port 45633"
time=2026-04-23T02:08:23.921Z level=INFO source=runner.go:464 msg="failure during GPU discovery" OLLAMA_LIBRARY_PATH="[/usr/lib/ollama /usr/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]" error="runner crashed"
time=2026-04-23T02:08:23.921Z level=INFO source=types.go:60 msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="58.6 GiB" available="58.5 GiB"
time=2026-04-23T02:08:23.921Z level=INFO source=routes.go:1860 msg="vram-based default context" total_vram="0 B" default_num_ctx=4096

ubuntu@llm-boy:/root$ rocm-smi


WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status

Exception caught: map::at
========================================= ROCm System Management Interface =========================================
=================================================== Concise Info ===================================================
Device  Node  IDs              Temp    Power     Partitions          SCLK  MCLK    Fan  Perf  PwrCap  VRAM%  GPU%
              (DID,     GUID)  (Edge)  (Socket)  (Mem, Compute, ID)
====================================================================================================================
0       1     0x15bf,   64092  53.0°C  13.165W   N/A, N/A, 0         N/A   400Mhz  0%   auto  N/A     76%    4%
====================================================================================================================
=============================================== End of ROCm SMI Log ================================================
ubuntu@llm-boy:/root$ rocminfo
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
Runtime Ext Version:     1.14
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
XNACK enabled:           NO
DMAbuf Support:          YES
VMM Support:             YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics
  Uuid:                    CPU-XX
  Marketing Name:          AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   5134
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            16
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    61399540(0x3a8e1f4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    61399540(0x3a8e1f4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    61399540(0x3a8e1f4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 4
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    61399540(0x3a8e1f4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    gfx1103
  Uuid:                    GPU-XX
  Marketing Name:          AMD Radeon 780M Graphics
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      2048(0x800) KB
  Chip ID:                 5567(0x15bf)
  ASIC Revision:           9(0x9)
  Cacheline Size:          128(0x80)
  Max Clock Freq. (MHz):   2700
  BDFID:                   25600
  Internal Node ID:        1
  Compute Unit:            12
  SIMDs per CU:            2
  Shader Engines:          1
  Shader Arrs. per Eng.:   2
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:       APU
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    1024(0x400)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        2147483647(0x7fffffff)
    y                        65535(0xffff)
    z                        65535(0xffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 35
  SDMA engine uCode::      16
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    30699768(0x1d470f8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    30699768(0x1d470f8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1103
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
    ISA 2
      Name:                    amdgcn-amd-amdhsa--gfx11-generic
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
*** Done ***
<!-- gh-comment-id:4301279953 --> @3v1n0 commented on GitHub (Apr 23, 2026): Note that the same happens also in the official docker container: ``` ❯ docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm Unable to find image 'ollama/ollama:rocm' locally rocm: Pulling from ollama/ollama b40150c1c271: Pull complete 2d0e9b7d523c: Pull complete 38c1d9600047: Pull complete bd051fc166cf: Pull complete Digest: sha256:e2e4157a138fd3bcebe5288fa2d2bbfa1dbf7e9290154e15eb8e9d0d76825dbc Status: Downloaded newer image for ollama/ollama:rocm 559c23fbaa1fb11e3c156a418a0cf3b81a3640593bf88f2ec3a620c329f7a27b tricky /tmp/ubuntu-release-notes 26.04-updates 30s 04:08:23 ❯ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 559c23fbaa1f ollama/ollama:rocm "/bin/ollama serve" 5 seconds ago Up 4 seconds 0.0.0.0:11434->11434/tcp, :::11434->11434/tcp ollama tricky /tmp/ubuntu-release-notes 26.04-updates 04:08:27 ❯ docker logs ollama Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDTeqlPHL47dxke6SWtMNww8w7EnapxY1xN/9QILbOH4 time=2026-04-23T02:08:23.739Z level=INFO source=routes.go:1752 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2026-04-23T02:08:23.740Z level=INFO source=routes.go:1754 msg="Ollama cloud disabled: false" time=2026-04-23T02:08:23.740Z level=INFO source=images.go:517 msg="total blobs: 0" time=2026-04-23T02:08:23.740Z level=INFO source=images.go:524 msg="total unused blobs removed: 0" time=2026-04-23T02:08:23.740Z level=INFO source=routes.go:1810 msg="Listening on [::]:11434 (version 0.21.1)" time=2026-04-23T02:08:23.741Z level=INFO source=runner.go:67 msg="discovering available GPUs..." time=2026-04-23T02:08:23.741Z level=INFO source=server.go:444 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --port 37145" time=2026-04-23T02:08:23.831Z level=INFO source=server.go:444 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --port 45633" time=2026-04-23T02:08:23.921Z level=INFO source=runner.go:464 msg="failure during GPU discovery" OLLAMA_LIBRARY_PATH="[/usr/lib/ollama /usr/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]" error="runner crashed" time=2026-04-23T02:08:23.921Z level=INFO source=types.go:60 msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="58.6 GiB" available="58.5 GiB" time=2026-04-23T02:08:23.921Z level=INFO source=routes.go:1860 msg="vram-based default context" total_vram="0 B" default_num_ctx=4096 ``` ``` ubuntu@llm-boy:/root$ rocm-smi WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status Exception caught: map::at ========================================= ROCm System Management Interface ========================================= =================================================== Concise Info =================================================== Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Edge) (Socket) (Mem, Compute, ID) ==================================================================================================================== 0 1 0x15bf, 64092 53.0°C 13.165W N/A, N/A, 0 N/A 400Mhz 0% auto N/A 76% 4% ==================================================================================================================== =============================================== End of ROCm SMI Log ================================================ ubuntu@llm-boy:/root$ rocminfo ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version: 1.1 Runtime Ext Version: 1.14 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE System Endianness: LITTLE Mwaitx: DISABLED XNACK enabled: NO DMAbuf Support: YES VMM Support: YES ========== HSA Agents ========== ******* Agent 1 ******* Name: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics Uuid: CPU-XX Marketing Name: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics Vendor Name: CPU Feature: None specified Profile: FULL_PROFILE Float Round Mode: NEAR Max Queue Number: 0(0x0) Queue Min Size: 0(0x0) Queue Max Size: 0(0x0) Queue Type: MULTI Node: 0 Device Type: CPU Cache Info: L1: 32768(0x8000) KB Chip ID: 0(0x0) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 5134 BDFID: 0 Internal Node ID: 0 Compute Unit: 16 SIMDs per CU: 0 Shader Engines: 0 Shader Arrs. per Eng.: 0 WatchPts on Addr. Ranges:1 Memory Properties: Features: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: FINE GRAINED Size: 61399540(0x3a8e1f4) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 61399540(0x3a8e1f4) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 3 Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 61399540(0x3a8e1f4) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 4 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 61399540(0x3a8e1f4) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE ISA Info: ******* Agent 2 ******* Name: gfx1103 Uuid: GPU-XX Marketing Name: AMD Radeon 780M Graphics Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 1 Device Type: GPU Cache Info: L1: 32(0x20) KB L2: 2048(0x800) KB Chip ID: 5567(0x15bf) ASIC Revision: 9(0x9) Cacheline Size: 128(0x80) Max Clock Freq. (MHz): 2700 BDFID: 25600 Internal Node ID: 1 Compute Unit: 12 SIMDs per CU: 2 Shader Engines: 1 Shader Arrs. per Eng.: 2 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Memory Properties: APU Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 32(0x20) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 32(0x20) Max Work-item Per CU: 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 2147483647(0x7fffffff) y 65535(0xffff) z 65535(0xffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 35 SDMA engine uCode:: 16 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 30699768(0x1d470f8) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 30699768(0x1d470f8) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Recommended Granule:0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx1103 Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 2147483647(0x7fffffff) y 65535(0xffff) z 65535(0xffff) FBarrier Max Size: 32 ISA 2 Name: amdgcn-amd-amdhsa--gfx11-generic Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 2147483647(0x7fffffff) y 65535(0xffff) z 65535(0xffff) FBarrier Max Size: 32 *** Done *** ```
GiteaMirror commented 2026-05-05 03:29:15 -05:00
Author
Owner
Copy Link

@PatrickPinace commented on GitHub (Apr 23, 2026):

System info:

  • OS: Ubuntu 24.04.4 LTS, kernel 6.17.0-20-generic
  • GPU: AMD Radeon RX 6750 XT (gfx1031, Navi 22)
  • ROCm: 7.2.2
  • Ollama: 0.21.1

Issue:
Same problem as described. rocminfo and amd-smi correctly detect the GPU, but Ollama falls back to CPU with failure during GPU discovery ... error="runner crashed".

The runner subprocess does NOT crash when launched manually with the same environment variables (LD_LIBRARY_PATH, ROCR_VISIBLE_DEVICES=0, GGML_CUDA_INIT=1).
The crash only occurs when ollama serve spawns it as a GPU probe subprocess.

Setting OLLAMA_LLM_LIBRARY=rocm does not bypass the GPU probe — discovery still runs and crashes.

Note: The same crash was present with Ollama 0.13.0 + ROCm 6.2.0 on this machine, so this regression may predate 0.21.x.

<!-- gh-comment-id:4302736698 --> @PatrickPinace commented on GitHub (Apr 23, 2026): System info: - OS: Ubuntu 24.04.4 LTS, kernel 6.17.0-20-generic - GPU: AMD Radeon RX 6750 XT (gfx1031, Navi 22) - ROCm: 7.2.2 - Ollama: 0.21.1 Issue: Same problem as described. rocminfo and amd-smi correctly detect the GPU, but Ollama falls back to CPU with failure during GPU discovery ... error="runner crashed". The runner subprocess does NOT crash when launched manually with the same environment variables (LD_LIBRARY_PATH, ROCR_VISIBLE_DEVICES=0, GGML_CUDA_INIT=1). The crash only occurs when ollama serve spawns it as a GPU probe subprocess. Setting OLLAMA_LLM_LIBRARY=rocm does not bypass the GPU probe — discovery still runs and crashes. Note: The same crash was present with Ollama 0.13.0 + ROCm 6.2.0 on this machine, so this regression may predate 0.21.x.
GiteaMirror commented 2026-05-05 03:29:15 -05:00
Author
Owner
Copy Link

@3v1n0 commented on GitHub (Apr 23, 2026):

The crash only occurs when ollama serve spawns it as a GPU probe subprocess.

Yeah, it also happen making ollama to load the upstream rocm drivers (removing the files in lib/rocm, a part the actual ollama lib).

However, in my case it seems to be mitigated by HSA_OVERRIDE_GFX_VERSION=11.0.2 which is weird, since the gfx1103 should now be supported for a while.

<!-- gh-comment-id:4304761466 --> @3v1n0 commented on GitHub (Apr 23, 2026): > The crash only occurs when ollama serve spawns it as a GPU probe subprocess. Yeah, it also happen making ollama to load the upstream rocm drivers (removing the files in `lib/rocm`, a part the actual ollama lib). However, in my case it seems to be mitigated by `HSA_OVERRIDE_GFX_VERSION=11.0.2` which is weird, since the [`gfx1103` should now be supported for a while](https://github.com/ROCm/ROCm/discussions/2631).
GiteaMirror commented 2026-05-05 03:29:16 -05:00
Author
Owner
Copy Link

@chejh-amd commented on GitHub (Apr 28, 2026):

Hi @3v1n0 I tried reproduce the issue on my side but without that crash. Could you share one debug run from the same environment?
OLLAMA_DEBUG=1 OLLAMA_DEBUG_LOG_REQUESTS=true ollama serve
Please include the lines around:

  • starting runner ... --ollama-engine --port ...
  • failure during GPU discovery ... error="runner crashed"

If possible, also run this once and share stderr/exit status:
/usr/bin/ollama runner --ollama-engine --port 38000

Also, does HSA_OVERRIDE_GFX_VERSION=11.0.2 change behavior on your setup?

<!-- gh-comment-id:4334486583 --> @chejh-amd commented on GitHub (Apr 28, 2026): Hi @3v1n0 I tried reproduce the issue on my side but without that crash. Could you share one debug run from the same environment? `OLLAMA_DEBUG=1 OLLAMA_DEBUG_LOG_REQUESTS=true ollama serve` Please include the lines around: - `starting runner ... --ollama-engine --port ...` - `failure during GPU discovery ... error="runner crashed"` If possible, also run this once and share stderr/exit status: `/usr/bin/ollama runner --ollama-engine --port 38000` Also, does `HSA_OVERRIDE_GFX_VERSION=11.0.2` change behavior on your setup?
GiteaMirror commented 2026-05-05 03:29:16 -05:00
Author
Owner
Copy Link

@abpai94 commented on GitHub (Apr 29, 2026):

I have a laptop and server with 780M (gfx1103) I have been seeing this issue for a while and can confirm that using HSA_OVERRIDE_GFX_VERSION=11.0.2 works well. However the issue appears to be that the drivers related to gfx1103 isn't included in the docker image for ollama-rocm.

Check in following directory it appears that we don't have the files needed for certain HSA_OVERRIDE_GFX_VERSION to work well with Ollama.

/usr/lib/ollama/rocm/rocblas/library
  |__ ...
  |__TensileLibrary_lazy_gfx1030.dat
  |__TensileLibrary_lazy_gfx1100.dat
  |__TensileLibrary_lazy_gfx1101.dat
  |__TensileLibrary_lazy_gfx1102.dat
  |__TensileLibrary_lazy_gfx1150.dat
  |__TensileLibrary_lazy_gfx1151.dat
  |__TensileLibrary_lazy_gfx1200.dat
  |__TensileLibrary_lazy_gfx1201.dat
  |__TensileLibrary_lazy_gfx908.dat
  |__TensileLibrary_lazy_gfx90a.dat
  |__TensileLibrary_lazy_gfx942.dat
  |__TensileLibrary_lazy_gfx950.dat
  |__ ...

Hope this shines a light on the issue, I'm deciphering the Dockerfile to provide a fix if I can but its slow progress as its rather complex. Will open a pull request if I can find the solution.

<!-- gh-comment-id:4341737273 --> @abpai94 commented on GitHub (Apr 29, 2026): I have a laptop and server with 780M (gfx1103) I have been seeing this issue for a while and can confirm that using `HSA_OVERRIDE_GFX_VERSION=11.0.2` works well. However the issue appears to be that the drivers related to gfx1103 isn't included in the docker image for ollama-rocm. Check in following directory it appears that we don't have the files needed for certain `HSA_OVERRIDE_GFX_VERSION` to work well with Ollama. ``` /usr/lib/ollama/rocm/rocblas/library |__ ... |__TensileLibrary_lazy_gfx1030.dat |__TensileLibrary_lazy_gfx1100.dat |__TensileLibrary_lazy_gfx1101.dat |__TensileLibrary_lazy_gfx1102.dat |__TensileLibrary_lazy_gfx1150.dat |__TensileLibrary_lazy_gfx1151.dat |__TensileLibrary_lazy_gfx1200.dat |__TensileLibrary_lazy_gfx1201.dat |__TensileLibrary_lazy_gfx908.dat |__TensileLibrary_lazy_gfx90a.dat |__TensileLibrary_lazy_gfx942.dat |__TensileLibrary_lazy_gfx950.dat |__ ... ``` Hope this shines a light on the issue, I'm deciphering the Dockerfile to provide a fix if I can but its slow progress as its rather complex. Will open a pull request if I can find the solution.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#72104
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?