github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7034] AMD GPU 7000 Generation: segmentation fault (core dumped) #4461

New Issue
Closed
opened 2026-04-12 15:23:15 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 15:23:15 -05:00
Owner
Copy Link

Originally created by @robertosw on GitHub (Sep 29, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7034

What is the issue?

I recently upgraded my GPU from an RX 6800 to an RX 7800 XT. Both GPUs are supported by ollama and everything ran fine on the 6800. If I run the same docker container on the new gpu (command and output below) and provide the same models folder, the models will not run on the new card.

With the following command command everything seems to work. I have not yet tried it with the exact same model (gemma2:27b), currently downloading it. But llama3.2:3b worked.

$ docker run --rm --device /dev/kfd --device /dev/dri \
                              --name ollama --network local-llms \
                              -e OLLAMA_DEBUG=1 \
                              ollama/ollama:0.3.12-rocm

I'd really like to keep using the already downloaded models. I switch between like 7-10 of them regularly and dont want to download them all again :D

Original command and its output:

$ docker run --device /dev/kfd --device /dev/dri \
                                  -v /home/robert/llms/ollama:/root/.ollama \
                                  --name ollama --network local-llms \
                                  -e OLLAMA_LLM_LIBRARY=rocm -e HSA_OVERRIDE_GFX_VERSION=11.0.1 \
                                  -e OLLAMA_KEEP_ALIVE=30m -e OLLAMA_MAX_LOADED_MODELS=1 \
                                  -e OLLAMA_MAX_QUEUE=2 -e OLLAMA_NUM_PARALLEL=1 \
                                  -e OLLAMA_DEBUG=1 \
                                  ollama/ollama:0.3.12-rocm

2024/09/29 10:36:15 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:11.0.1 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:30m0s OLLAMA_LLM_LIBRARY:rocm OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:2 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2024-09-29T10:36:15.912Z level=INFO source=images.go:753 msg="total blobs: 49"
time=2024-09-29T10:36:15.914Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-09-29T10:36:15.915Z level=INFO source=routes.go:1200 msg="Listening on [::]:11434 (version 0.3.12)"
time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server
time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server
time=2024-09-29T10:36:15.918Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm_v60102]"
time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-09-29T10:36:15.918Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-09-29T10:36:15.918Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so*
time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /libcudart.so* /usr/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2024-09-29T10:36:15.920Z level=WARN source=amd_linux.go:60 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:128 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:218 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29822 unique_id=13327060393097130855
time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:252 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device
time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:284 msg="amdgpu memory" gpu=0 total="16.0 GiB"
time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:285 msg="amdgpu memory" gpu=0 available="14.0 GiB"
time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama"
time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama"
time=2024-09-29T10:36:15.921Z level=INFO source=amd_linux.go:349 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1
time=2024-09-29T10:36:15.922Z level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1101 driver=0.0 name=1002:747e total="16.0 GiB" available="14.0 GiB"
[GIN] 2024/09/29 - 10:40:33 | 200 |    8.312807ms |      172.20.0.3 | GET      "/api/tags"
[GIN] 2024/09/29 - 10:40:40 | 200 |     783.939µs |      172.20.0.3 | GET      "/api/tags"
[GIN] 2024/09/29 - 10:40:40 | 200 |      34.173µs |      172.20.0.3 | GET      "/api/version"
time=2024-09-29T10:40:44.044Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.9 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:44.044Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.1 GiB"
time=2024-09-29T10:40:44.074Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:44.074Z level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[14.1 GiB]"
time=2024-09-29T10:40:44.075Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=0 parallel=1 available=15094235136 required="13.9 GiB"
time=2024-09-29T10:40:44.075Z level=INFO source=server.go:103 msg="system memory" total="31.0 GiB" free="13.2 GiB" free_swap="15.5 GiB"
time=2024-09-29T10:40:44.075Z level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[14.1 GiB]"
time=2024-09-29T10:40:44.075Z level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[14.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="4.9 GiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="10.6 GiB" memory.weights.repeating="10.1 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="2.1 GiB" memory.graph.partial="2.2 GiB"
time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server
time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server
time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server
time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server
time=2024-09-29T10:40:44.076Z level=INFO source=server.go:169 msg="Invalid OLLAMA_LLM_LIBRARY rocm - not found"
time=2024-09-29T10:40:44.077Z level=INFO source=server.go:388 msg="starting llama server" cmd="/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 32000 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --verbose --no-mmap --parallel 1 --port 44993"
time=2024-09-29T10:40:44.077Z level=DEBUG source=server.go:405 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HSA_OVERRIDE_GFX_VERSION=11.0.1 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama/runners/rocm_v60102 HIP_VISIBLE_DEVICES=0]"
time=2024-09-29T10:40:44.078Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-09-29T10:40:44.078Z level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
time=2024-09-29T10:40:44.078Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=10 commit="86e6d1c" tid="140155691053888" timestamp=1727606444
INFO [main] system info | n_threads=8 n_threads_batch=8 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140155691053888" timestamp=1727606444 total_threads=16
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="44993" tid="140155691053888" timestamp=1727606444
llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Nemo Instruct 2407
llama_model_loader: - kv   3:                            general.version str              = 2407
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral-Nemo
llama_model_loader: - kv   6:                         general.size_label str              = 12B
llama_model_loader: - kv   7:                            general.license str              = apache-2.0
llama_model_loader: - kv   8:                          general.languages arr[str,9]       = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 40
llama_model_loader: - kv  10:                       llama.context_length u32              = 1024000
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 5120
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  18:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  19:                          general.file_type u32              = 2
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 131072
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
time=2024-09-29T10:40:44.329Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,269443]  = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {%- if messages[0]['role'] == 'system...'
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type q4_0:  281 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 1000
llm_load_vocab: token to piece cache size = 0.8498 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 131072
llm_load_print_meta: n_merges         = 269443
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 1024000
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 1024000
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 12.25 B
llm_load_print_meta: model size       = 6.58 GiB (4.61 BPW) 
llm_load_print_meta: general.name     = Mistral Nemo Instruct 2407
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 1196 'Ä'
llm_load_print_meta: max token length = 150
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.34 MiB
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  6376.58 MiB
llm_load_tensors:  ROCm_Host buffer size =   360.00 MiB
time=2024-09-29T10:40:46.033Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding"
time=2024-09-29T10:40:47.348Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
[GIN] 2024/09/29 - 10:40:47 | 500 |  3.575035488s |      172.20.0.3 | POST     "/api/chat"
time=2024-09-29T10:40:47.599Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)"
time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:47.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:47.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="7.8 GiB"
time=2024-09-29T10:40:47.599Z level=DEBUG source=server.go:1044 msg="stopping llama server"
time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:380 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:47.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:47.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="7.8 GiB" now="14.0 GiB"
time=2024-09-29T10:40:48.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:48.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB"
time=2024-09-29T10:40:48.350Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:48.350Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB"
time=2024-09-29T10:40:48.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:48.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB"
time=2024-09-29T10:40:48.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:48.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.1 GiB"
time=2024-09-29T10:40:49.099Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:49.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:49.350Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:49.350Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:49.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:49.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:49.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:49.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:50.099Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:50.099Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:50.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:50.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:50.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:50.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:50.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:50.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:51.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:51.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:51.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:51.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:51.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:51.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:51.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:51.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:52.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:52.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:52.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:52.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:52.599Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000642697 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:52.599Z level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:52.599Z level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests"
time=2024-09-29T10:40:52.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:52.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:52.850Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.251156293 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-09-29T10:40:52.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB"
time=2024-09-29T10:40:52.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB"
time=2024-09-29T10:40:53.099Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.50064399 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
^C

OS

Linux, Docker

GPU

AMD

CPU

AMD

Ollama version

0.3.12-rocm

Originally created by @robertosw on GitHub (Sep 29, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7034 ### What is the issue? I recently upgraded my GPU from an RX 6800 to an RX 7800 XT. Both GPUs are supported by ollama and everything ran fine on the 6800. If I run the same docker container on the new gpu (command and output below) and provide the same models folder, the models will not run on the new card. With the following command command everything seems to work. I have not yet tried it with the exact same model (gemma2:27b), currently downloading it. But llama3.2:3b worked. ```shell $ docker run --rm --device /dev/kfd --device /dev/dri \ --name ollama --network local-llms \ -e OLLAMA_DEBUG=1 \ ollama/ollama:0.3.12-rocm ``` I'd really like to keep using the already downloaded models. I switch between like 7-10 of them regularly and dont want to download them all again :D --- Original command and its output: ```shell $ docker run --device /dev/kfd --device /dev/dri \ -v /home/robert/llms/ollama:/root/.ollama \ --name ollama --network local-llms \ -e OLLAMA_LLM_LIBRARY=rocm -e HSA_OVERRIDE_GFX_VERSION=11.0.1 \ -e OLLAMA_KEEP_ALIVE=30m -e OLLAMA_MAX_LOADED_MODELS=1 \ -e OLLAMA_MAX_QUEUE=2 -e OLLAMA_NUM_PARALLEL=1 \ -e OLLAMA_DEBUG=1 \ ollama/ollama:0.3.12-rocm 2024/09/29 10:36:15 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:11.0.1 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:30m0s OLLAMA_LLM_LIBRARY:rocm OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:2 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2024-09-29T10:36:15.912Z level=INFO source=images.go:753 msg="total blobs: 49" time=2024-09-29T10:36:15.914Z level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-09-29T10:36:15.915Z level=INFO source=routes.go:1200 msg="Listening on [::]:11434 (version 0.3.12)" time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server time=2024-09-29T10:36:15.918Z level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm_v60102]" time=2024-09-29T10:36:15.918Z level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-09-29T10:36:15.918Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-09-29T10:36:15.918Z level=INFO source=gpu.go:199 msg="looking for compatible GPUs" time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA" time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so* time=2024-09-29T10:36:15.918Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[] time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so* time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /libcudart.so* /usr/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2024-09-29T10:36:15.920Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[] time=2024-09-29T10:36:15.920Z level=WARN source=amd_linux.go:60 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:128 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties" time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties" time=2024-09-29T10:36:15.920Z level=DEBUG source=amd_linux.go:218 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29822 unique_id=13327060393097130855 time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:252 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card1/device time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:284 msg="amdgpu memory" gpu=0 total="16.0 GiB" time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_linux.go:285 msg="amdgpu memory" gpu=0 available="14.0 GiB" time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /usr/lib/ollama" time=2024-09-29T10:36:15.921Z level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable /usr/lib/ollama" time=2024-09-29T10:36:15.921Z level=INFO source=amd_linux.go:349 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.1 time=2024-09-29T10:36:15.922Z level=INFO source=types.go:107 msg="inference compute" id=0 library=rocm variant="" compute=gfx1101 driver=0.0 name=1002:747e total="16.0 GiB" available="14.0 GiB" [GIN] 2024/09/29 - 10:40:33 | 200 | 8.312807ms | 172.20.0.3 | GET "/api/tags" [GIN] 2024/09/29 - 10:40:40 | 200 | 783.939µs | 172.20.0.3 | GET "/api/tags" [GIN] 2024/09/29 - 10:40:40 | 200 | 34.173µs | 172.20.0.3 | GET "/api/version" time=2024-09-29T10:40:44.044Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.9 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:44.044Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.1 GiB" time=2024-09-29T10:40:44.074Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:44.074Z level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[14.1 GiB]" time=2024-09-29T10:40:44.075Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=0 parallel=1 available=15094235136 required="13.9 GiB" time=2024-09-29T10:40:44.075Z level=INFO source=server.go:103 msg="system memory" total="31.0 GiB" free="13.2 GiB" free_swap="15.5 GiB" time=2024-09-29T10:40:44.075Z level=DEBUG source=memory.go:103 msg=evaluating library=rocm gpu_count=1 available="[14.1 GiB]" time=2024-09-29T10:40:44.075Z level=INFO source=memory.go:326 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[14.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="4.9 GiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="10.6 GiB" memory.weights.repeating="10.1 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="2.1 GiB" memory.graph.partial="2.2 GiB" time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2024-09-29T10:40:44.075Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu/ollama_llama_server time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2024-09-29T10:40:44.076Z level=DEBUG source=common.go:294 msg="availableServers : found" file=/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server time=2024-09-29T10:40:44.076Z level=INFO source=server.go:169 msg="Invalid OLLAMA_LLM_LIBRARY rocm - not found" time=2024-09-29T10:40:44.077Z level=INFO source=server.go:388 msg="starting llama server" cmd="/usr/lib/ollama/runners/rocm_v60102/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 32000 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --verbose --no-mmap --parallel 1 --port 44993" time=2024-09-29T10:40:44.077Z level=DEBUG source=server.go:405 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HSA_OVERRIDE_GFX_VERSION=11.0.1 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama/runners/rocm_v60102 HIP_VISIBLE_DEVICES=0]" time=2024-09-29T10:40:44.078Z level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-09-29T10:40:44.078Z level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-09-29T10:40:44.078Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=10 commit="86e6d1c" tid="140155691053888" timestamp=1727606444 INFO [main] system info | n_threads=8 n_threads_batch=8 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140155691053888" timestamp=1727606444 total_threads=16 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="44993" tid="140155691053888" timestamp=1727606444 llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407 llama_model_loader: - kv 3: general.version str = 2407 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Mistral-Nemo llama_model_loader: - kv 6: general.size_label str = 12B llama_model_loader: - kv 7: general.license str = apache-2.0 llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ... llama_model_loader: - kv 9: llama.block_count u32 = 40 llama_model_loader: - kv 10: llama.context_length u32 = 1024000 llama_model_loader: - kv 11: llama.embedding_length u32 = 5120 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: llama.attention.key_length u32 = 128 llama_model_loader: - kv 18: llama.attention.value_length u32 = 128 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - kv 20: llama.vocab_size u32 = 131072 llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... time=2024-09-29T10:40:44.329Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system...' llama_model_loader: - kv 34: general.quantization_version u32 = 2 llama_model_loader: - type f32: 81 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 1000 llm_load_vocab: token to piece cache size = 0.8498 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 131072 llm_load_print_meta: n_merges = 269443 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 1024000 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 1024000 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 12.25 B llm_load_print_meta: model size = 6.58 GiB (4.61 BPW) llm_load_print_meta: general.name = Mistral Nemo Instruct 2407 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 1196 'Ä' llm_load_print_meta: max token length = 150 /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon Graphics, compute capability 11.0, VMM: no llm_load_tensors: ggml ctx size = 0.34 MiB llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 41/41 layers to GPU llm_load_tensors: ROCm0 buffer size = 6376.58 MiB llm_load_tensors: ROCm_Host buffer size = 360.00 MiB time=2024-09-29T10:40:46.033Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding" time=2024-09-29T10:40:47.348Z level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" [GIN] 2024/09/29 - 10:40:47 | 500 | 3.575035488s | 172.20.0.3 | POST "/api/chat" time=2024-09-29T10:40:47.599Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped)" time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:47.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:47.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="7.8 GiB" time=2024-09-29T10:40:47.599Z level=DEBUG source=server.go:1044 msg="stopping llama server" time=2024-09-29T10:40:47.599Z level=DEBUG source=sched.go:380 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:47.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:47.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="7.8 GiB" now="14.0 GiB" time=2024-09-29T10:40:48.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:48.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB" time=2024-09-29T10:40:48.350Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:48.350Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB" time=2024-09-29T10:40:48.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:48.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.0 GiB" time=2024-09-29T10:40:48.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:48.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.0 GiB" now="14.1 GiB" time=2024-09-29T10:40:49.099Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:49.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:49.350Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:49.350Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:49.599Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:49.599Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:49.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:49.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:50.099Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:50.099Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:50.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:50.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:50.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:50.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:50.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:50.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:51.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.2 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:51.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:51.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.2 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:51.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:51.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:51.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:51.849Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:51.849Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:52.100Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:52.100Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:52.349Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:52.349Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:52.599Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000642697 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:52.599Z level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:52.599Z level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests" time=2024-09-29T10:40:52.600Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:52.600Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:52.850Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.251156293 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-09-29T10:40:52.850Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="31.0 GiB" before.free="13.3 GiB" before.free_swap="15.5 GiB" now.total="31.0 GiB" now.free="13.3 GiB" now.free_swap="15.5 GiB" time=2024-09-29T10:40:52.850Z level=DEBUG source=amd_linux.go:445 msg="updating rocm free memory" gpu=0 name=1002:747e before="14.1 GiB" now="14.1 GiB" time=2024-09-29T10:40:53.099Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.50064399 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 ^C ``` ### OS Linux, Docker ### GPU AMD ### CPU AMD ### Ollama version 0.3.12-rocm
GiteaMirror added the bug label 2026-04-12 15:23:15 -05:00
GiteaMirror commented 2026-04-12 15:23:17 -05:00
Author
Owner
Copy Link

@robertosw commented on GitHub (Sep 29, 2024):

Well.. downloading the same model (that failed when using the original docker container) inside the new one without a volume leads to the same problem. Only small models like llama3.2:3b run fine. All larger models that I would be using fail.

I've tested every model that previously ran fine. Both GPUs have 16GB VRAM so I dont see any reason why only smaller models run fine.

We can atleast be sure that this is not a problem of spreading a model across GPU and CPU.

<!-- gh-comment-id:2381317300 --> @robertosw commented on GitHub (Sep 29, 2024): Well.. downloading the same model (that failed when using the original docker container) inside the new one without a volume leads to the same problem. Only small models like llama3.2:3b run fine. All larger models that I would be using fail. I've tested every model that previously ran fine. Both GPUs have 16GB VRAM so I dont see any reason why only smaller models run fine. We can atleast be sure that this is not a problem of spreading a model across GPU and CPU.
GiteaMirror commented 2026-04-12 15:23:17 -05:00
Author
Owner
Copy Link

@robertosw commented on GitHub (Sep 29, 2024):

Through a thread from the ollama discord I've found this this issue discussing a problem with the llama.cpp version thats included with ollama since 0.3.7
I downgraded my docker container to the 0.3.6-rocm tag, and now everything works.

In the issue, it was mentioned that the segmentation fault only happens for models that dont fit into VRAM, which is not a problem for me. The model I use most often is Mistral Nemo (12.2B) which uses 15.4 / 16.4 GiB of my VRAM "at idle" - after a prompt was finished and the model is still loaded.
But I am no expert, maybe while processing the prompt or while loading the model my VRAM is not enough for a short duration.

<!-- gh-comment-id:2381414747 --> @robertosw commented on GitHub (Sep 29, 2024): Through a thread from the ollama discord I've found this [this issue](https://github.com/ollama/ollama/issues/6595) discussing a problem with the llama.cpp version thats included with ollama since 0.3.7 I downgraded my docker container to the 0.3.6-rocm tag, and now everything works. In the issue, it was mentioned that the segmentation fault only happens for models that dont fit into VRAM, which is not a problem for me. The model I use most often is Mistral Nemo (12.2B) which uses 15.4 / 16.4 GiB of my VRAM "at idle" - after a prompt was finished and the model is still loaded. But I am no expert, maybe while processing the prompt or while loading the model my VRAM is not enough for a short duration.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4461
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?