github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3390] Models larger than AMD GPU VRAM won't load #27845

New Issue
Closed
opened 2026-04-22 05:28:37 -05:00 by GiteaMirror · 12 comments
GiteaMirror commented 2026-04-22 05:28:37 -05:00
Owner
Copy Link

Originally created by @oemsysadm on GitHub (Mar 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3390

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

When trying to load a model that won't fit in VRAM (16G) on my RX6800XT GPU, it produces an error. This is on Linux, running in the ubuntu rocm docker container.

Here is the error:

llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloaded 40/41 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 12857.81 MiB
llm_load_tensors:        CPU buffer size = 13189.86 MiB
...........................Memory access fault by GPU node-1 (Agent handle: 0x7f47bc928480) on address 0x7f43104b1000. Reason: Page not present or supervisor privilege.

What did you expect to see?

Ollama offloading to RAM working correctly.

Steps to reproduce

Using the following Dockerfile:

FROM --platform=linux/amd64 rocm/dev-ubuntu-22.04:6.0-complete

ENV OLLAMA_HOST 0.0.0.0

RUN update-pciids
RUN apt update && apt -y install curl
RUN curl https://ollama.ai/install.sh | sh

EXPOSE 11434

CMD ["ollama", "serve"]

# docker build --no-cache -t ollama_test .

# docker run --device /dev/kfd --device /dev/dri --rm --name ollama -v "$(pwd)/data":"/root/.ollama" -p 11434:11434 --name ollama ollama_test

# docker exec -it ollama ollama run llava:13b-v1.5-q8_0

Are there any recent changes that introduced the issue?

No response

OS

Linux

Architecture

amd64

Platform

Docker

Ollama version

0.1.29

GPU

AMD

GPU info

AMD Radeon RX 6800 XT (RADV NAVI21)
RADV v24.0.3

CPU

AMD

Other software

No response

Originally created by @oemsysadm on GitHub (Mar 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3390 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? When trying to load a model that won't fit in VRAM (16G) on my RX6800XT GPU, it produces an error. This is on Linux, running in the ubuntu rocm docker container. Here is the error: ``` llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloaded 40/41 layers to GPU llm_load_tensors: ROCm0 buffer size = 12857.81 MiB llm_load_tensors: CPU buffer size = 13189.86 MiB ...........................Memory access fault by GPU node-1 (Agent handle: 0x7f47bc928480) on address 0x7f43104b1000. Reason: Page not present or supervisor privilege. ``` ### What did you expect to see? Ollama offloading to RAM working correctly. ### Steps to reproduce Using the following Dockerfile: ``` FROM --platform=linux/amd64 rocm/dev-ubuntu-22.04:6.0-complete ENV OLLAMA_HOST 0.0.0.0 RUN update-pciids RUN apt update && apt -y install curl RUN curl https://ollama.ai/install.sh | sh EXPOSE 11434 CMD ["ollama", "serve"] ``` ```# docker build --no-cache -t ollama_test .``` ```# docker run --device /dev/kfd --device /dev/dri --rm --name ollama -v "$(pwd)/data":"/root/.ollama" -p 11434:11434 --name ollama ollama_test``` ```# docker exec -it ollama ollama run llava:13b-v1.5-q8_0``` ### Are there any recent changes that introduced the issue? _No response_ ### OS Linux ### Architecture amd64 ### Platform Docker ### Ollama version 0.1.29 ### GPU AMD ### GPU info AMD Radeon RX 6800 XT (RADV NAVI21) RADV v24.0.3 ### CPU AMD ### Other software _No response_
GiteaMirror added the amdbug labels 2026-04-22 05:28:37 -05:00
GiteaMirror commented 2026-04-22 05:28:39 -05:00
Author
Owner
Copy Link

@hqnicolas commented on GitHub (Mar 31, 2024):

Hello!
i'm using RX7800XT with 32GB and when I load 24GB model,
the ollama uses 16gb Vram + 20gb RAM,
so, you will need allocate first 80% of the model on RAM

keep focus on models like
7b_8Q
14b_Q4_k_M
13b_Q4_k_M
it will use just the GPU....
if you need something bigger https://www.hardware-corner.net/llm-database/CodeLlama/

<!-- gh-comment-id:2028514640 --> @hqnicolas commented on GitHub (Mar 31, 2024): Hello! i'm using RX7800XT with 32GB and when I load 24GB model, the ollama uses 16gb Vram + 20gb RAM, so, you will need allocate first 80% of the model on RAM keep focus on models like 7b_8Q 14b_Q4_k_M 13b_Q4_k_M it will use just the GPU.... if you need something bigger https://www.hardware-corner.net/llm-database/CodeLlama/
GiteaMirror commented 2026-04-22 05:28:40 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 1, 2024):

It sounds like we're mistakenly trying to load too many layers.

<!-- gh-comment-id:2030486314 --> @dhiltgen commented on GitHub (Apr 1, 2024): It sounds like we're mistakenly trying to load too many layers.
GiteaMirror commented 2026-04-22 05:28:41 -05:00
Author
Owner
Copy Link

@hqnicolas commented on GitHub (Apr 1, 2024):

@dhiltgen > It sounds like we're mistakenly trying to load too many layers.

num_gpu (60)

The number of layers to send to the GPU(s).
On macOS it defaults to 1 to enable metal support, 0 to disable.

############################
GPU 10977MB (67%) VRAM
CPU 700MB (2%) RAM
num_gpu 30 ~ 200
model: llava:13b (7.5GB)
Result: Regular inference.
############################
GPU 16380MB (99.90%) VRAM
CPU 12700MB (39%) RAM
num_gpu 10 ~ 30
model: llava:13b-v1.5-q8_0 (13.5GB)
Result: inference stop working
############################

@oemsysadm Nothing Wrong....
This model is too big for visual inference.
Try a small one

<!-- gh-comment-id:2030623111 --> @hqnicolas commented on GitHub (Apr 1, 2024): @dhiltgen > It sounds like we're mistakenly trying to load too many layers. # num_gpu (60) The number of layers to send to the GPU(s). On macOS it defaults to 1 to enable metal support, 0 to disable. ############################ GPU 10977MB (67%) VRAM CPU 700MB (2%) RAM num_gpu 30 ~ 200 model: llava:13b (7.5GB) Result: Regular inference. ############################ GPU 16380MB (99.90%) VRAM CPU 12700MB (39%) RAM num_gpu 10 ~ 30 model: llava:13b-v1.5-q8_0 (13.5GB) Result: inference stop working ############################ @oemsysadm Nothing Wrong.... This model is too big for visual inference. Try a small one
GiteaMirror commented 2026-04-22 05:28:41 -05:00
Author
Owner
Copy Link

@oemsysadm commented on GitHub (Apr 12, 2024):

Still having this issue on Ollama v0.1.31, can't load a 14G model into 16G VRAM, and it errors out with:

llm_load_tensors: offloaded 40/41 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 12857.81 MiB
llm_load_tensors:        CPU buffer size = 13189.86 MiB
........Memory access fault by GPU node-1 (Agent handle: 0x7ff86c92aa00) on address 0x7ff2eb445000. Reason: Page not present or supervisor privilege.```
<!-- gh-comment-id:2051995599 --> @oemsysadm commented on GitHub (Apr 12, 2024): Still having this issue on Ollama v0.1.31, can't load a 14G model into 16G VRAM, and it errors out with: ```llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloaded 40/41 layers to GPU llm_load_tensors: ROCm0 buffer size = 12857.81 MiB llm_load_tensors: CPU buffer size = 13189.86 MiB ........Memory access fault by GPU node-1 (Agent handle: 0x7ff86c92aa00) on address 0x7ff2eb445000. Reason: Page not present or supervisor privilege.```
GiteaMirror commented 2026-04-22 05:28:42 -05:00
Author
Owner
Copy Link

@hqnicolas commented on GitHub (Apr 12, 2024):

can't load a 14G model into 16G VRAM

@oemsysadm Buy an Apple 192gb M3 bro

<!-- gh-comment-id:2052021162 --> @hqnicolas commented on GitHub (Apr 12, 2024): > can't load a 14G model into 16G VRAM @oemsysadm Buy an Apple 192gb M3 bro
GiteaMirror commented 2026-04-22 05:28:43 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 4, 2024):

@oemsysadm can you update to the latest version and share more of the logs if you're still seeing OOM crashes? In particular, I'd like to see how much VRAM we believe is available, what what our prediction log lines look like to understand what we got wrong.

<!-- gh-comment-id:2148173515 --> @dhiltgen commented on GitHub (Jun 4, 2024): @oemsysadm can you update to the latest version and share more of the logs if you're still seeing OOM crashes? In particular, I'd like to see how much VRAM we believe is available, what what our prediction log lines look like to understand what we got wrong.
GiteaMirror commented 2026-04-22 05:28:45 -05:00
Author
Owner
Copy Link

@oemsysadm commented on GitHub (Jun 5, 2024):

I rebuilt the docker image with no cache (see original post), and then tried using the following (too big for my VRAM) model: https://ollama.com/library/deepseek-v2:16b-lite-chat-q8_0

root@08769f0efba6:~# ollama run deepseek-v2:16b-lite-chat-q8_0
Error: llama runner process has terminated: signal: aborted (core dumped)

The 'error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"' doesn't seem to matter with models that do fit comfortably in VRAM.

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.35 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloaded 26/28 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 15417.86 MiB
llm_load_tensors:        CPU buffer size =  4097.05 MiB
time=2024-06-05T17:58:47.495Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding"
time=2024-06-05T17:58:47.752Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model"
Memory access fault by GPU node-1 (Agent handle: 0x9b71110) on address 0x7f7760326000. Reason: Page not present or supervisor privilege.
time=2024-06-05T17:58:49.824Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding"
time=2024-06-05T17:58:57.296Z level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) "
[GIN] 2024/06/05 - 17:58:57 | 500 |          1m4s |       127.0.0.1 | POST     "/api/chat"
time=2024-06-05T17:58:57.315Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:57.317Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:57.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:57.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:57.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:57.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:58.074Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:58.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:58.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:58.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:58.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:58.573Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:58.822Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:58.823Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:59.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:59.077Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:59.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:59.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:59.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:59.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:58:59.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:58:59.823Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:00.074Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:00.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:00.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:00.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:00.574Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:00.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:00.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:00.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:01.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:01.073Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:01.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:01.323Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:01.572Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:01.573Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:01.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:01.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:02.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:02.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:02.318Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.022040723
time=2024-06-05T17:59:02.325Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:02.326Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:02.568Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.271570078
time=2024-06-05T17:59:02.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T17:59:02.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-05T17:59:02.817Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.521296609

A little more info on my graphics card/driver:

# lshw -c video
  *-display                 
       description: VGA compatible controller
       product: Navi 21 [Radeon RX 6800/6800 XT / 6900 XT]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:45:00.0
       logical name: /dev/fb0
       version: c1
       width: 64 bits
       clock: 33MHz
       capabilities: pm pciexpress msi vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=amdgpu latency=0 resolution=3840,2160
       resources: iomemory:7f80-7f7f iomemory:7fc0-7fbf irq:166 memory:7f800000000-7fbffffffff memory:7fc00000000-7fc0fffffff ioport:2000(size=256) memory:82300000-823fffff memory:82400000-8241ffff
<!-- gh-comment-id:2150645923 --> @oemsysadm commented on GitHub (Jun 5, 2024): I rebuilt the docker image with no cache (see original post), and then tried using the following (too big for my VRAM) model: https://ollama.com/library/deepseek-v2:16b-lite-chat-q8_0 ``` root@08769f0efba6:~# ollama run deepseek-v2:16b-lite-chat-q8_0 Error: llama runner process has terminated: signal: aborted (core dumped) ``` The 'error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"' doesn't seem to matter with models that do fit comfortably in VRAM. ``` ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.35 MiB llm_load_tensors: offloading 26 repeating layers to GPU llm_load_tensors: offloaded 26/28 layers to GPU llm_load_tensors: ROCm0 buffer size = 15417.86 MiB llm_load_tensors: CPU buffer size = 4097.05 MiB time=2024-06-05T17:58:47.495Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding" time=2024-06-05T17:58:47.752Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model" Memory access fault by GPU node-1 (Agent handle: 0x9b71110) on address 0x7f7760326000. Reason: Page not present or supervisor privilege. time=2024-06-05T17:58:49.824Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding" time=2024-06-05T17:58:57.296Z level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) " [GIN] 2024/06/05 - 17:58:57 | 500 | 1m4s | 127.0.0.1 | POST "/api/chat" time=2024-06-05T17:58:57.315Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:57.317Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:57.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:57.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:57.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:57.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:58.074Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:58.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:58.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:58.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:58.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:58.573Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:58.822Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:58.823Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:59.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:59.077Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:59.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:59.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:59.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:59.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:58:59.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:58:59.823Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:00.074Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:00.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:00.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:00.324Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:00.574Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:00.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:00.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:00.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:01.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:01.073Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:01.323Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:01.323Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:01.572Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:01.573Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:01.823Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:01.824Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:02.073Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:02.074Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:02.318Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.022040723 time=2024-06-05T17:59:02.325Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:02.326Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:02.568Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.271570078 time=2024-06-05T17:59:02.573Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-05T17:59:02.574Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-05T17:59:02.817Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.521296609 ``` A little more info on my graphics card/driver: ``` # lshw -c video *-display description: VGA compatible controller product: Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] vendor: Advanced Micro Devices, Inc. [AMD/ATI] physical id: 0 bus info: pci@0000:45:00.0 logical name: /dev/fb0 version: c1 width: 64 bits clock: 33MHz capabilities: pm pciexpress msi vga_controller bus_master cap_list rom fb configuration: depth=32 driver=amdgpu latency=0 resolution=3840,2160 resources: iomemory:7f80-7f7f iomemory:7fc0-7fbf irq:166 memory:7f800000000-7fbffffffff memory:7fc00000000-7fc0fffffff ioport:2000(size=256) memory:82300000-823fffff memory:82400000-8241ffff ```
GiteaMirror commented 2026-04-22 05:28:46 -05:00
Author
Owner
Copy Link

@oemsysadm commented on GitHub (Jun 5, 2024):

Same core dump (though during first inference, not right off the bat) with "deepseek-v2:16b-lite-chat-q5_K_M" (around 12G):

Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.35 MiB
llm_load_tensors: offloading 27 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 28/28 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 11160.99 MiB
llm_load_tensors:        CPU buffer size =   137.50 MiB

But works fine (well it responds in Chinese I think) with "ollama run deepseek-v2:latest" (around 8.9G):

Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.35 MiB
llm_load_tensors: offloading 27 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 28/28 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  8376.27 MiB
llm_load_tensors:        CPU buffer size =   112.50 MiB
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      ROCm0 KV buffer size =   540.00 MiB
llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     0.40 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =   212.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =     8.01 MiB
llama_new_context_with_model: graph nodes  = 1924
llama_new_context_with_model: graph splits = 2
INFO [main] model loaded | tid="139928864717888" timestamp=1717612929
ERROR [validate_model_chat_template] The chat template comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses | tid="139928864717888" timestamp=1717612929
time=2024-06-05T18:42:09.106Z level=INFO source=server.go:572 msg="llama runner started in 27.36 seconds"
<!-- gh-comment-id:2150725942 --> @oemsysadm commented on GitHub (Jun 5, 2024): Same core dump (though during first inference, not right off the bat) with "deepseek-v2:16b-lite-chat-q5_K_M" (around 12G): ``` Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.35 MiB llm_load_tensors: offloading 27 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 28/28 layers to GPU llm_load_tensors: ROCm0 buffer size = 11160.99 MiB llm_load_tensors: CPU buffer size = 137.50 MiB ``` But works fine (well it responds in Chinese I think) with "ollama run deepseek-v2:latest" (around 8.9G): ``` Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.35 MiB llm_load_tensors: offloading 27 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 28/28 layers to GPU llm_load_tensors: ROCm0 buffer size = 8376.27 MiB llm_load_tensors: CPU buffer size = 112.50 MiB llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 0.025 llama_kv_cache_init: ROCm0 KV buffer size = 540.00 MiB llama_new_context_with_model: KV self size = 540.00 MiB, K (f16): 324.00 MiB, V (f16): 216.00 MiB llama_new_context_with_model: ROCm_Host output buffer size = 0.40 MiB llama_new_context_with_model: ROCm0 compute buffer size = 212.00 MiB llama_new_context_with_model: ROCm_Host compute buffer size = 8.01 MiB llama_new_context_with_model: graph nodes = 1924 llama_new_context_with_model: graph splits = 2 INFO [main] model loaded | tid="139928864717888" timestamp=1717612929 ERROR [validate_model_chat_template] The chat template comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses | tid="139928864717888" timestamp=1717612929 time=2024-06-05T18:42:09.106Z level=INFO source=server.go:572 msg="llama runner started in 27.36 seconds" ```
GiteaMirror commented 2026-04-22 05:28:48 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 6, 2024):

@oemsysadm your logs are missing some earlier lines that will help me understand why we incorrectly loaded the layer counts we did

time=2024-06-06T22:40:50.789Z level=INFO source=types.go:71 msg="inference compute" id=0 library=rocm compute=gfx1102 driver=6.3 name=1002:7480 total="8.0 GiB" available="8.0 GiB"
...
time=2024-06-06T22:42:56.424Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="8.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"
...
time=2024-06-06T22:42:56.424Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama3516298434/runners/rocm_v60002/ollama_llama_server --model /home/daniel/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 1 --port 33275"
<!-- gh-comment-id:2153517697 --> @dhiltgen commented on GitHub (Jun 6, 2024): @oemsysadm your logs are missing some earlier lines that will help me understand why we incorrectly loaded the layer counts we did ``` time=2024-06-06T22:40:50.789Z level=INFO source=types.go:71 msg="inference compute" id=0 library=rocm compute=gfx1102 driver=6.3 name=1002:7480 total="8.0 GiB" available="8.0 GiB" ... time=2024-06-06T22:42:56.424Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="8.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" ... time=2024-06-06T22:42:56.424Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama3516298434/runners/rocm_v60002/ollama_llama_server --model /home/daniel/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 1 --port 33275" ```
GiteaMirror commented 2026-04-22 05:28:48 -05:00
Author
Owner
Copy Link

@oemsysadm commented on GitHub (Jun 7, 2024):

@oemsysadm your logs are missing some earlier lines that will help me understand why we incorrectly loaded the layer counts we did

Sorry, here is the full log after "ollama run deepseek-v2:16b-lite-chat-q8_0"

[GIN] 2024/06/07 - 19:29:03 | 200 |     588.639µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/06/07 - 19:29:03 | 200 |    9.543784ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2024/06/07 - 19:29:13 | 200 |        22.7µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/06/07 - 19:29:13 | 200 |   15.136101ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2024/06/07 - 19:29:13 | 200 |     662.411µs |       127.0.0.1 | POST     "/api/show"
time=2024-06-07T19:29:13.556Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:29:13.557Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:29:15.826Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB"
time=2024-06-07T19:29:15.827Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB"
time=2024-06-07T19:29:15.828Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB"
time=2024-06-07T19:29:15.828Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama1762434713/runners/rocm_v60002/ollama_llama_server --model /root/.ollama/models/blobs/sha256-458d8dbb5c64109623751f3c7e691f285770a6521bf06bf86172980b995b3bde --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --parallel 1 --port 42147"
time=2024-06-07T19:29:15.828Z level=INFO source=sched.go:338 msg="loaded runners" count=1
time=2024-06-07T19:29:15.828Z level=INFO source=server.go:529 msg="waiting for llama runner to start responding"
time=2024-06-07T19:29:15.829Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="5921b8f" tid="140235046521920" timestamp=1717788556
INFO [main] system info | n_threads=16 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140235046521920" timestamp=1717788556 total_threads=32
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="42147" tid="140235046521920" timestamp=1717788556
llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from /root/.ollama/models/blobs/sha256-458d8dbb5c64109623751f3c7e691f285770a6521bf06bf86172980b995b3bde (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.name str              = DeepSeek-V2-Lite-Chat
llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
llama_model_loader: - kv  11:                          general.file_type u32              = 7
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  37:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  108 tensors
llama_model_loader: - type q8_0:  269 tensors
time=2024-06-07T19:29:16.331Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model"
llm_load_vocab: special tokens cache size = 2400
llm_load_vocab: token to piece cache size = 1.3318 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 102400
llm_load_print_meta: n_merges         = 99757
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 27
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3072
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 6
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 16B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 15.71 B
llm_load_print_meta: model size       = 15.55 GiB (8.51 BPW) 
llm_load_print_meta: general.name     = DeepSeek-V2-Lite-Chat
llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 126 'Ä'
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 0
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1408
llm_load_print_meta: n_expert_shared      = 2
llm_load_print_meta: expert_weights_scale = 1.0
llm_load_print_meta: rope_yarn_log_mul    = 0.0707
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.35 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloaded 26/28 layers to GPU
llm_load_tensors:      ROCm0 buffer size = 15417.86 MiB
llm_load_tensors:        CPU buffer size =  4097.05 MiB
time=2024-06-07T19:30:12.008Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding"
Memory access fault by GPU node-1 (Agent handle: 0x9c3dcf0) on address 0x7f8673233000. Reason: Page not present or supervisor privilege.
time=2024-06-07T19:30:19.933Z level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) "
[GIN] 2024/06/07 - 19:30:19 | 500 |          1m6s |       127.0.0.1 | POST     "/api/chat"
time=2024-06-07T19:30:19.958Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:19.961Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:20.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:20.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:20.467Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:20.468Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:20.716Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:20.717Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:20.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:20.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:21.217Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:21.217Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:21.467Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:21.468Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:21.717Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:21.717Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:21.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:21.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:22.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:22.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:22.468Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:22.469Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:22.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:22.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:22.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:22.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:23.219Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:23.220Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:23.469Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:23.470Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:23.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:23.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:23.968Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:23.969Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:24.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:24.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:24.468Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:24.469Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:24.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:24.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:24.962Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.028521426
time=2024-06-07T19:30:24.970Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:24.971Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:25.212Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.278056016
time=2024-06-07T19:30:25.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-07T19:30:25.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
time=2024-06-07T19:30:25.462Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.528029019
<!-- gh-comment-id:2155406070 --> @oemsysadm commented on GitHub (Jun 7, 2024): > @oemsysadm your logs are missing some earlier lines that will help me understand why we incorrectly loaded the layer counts we did Sorry, here is the full log after "ollama run deepseek-v2:16b-lite-chat-q8_0" ``` [GIN] 2024/06/07 - 19:29:03 | 200 | 588.639µs | 127.0.0.1 | HEAD "/" [GIN] 2024/06/07 - 19:29:03 | 200 | 9.543784ms | 127.0.0.1 | GET "/api/tags" [GIN] 2024/06/07 - 19:29:13 | 200 | 22.7µs | 127.0.0.1 | HEAD "/" [GIN] 2024/06/07 - 19:29:13 | 200 | 15.136101ms | 127.0.0.1 | POST "/api/show" [GIN] 2024/06/07 - 19:29:13 | 200 | 662.411µs | 127.0.0.1 | POST "/api/show" time=2024-06-07T19:29:13.556Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:29:13.557Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:29:15.826Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB" time=2024-06-07T19:29:15.827Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB" time=2024-06-07T19:29:15.828Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=26 memory.available="16.0 GiB" memory.required.full="16.4 GiB" memory.required.partial="15.6 GiB" memory.required.kv="432.0 MiB" memory.weights.total="15.3 GiB" memory.weights.repeating="15.1 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="72.0 MiB" memory.graph.partial="72.0 MiB" time=2024-06-07T19:29:15.828Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama1762434713/runners/rocm_v60002/ollama_llama_server --model /root/.ollama/models/blobs/sha256-458d8dbb5c64109623751f3c7e691f285770a6521bf06bf86172980b995b3bde --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --parallel 1 --port 42147" time=2024-06-07T19:29:15.828Z level=INFO source=sched.go:338 msg="loaded runners" count=1 time=2024-06-07T19:29:15.828Z level=INFO source=server.go:529 msg="waiting for llama runner to start responding" time=2024-06-07T19:29:15.829Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="5921b8f" tid="140235046521920" timestamp=1717788556 INFO [main] system info | n_threads=16 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140235046521920" timestamp=1717788556 total_threads=32 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="42147" tid="140235046521920" timestamp=1717788556 llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from /root/.ollama/models/blobs/sha256-458d8dbb5c64109623751f3c7e691f285770a6521bf06bf86172980b995b3bde (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.name str = DeepSeek-V2-Lite-Chat llama_model_loader: - kv 2: deepseek2.block_count u32 = 27 llama_model_loader: - kv 3: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 4: deepseek2.embedding_length u32 = 2048 llama_model_loader: - kv 5: deepseek2.feed_forward_length u32 = 10944 llama_model_loader: - kv 6: deepseek2.attention.head_count u32 = 16 llama_model_loader: - kv 7: deepseek2.attention.head_count_kv u32 = 16 llama_model_loader: - kv 8: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 9: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 10: deepseek2.expert_used_count u32 = 6 llama_model_loader: - kv 11: general.file_type u32 = 7 llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 1 llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 102400 llama_model_loader: - kv 14: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 15: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 16: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 17: deepseek2.expert_feed_forward_length u32 = 1408 llama_model_loader: - kv 18: deepseek2.expert_count u32 = 64 llama_model_loader: - kv 19: deepseek2.expert_shared_count u32 = 2 llama_model_loader: - kv 20: deepseek2.expert_weights_scale f32 = 1.000000 llama_model_loader: - kv 21: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 22: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 23: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 24: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 25: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.070700 llama_model_loader: - kv 26: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 27: tokenizer.ggml.pre str = deepseek-llm llama_model_loader: - kv 28: tokenizer.ggml.tokens arr[str,102400] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 29: tokenizer.ggml.token_type arr[i32,102400] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 30: tokenizer.ggml.merges arr[str,99757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e... llama_model_loader: - kv 31: tokenizer.ggml.bos_token_id u32 = 100000 llama_model_loader: - kv 32: tokenizer.ggml.eos_token_id u32 = 100001 llama_model_loader: - kv 33: tokenizer.ggml.padding_token_id u32 = 100001 llama_model_loader: - kv 34: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 35: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 36: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 37: general.quantization_version u32 = 2 llama_model_loader: - type f32: 108 tensors llama_model_loader: - type q8_0: 269 tensors time=2024-06-07T19:29:16.331Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model" llm_load_vocab: special tokens cache size = 2400 llm_load_vocab: token to piece cache size = 1.3318 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 102400 llm_load_print_meta: n_merges = 99757 llm_load_print_meta: n_ctx_train = 163840 llm_load_print_meta: n_embd = 2048 llm_load_print_meta: n_head = 16 llm_load_print_meta: n_head_kv = 16 llm_load_print_meta: n_layer = 27 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 3072 llm_load_print_meta: n_embd_v_gqa = 2048 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 10944 llm_load_print_meta: n_expert = 64 llm_load_print_meta: n_expert_used = 6 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 0.025 llm_load_print_meta: n_yarn_orig_ctx = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 16B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 15.71 B llm_load_print_meta: model size = 15.55 GiB (8.51 BPW) llm_load_print_meta: general.name = DeepSeek-V2-Lite-Chat llm_load_print_meta: BOS token = 100000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 100001 '<｜end▁of▁sentence｜>' llm_load_print_meta: PAD token = 100001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 126 'Ä' llm_load_print_meta: n_layer_dense_lead = 1 llm_load_print_meta: n_lora_q = 0 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 1408 llm_load_print_meta: n_expert_shared = 2 llm_load_print_meta: expert_weights_scale = 1.0 llm_load_print_meta: rope_yarn_log_mul = 0.0707 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.35 MiB llm_load_tensors: offloading 26 repeating layers to GPU llm_load_tensors: offloaded 26/28 layers to GPU llm_load_tensors: ROCm0 buffer size = 15417.86 MiB llm_load_tensors: CPU buffer size = 4097.05 MiB time=2024-06-07T19:30:12.008Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding" Memory access fault by GPU node-1 (Agent handle: 0x9c3dcf0) on address 0x7f8673233000. Reason: Page not present or supervisor privilege. time=2024-06-07T19:30:19.933Z level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) " [GIN] 2024/06/07 - 19:30:19 | 500 | 1m6s | 127.0.0.1 | POST "/api/chat" time=2024-06-07T19:30:19.958Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:19.961Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:20.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:20.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:20.467Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:20.468Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:20.716Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:20.717Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:20.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:20.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:21.217Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:21.217Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:21.467Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:21.468Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:21.717Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:21.717Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:21.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:21.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:22.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:22.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:22.468Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:22.469Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:22.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:22.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:22.967Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:22.968Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:23.219Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:23.220Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:23.469Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:23.470Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:23.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:23.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:23.968Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:23.969Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:24.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:24.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:24.468Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:24.469Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:24.718Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:24.719Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:24.962Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.028521426 time=2024-06-07T19:30:24.970Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:24.971Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:25.212Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.278056016 time=2024-06-07T19:30:25.218Z level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-06-07T19:30:25.219Z level=INFO source=amd_linux.go:304 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0 time=2024-06-07T19:30:25.462Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.528029019 ```
GiteaMirror commented 2026-04-22 05:28:48 -05:00
Author
Owner
Copy Link

@gigascake commented on GitHub (Jun 20, 2024):

Oh, I didn't have money, so I bought amdgpu, but I'm worried.

<!-- gh-comment-id:2179705882 --> @gigascake commented on GitHub (Jun 20, 2024): Oh, I didn't have money, so I bought amdgpu, but I'm worried.
GiteaMirror commented 2026-04-22 05:28:49 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 20, 2024):

With PR #5192 deepseek v2 models should load properly now. The fix will be in 0.1.45.

<!-- gh-comment-id:2181293707 --> @dhiltgen commented on GitHub (Jun 20, 2024): With PR #5192 deepseek v2 models should load properly now. The fix will be in 0.1.45.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label amd bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#27845
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?