github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5669] "error loading llama server" error="llama runner process has terminated: exit status 0xc0000135 " #3535

New Issue
Closed
opened 2026-04-12 14:15:06 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 14:15:06 -05:00
Owner
Copy Link

Originally created by @lorenzodimauro97 on GitHub (Jul 13, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5669

What is the issue?

Cannot load any model with ollama 0.2.3, this is some of the logs:

time=2024-07-13T12:06:59.113+02:00 level=INFO source=sched.go:179 msg="one or more GPUs detected that are unable to accurately report free memory - disabling default concurrency"
time=2024-07-13T12:06:59.126+02:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\Administrator.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=17028874240 required="6.2 GiB"
time=2024-07-13T12:06:59.126+02:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.9 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-07-13T12:06:59.132+02:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\Users\Administrator\AppData\Local\Programs\Ollama\ollama_runners\rocm_v6.1\ollama_llama_server.exe --model C:\Users\Administrator\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 4 --port 61134"
time=2024-07-13T12:06:59.149+02:00 level=INFO source=sched.go:437 msg="loaded runners" count=1
time=2024-07-13T12:06:59.149+02:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding"
time=2024-07-13T12:06:59.149+02:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error"
time=2024-07-13T12:07:01.181+02:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000135 "

OS

Windows

GPU

AMD

CPU

AMD

Ollama version

0.2.3

Originally created by @lorenzodimauro97 on GitHub (Jul 13, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5669 ### What is the issue? Cannot load any model with ollama 0.2.3, this is some of the logs: time=2024-07-13T12:06:59.113+02:00 level=INFO source=sched.go:179 msg="one or more GPUs detected that are unable to accurately report free memory - disabling default concurrency" time=2024-07-13T12:06:59.126+02:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\Administrator\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=17028874240 required="6.2 GiB" time=2024-07-13T12:06:59.126+02:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.9 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" time=2024-07-13T12:06:59.132+02:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\ollama_runners\\rocm_v6.1\\ollama_llama_server.exe --model C:\\Users\\Administrator\\.ollama\\models\\blobs\\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 4 --port 61134" time=2024-07-13T12:06:59.149+02:00 level=INFO source=sched.go:437 msg="loaded runners" count=1 time=2024-07-13T12:06:59.149+02:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding" time=2024-07-13T12:06:59.149+02:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error" time=2024-07-13T12:07:01.181+02:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000135 " ### OS Windows ### GPU AMD ### CPU AMD ### Ollama version 0.2.3
GiteaMirror added the bug label 2026-04-12 14:15:06 -05:00
GiteaMirror commented 2026-04-12 14:15:08 -05:00
Author
Owner
Copy Link

@jf88888 commented on GitHub (Jul 15, 2024):

looks like a Radeon related issue.

[GIN] 2024/07/14 - 23:57:22 | 404 | 556.6µs | 127.0.0.1 | POST "/api/show"
time=2024-07-14T23:57:26.987+08:00 level=INFO source=download.go:136 msg="downloading 6a0746a1ec1a in 47 100 MB part(s)"
time=2024-07-15T00:00:12.557+08:00 level=INFO source=download.go:136 msg="downloading 4fa551d4f938 in 1 12 KB part(s)"
time=2024-07-15T00:00:16.588+08:00 level=INFO source=download.go:136 msg="downloading 8ab4849b038c in 1 254 B part(s)"
time=2024-07-15T00:00:19.905+08:00 level=INFO source=download.go:136 msg="downloading 577073ffcc6c in 1 110 B part(s)"
time=2024-07-15T00:00:23.195+08:00 level=INFO source=download.go:136 msg="downloading 3f8eb4da87fa in 1 485 B part(s)"
[GIN] 2024/07/15 - 00:00:44 | 200 | 3m21s | 127.0.0.1 | POST "/api/pull"
[GIN] 2024/07/15 - 00:00:44 | 200 | 31.466ms | 127.0.0.1 | POST "/api/show"
time=2024-07-15T00:00:45.203+08:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\j.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=17028874240 required="6.2 GiB"
time=2024-07-15T00:00:45.204+08:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.9 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-07-15T00:00:45.211+08:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\Users\j\AppData\Local\Programs\Ollama\ollama_runners\rocm_v6.1\ollama_llama_server.exe --model C:\Users\j\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 4 --port 54452"
time=2024-07-15T00:00:45.216+08:00 level=INFO source=sched.go:437 msg="loaded runners" count=1
time=2024-07-15T00:00:45.216+08:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding"
time=2024-07-15T00:00:45.217+08:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error"
time=2024-07-15T00:00:45.482+08:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000135 "

OS
Windows 11 Pro

GPU
AMD Radeon 6800XT

CPU
Intel i9

Ollama version
0.2.5

<!-- gh-comment-id:2227622973 --> @jf88888 commented on GitHub (Jul 15, 2024): looks like a Radeon related issue. [GIN] 2024/07/14 - 23:57:22 | 404 | 556.6µs | 127.0.0.1 | POST "/api/show" time=2024-07-14T23:57:26.987+08:00 level=INFO source=download.go:136 msg="downloading 6a0746a1ec1a in 47 100 MB part(s)" time=2024-07-15T00:00:12.557+08:00 level=INFO source=download.go:136 msg="downloading 4fa551d4f938 in 1 12 KB part(s)" time=2024-07-15T00:00:16.588+08:00 level=INFO source=download.go:136 msg="downloading 8ab4849b038c in 1 254 B part(s)" time=2024-07-15T00:00:19.905+08:00 level=INFO source=download.go:136 msg="downloading 577073ffcc6c in 1 110 B part(s)" time=2024-07-15T00:00:23.195+08:00 level=INFO source=download.go:136 msg="downloading 3f8eb4da87fa in 1 485 B part(s)" [GIN] 2024/07/15 - 00:00:44 | 200 | 3m21s | 127.0.0.1 | POST "/api/pull" [GIN] 2024/07/15 - 00:00:44 | 200 | 31.466ms | 127.0.0.1 | POST "/api/show" time=2024-07-15T00:00:45.203+08:00 level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\j\.ollama\models\blobs\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=0 parallel=4 available=17028874240 required="6.2 GiB" time=2024-07-15T00:00:45.204+08:00 level=INFO source=memory.go:309 msg="offload to rocm" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.9 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" time=2024-07-15T00:00:45.211+08:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\\Users\\j\\AppData\\Local\\Programs\\Ollama\\ollama_runners\\rocm_v6.1\\ollama_llama_server.exe --model C:\\Users\\j\\.ollama\\models\\blobs\\sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 4 --port 54452" time=2024-07-15T00:00:45.216+08:00 level=INFO source=sched.go:437 msg="loaded runners" count=1 time=2024-07-15T00:00:45.216+08:00 level=INFO source=server.go:571 msg="waiting for llama runner to start responding" time=2024-07-15T00:00:45.217+08:00 level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error" time=2024-07-15T00:00:45.482+08:00 level=ERROR source=sched.go:443 msg="error loading llama server" error="llama runner process has terminated: exit status 0xc0000135 " OS Windows 11 Pro GPU AMD Radeon 6800XT CPU Intel i9 Ollama version 0.2.5
GiteaMirror commented 2026-04-12 14:15:09 -05:00
Author
Owner
Copy Link

@LuisMalhadas commented on GitHub (Jul 15, 2024):

I have been looking for reports of similar issues to mine.
This is what gets closest.

@lorenzodimauro97 Have you kept trying to load??
Because I am experience the following:

I am trying to load Llama3:70b into 3 rtx3090 takes me around half hour to an hour.
I am currently using it through docker compose:

docker run -d --gpus '"device=0,2,3"' -v ollama:/root/.ollama -v .../ollama:/model -p 11434:11434 -e OLLAMA_HOST=0.0.0.0 -e OLLAMA_ORIGINS=*  -e OLLAMA_MAX_LOADED_MODELS=2 -e OLLAMA_NUM_PARALLEL=2 -e OLLAMA_DEBUG=1 OLLAMA_DEBUG=1 -e CUDA_ERROR_LEVEL=50 --name ollama2 ollama/ollama

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Mon Jul 15 08:01:44 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090 Ti     Off |   00000000:04:00.0 Off |                  Off |
| 35%   61C    P2            215W /  450W |   17597MiB /  24564MiB |     27%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:43:00.0 Off |                  N/A |
| 55%   66C    P2            213W /  350W |   17293MiB /  24576MiB |     31%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off |   00000000:88:00.0 Off |                  N/A |
| 78%   58C    P2            241W /  420W |   17293MiB /  24576MiB |     37%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        Off |   00000000:C4:00.0 Off |                  N/A |
|  0%   36C    P8             21W /  350W |   10499MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2236599      C   ...unners/cuda_v11/ollama_llama_server      17590MiB |
|    1   N/A  N/A   2236599      C   ...unners/cuda_v11/ollama_llama_server      17286MiB |
|    2   N/A  N/A   2236599      C   ...unners/cuda_v11/ollama_llama_server      17286MiB |
|    3   N/A  N/A     19625      C   /app/.venv/bin/python                       10492MiB |
+-----------------------------------------------------------------------------------------+

sudo dmesg | grep -i nvrm
[sudo] password for djfil: 
[    5.645486] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.90.07  Fri May 31 09:35:42 UTC 2024
[20922.713866] NVRM: GPU at PCI:0000:88:00: GPU-0a469c40-39b4-37e0-3229-5ff659d33432
[20922.713885] NVRM: Xid (PCI:0000:88:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[20922.713893] NVRM: GPU 0000:88:00.0: GPU has fallen off the bus.
[20922.713907] NVRM: A GPU crash dump has been created. If possible, please run
               NVRM: nvidia-bug-report.sh as root to collect this data before
               NVRM: the NVIDIA kernel module is unloaded.
djfil@antonio:~$ sudo dmesg | grep -i nvidia
[    5.401223] nvidia: loading out-of-tree module taints kernel.
[    5.401286] nvidia: module license 'NVIDIA' taints kernel.
[    5.432111] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    5.453959] nvidia-nvlink: Nvlink Core is being initialized, major device number 509
[    5.457233] nvidia 0000:04:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.508309] nvidia 0000:43:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.551675] nvidia 0000:88:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.599148] nvidia 0000:c4:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.645486] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.90.07  Fri May 31 09:35:42 UTC 2024
[    5.666118] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  550.90.07  Fri May 31 09:30:47 UTC 2024
[    5.670384] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
[    6.798471] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 1
[    6.799105] [drm] [nvidia-drm] [GPU ID 0x00004300] Loading driver
[    7.768095] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:43:00.0 on minor 2
[    7.774306] [drm] [nvidia-drm] [GPU ID 0x00008800] Loading driver
[   10.529242] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:88:00.0 on minor 3
[   10.534219] [drm] [nvidia-drm] [GPU ID 0x0000c400] Loading driver
[   11.551526] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:c4:00.0 on minor 4
[   25.182630] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[   25.188515] nvidia-uvm: Loaded the UVM driver, major device number 506.
[   29.637075] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input8
[   29.637208] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input9
[   29.637326] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input10
[   29.637466] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input11
[   29.637565] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input12
[   29.637666] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input13
[   29.637765] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input14
[   29.637928] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input15
[   29.640361] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input1
[   29.641554] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input16
[   29.649335] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input17
[   29.650121] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input2
[   29.654941] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input18
[   29.660677] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input19
[   29.667755] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input3
[   29.676380] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input20
[   29.692111] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input4
[   29.698743] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input5
[   29.699110] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input21
[   29.707294] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input6
[   29.715194] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input7
[   32.513871] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input22
[   32.514018] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input23
[   32.514146] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input24
[   32.514302] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input25
[   32.514441] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input26
[   32.514575] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input27
[   32.514720] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input28
[   34.566728] audit: type=1400 audit(1720623481.265:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1246 comm="apparmor_parser"
[   34.566737] audit: type=1400 audit(1720623481.265:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1246 comm="apparmor_parser"
[  181.782373] audit: type=1400 audit(1720623628.941:113): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=1858 comm="apparmor_parser"
[  181.782381] audit: type=1400 audit(1720623628.941:114): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=1858 comm="apparmor_parser"
[  210.793993] audit: type=1400 audit(1720623657.953:137): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=2440 comm="apparmor_parser"
[  210.793998] audit: type=1400 audit(1720623657.953:138): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=2440 comm="apparmor_parser"
[  340.825361] audit: type=1400 audit(1720623787.980:162): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=4103 comm="apparmor_parser"
[  340.825368] audit: type=1400 audit(1720623787.980:163): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=4103 comm="apparmor_parser"
[  631.337720] audit: type=1400 audit(1720624078.493:186): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=18203 comm="apparmor_parser"
[  631.337728] audit: type=1400 audit(1720624078.493:187): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=18203 comm="apparmor_parser"
               NVRM: nvidia-bug-report.sh as root to collect this data before
               NVRM: the NVIDIA kernel module is unloaded.

In the end it loads, but I get lots of:

time=2024-07-15T03:40:01.397Z level=DEBUG source=sched.go:348 msg="context for request finished"
time=2024-07-15T03:40:01.398Z level=DEBUG source=sched.go:281 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-87d5b13e5157d3a67f8e10a46d8a846ec2b68c1f731e3dfe1546a585432b8fa0 duration=5m0s
time=2024-07-15T03:40:01.398Z level=DEBUG source=sched.go:299 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-87d5b13e5157d3a67f8e10a46d8a846ec2b68c1f731e3dfe1546a585432b8fa0 refCount=0
time=2024-07-15T03:40:09.124Z level=DEBUG source=gpu.go:333 msg="updating system memory data" before.total="251.6 GiB" before.free="240.9 GiB" now.total="251.6 GiB" now.free="239.9 GiB"
time=2024-07-15T03:40:09.273Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de name="NVIDIA GeForce RTX 3090 Ti" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="23.4 GiB" now.used="266.9 MiB"
time=2024-07-15T03:40:09.372Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 name="NVIDIA GeForce RTX 3090" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="8.5 GiB" now.used="15.2 GiB"
time=2024-07-15T03:40:09.493Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 name="NVIDIA GeForce RTX 3090" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="23.4 GiB" now.used="260.9 MiB"
time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de library=cuda available="23.4 GiB"
time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de library=cuda total="23.7 GiB" available="23.4 GiB"
time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 library=cuda available="8.5 GiB"
time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 library=cuda total="23.7 GiB" available="8.5 GiB"
time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 library=cuda available="23.4 GiB"
time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 library=cuda total="23.7 GiB" available="23.4 GiB"
time=2024-07-15T03:40:09.516Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[23.4 GiB]"
time=2024-07-15T03:40:09.517Z level=DEBUG source=sched.go:628 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 available=25157238784 required="18.0 GiB"
time=2024-07-15T03:40:09.517Z level=DEBUG source=sched.go:191 msg="new model fits with existing models, loading"
time=2024-07-15T03:40:09.517Z level=DEBUG source=server.go:98 msg="system memory" total="251.6 GiB" free=257536163840
time=2024-07-15T03:40:09.517Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[23.4 GiB]"
time=2024-07-15T03:40:09.517Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[23.4 GiB]" memory.required.full="18.0 GiB" memory.required.partial="18.0 GiB" memory.required.kv="2.0 GiB" memory.required.allocations="[18.0 GiB]" memory.weights.total="15.0 GiB" memory.weights.repeating="14.0 GiB" memory.weights.nonrepeating="1002.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB"
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx2/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/rocm_v60101/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx2/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/rocm_v60101/ollama_llama_server
time=2024-07-15T03:40:09.518Z level=INFO source=server.go:368 msg="starting llama server" cmd="/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e --ctx-size 16384 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 2 --port 45355"
time=2024-07-15T03:40:09.518Z level=DEBUG source=server.go:383 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama4250454338/runners/cuda_v11:/tmp/ollama4250454338/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-0a469c40-39b4-37e0-3229-5ff659d33432]"
time=2024-07-15T03:40:09.518Z level=INFO source=sched.go:382 msg="loaded runners" count=2
time=2024-07-15T03:40:09.518Z level=INFO source=server.go:556 msg="waiting for llama runner to start responding"
time=2024-07-15T03:40:09.519Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server error"
time=2024-07-15T03:40:09.770Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model"
time=2024-07-15T03:40:11.227Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server not responding"
time=2024-07-15T03:40:11.478Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model"
time=2024-07-15T03:40:11.478Z level=DEBUG source=server.go:605 msg="model load progress 0.22"
time=2024-07-15T03:40:11.730Z level=DEBUG source=server.go:605 msg="model load progress 0.37"
time=2024-07-15T03:40:11.981Z level=DEBUG source=server.go:605 msg="model load progress 0.53"
time=2024-07-15T03:40:12.233Z level=DEBUG source=server.go:605 msg="model load progress 0.69"
time=2024-07-15T03:40:12.484Z level=DEBUG source=server.go:605 msg="model load progress 0.84"
time=2024-07-15T03:40:12.935Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server not responding"
time=2024-07-15T03:40:13.186Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model"
time=2024-07-15T03:40:13.186Z level=DEBUG source=server.go:605 msg="model load progress 1.00"
time=2024-07-15T03:40:13.437Z level=DEBUG source=server.go:608 msg="model load completed, waiting for server to become available" status="llm server loading model"
time=2024-07-15T03:40:13.940Z level=INFO source=server.go:599 msg="llama runner started in 4.42 seconds"
time=2024-07-15T03:40:13.940Z level=DEBUG source=sched.go:395 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e
<!-- gh-comment-id:2227960766 --> @LuisMalhadas commented on GitHub (Jul 15, 2024): I have been looking for reports of similar issues to mine. This is what gets closest. @lorenzodimauro97 Have you kept trying to load?? Because I am experience the following: I am trying to load Llama3:70b into 3 rtx3090 takes me around half hour to an hour. I am currently using it through docker compose: ``` docker run -d --gpus '"device=0,2,3"' -v ollama:/root/.ollama -v .../ollama:/model -p 11434:11434 -e OLLAMA_HOST=0.0.0.0 -e OLLAMA_ORIGINS=* -e OLLAMA_MAX_LOADED_MODELS=2 -e OLLAMA_NUM_PARALLEL=2 -e OLLAMA_DEBUG=1 OLLAMA_DEBUG=1 -e CUDA_ERROR_LEVEL=50 --name ollama2 ollama/ollama ``` ``` $ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` ``` Mon Jul 15 08:01:44 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 Ti Off | 00000000:04:00.0 Off | Off | | 35% 61C P2 215W / 450W | 17597MiB / 24564MiB | 27% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3090 Off | 00000000:43:00.0 Off | N/A | | 55% 66C P2 213W / 350W | 17293MiB / 24576MiB | 31% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 3090 Off | 00000000:88:00.0 Off | N/A | | 78% 58C P2 241W / 420W | 17293MiB / 24576MiB | 37% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA GeForce RTX 3090 Off | 00000000:C4:00.0 Off | N/A | | 0% 36C P8 21W / 350W | 10499MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2236599 C ...unners/cuda_v11/ollama_llama_server 17590MiB | | 1 N/A N/A 2236599 C ...unners/cuda_v11/ollama_llama_server 17286MiB | | 2 N/A N/A 2236599 C ...unners/cuda_v11/ollama_llama_server 17286MiB | | 3 N/A N/A 19625 C /app/.venv/bin/python 10492MiB | +-----------------------------------------------------------------------------------------+ ``` ``` sudo dmesg | grep -i nvrm [sudo] password for djfil: [ 5.645486] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 550.90.07 Fri May 31 09:35:42 UTC 2024 [20922.713866] NVRM: GPU at PCI:0000:88:00: GPU-0a469c40-39b4-37e0-3229-5ff659d33432 [20922.713885] NVRM: Xid (PCI:0000:88:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus. [20922.713893] NVRM: GPU 0000:88:00.0: GPU has fallen off the bus. [20922.713907] NVRM: A GPU crash dump has been created. If possible, please run NVRM: nvidia-bug-report.sh as root to collect this data before NVRM: the NVIDIA kernel module is unloaded. djfil@antonio:~$ sudo dmesg | grep -i nvidia [ 5.401223] nvidia: loading out-of-tree module taints kernel. [ 5.401286] nvidia: module license 'NVIDIA' taints kernel. [ 5.432111] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 5.453959] nvidia-nvlink: Nvlink Core is being initialized, major device number 509 [ 5.457233] nvidia 0000:04:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none [ 5.508309] nvidia 0000:43:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none [ 5.551675] nvidia 0000:88:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none [ 5.599148] nvidia 0000:c4:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none [ 5.645486] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 550.90.07 Fri May 31 09:35:42 UTC 2024 [ 5.666118] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 550.90.07 Fri May 31 09:30:47 UTC 2024 [ 5.670384] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver [ 6.798471] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 1 [ 6.799105] [drm] [nvidia-drm] [GPU ID 0x00004300] Loading driver [ 7.768095] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:43:00.0 on minor 2 [ 7.774306] [drm] [nvidia-drm] [GPU ID 0x00008800] Loading driver [ 10.529242] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:88:00.0 on minor 3 [ 10.534219] [drm] [nvidia-drm] [GPU ID 0x0000c400] Loading driver [ 11.551526] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:c4:00.0 on minor 4 [ 25.182630] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint. [ 25.188515] nvidia-uvm: Loaded the UVM driver, major device number 506. [ 29.637075] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input8 [ 29.637208] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input9 [ 29.637326] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input10 [ 29.637466] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input11 [ 29.637565] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input12 [ 29.637666] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input13 [ 29.637765] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:80/0000:80:03.1/0000:86:00.0/0000:87:00.0/0000:88:00.1/sound/card2/input14 [ 29.637928] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input15 [ 29.640361] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input1 [ 29.641554] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input16 [ 29.649335] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input17 [ 29.650121] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input2 [ 29.654941] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input18 [ 29.660677] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input19 [ 29.667755] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input3 [ 29.676380] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input20 [ 29.692111] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input4 [ 29.698743] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input5 [ 29.699110] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:c0/0000:c0:01.1/0000:c1:00.0/0000:c2:01.0/0000:c4:00.1/sound/card3/input21 [ 29.707294] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input6 [ 29.715194] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:00.0/0000:43:00.1/sound/card1/input7 [ 32.513871] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input22 [ 32.514018] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input23 [ 32.514146] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input24 [ 32.514302] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input25 [ 32.514441] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input26 [ 32.514575] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input27 [ 32.514720] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:03.1/0000:01:00.0/0000:02:01.0/0000:04:00.1/sound/card0/input28 [ 34.566728] audit: type=1400 audit(1720623481.265:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1246 comm="apparmor_parser" [ 34.566737] audit: type=1400 audit(1720623481.265:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1246 comm="apparmor_parser" [ 181.782373] audit: type=1400 audit(1720623628.941:113): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=1858 comm="apparmor_parser" [ 181.782381] audit: type=1400 audit(1720623628.941:114): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=1858 comm="apparmor_parser" [ 210.793993] audit: type=1400 audit(1720623657.953:137): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=2440 comm="apparmor_parser" [ 210.793998] audit: type=1400 audit(1720623657.953:138): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=2440 comm="apparmor_parser" [ 340.825361] audit: type=1400 audit(1720623787.980:162): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=4103 comm="apparmor_parser" [ 340.825368] audit: type=1400 audit(1720623787.980:163): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=4103 comm="apparmor_parser" [ 631.337720] audit: type=1400 audit(1720624078.493:186): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=18203 comm="apparmor_parser" [ 631.337728] audit: type=1400 audit(1720624078.493:187): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=18203 comm="apparmor_parser" NVRM: nvidia-bug-report.sh as root to collect this data before NVRM: the NVIDIA kernel module is unloaded. ``` In the end it loads, but I get lots of: ``` time=2024-07-15T03:40:01.397Z level=DEBUG source=sched.go:348 msg="context for request finished" time=2024-07-15T03:40:01.398Z level=DEBUG source=sched.go:281 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-87d5b13e5157d3a67f8e10a46d8a846ec2b68c1f731e3dfe1546a585432b8fa0 duration=5m0s time=2024-07-15T03:40:01.398Z level=DEBUG source=sched.go:299 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-87d5b13e5157d3a67f8e10a46d8a846ec2b68c1f731e3dfe1546a585432b8fa0 refCount=0 time=2024-07-15T03:40:09.124Z level=DEBUG source=gpu.go:333 msg="updating system memory data" before.total="251.6 GiB" before.free="240.9 GiB" now.total="251.6 GiB" now.free="239.9 GiB" time=2024-07-15T03:40:09.273Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de name="NVIDIA GeForce RTX 3090 Ti" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="23.4 GiB" now.used="266.9 MiB" time=2024-07-15T03:40:09.372Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 name="NVIDIA GeForce RTX 3090" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="8.5 GiB" now.used="15.2 GiB" time=2024-07-15T03:40:09.493Z level=DEBUG source=gpu.go:374 msg="updating cuda memory data" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 name="NVIDIA GeForce RTX 3090" before.total="23.7 GiB" before.free="23.4 GiB" now.total="23.7 GiB" now.free="23.4 GiB" now.used="260.9 MiB" time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de library=cuda available="23.4 GiB" time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-875fb951-07e8-0173-63ca-3926ddbd69de library=cuda total="23.7 GiB" available="23.4 GiB" time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 library=cuda available="8.5 GiB" time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-831da45a-c458-4027-02e2-c35737c26225 library=cuda total="23.7 GiB" available="8.5 GiB" time=2024-07-15T03:40:09.516Z level=DEBUG source=sched.go:429 msg="gpu reported" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 library=cuda available="23.4 GiB" time=2024-07-15T03:40:09.516Z level=INFO source=sched.go:440 msg="updated VRAM based on existing loaded models" gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 library=cuda total="23.7 GiB" available="23.4 GiB" time=2024-07-15T03:40:09.516Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[23.4 GiB]" time=2024-07-15T03:40:09.517Z level=DEBUG source=sched.go:628 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e gpu=GPU-0a469c40-39b4-37e0-3229-5ff659d33432 available=25157238784 required="18.0 GiB" time=2024-07-15T03:40:09.517Z level=DEBUG source=sched.go:191 msg="new model fits with existing models, loading" time=2024-07-15T03:40:09.517Z level=DEBUG source=server.go:98 msg="system memory" total="251.6 GiB" free=257536163840 time=2024-07-15T03:40:09.517Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[23.4 GiB]" time=2024-07-15T03:40:09.517Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[23.4 GiB]" memory.required.full="18.0 GiB" memory.required.partial="18.0 GiB" memory.required.kv="2.0 GiB" memory.required.allocations="[18.0 GiB]" memory.weights.total="15.0 GiB" memory.weights.repeating="14.0 GiB" memory.weights.nonrepeating="1002.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB" time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx2/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/rocm_v60101/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cpu_avx2/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server time=2024-07-15T03:40:09.518Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama4250454338/runners/rocm_v60101/ollama_llama_server time=2024-07-15T03:40:09.518Z level=INFO source=server.go:368 msg="starting llama server" cmd="/tmp/ollama4250454338/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e --ctx-size 16384 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 2 --port 45355" time=2024-07-15T03:40:09.518Z level=DEBUG source=server.go:383 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama4250454338/runners/cuda_v11:/tmp/ollama4250454338/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-0a469c40-39b4-37e0-3229-5ff659d33432]" time=2024-07-15T03:40:09.518Z level=INFO source=sched.go:382 msg="loaded runners" count=2 time=2024-07-15T03:40:09.518Z level=INFO source=server.go:556 msg="waiting for llama runner to start responding" time=2024-07-15T03:40:09.519Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server error" time=2024-07-15T03:40:09.770Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model" time=2024-07-15T03:40:11.227Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server not responding" time=2024-07-15T03:40:11.478Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model" time=2024-07-15T03:40:11.478Z level=DEBUG source=server.go:605 msg="model load progress 0.22" time=2024-07-15T03:40:11.730Z level=DEBUG source=server.go:605 msg="model load progress 0.37" time=2024-07-15T03:40:11.981Z level=DEBUG source=server.go:605 msg="model load progress 0.53" time=2024-07-15T03:40:12.233Z level=DEBUG source=server.go:605 msg="model load progress 0.69" time=2024-07-15T03:40:12.484Z level=DEBUG source=server.go:605 msg="model load progress 0.84" time=2024-07-15T03:40:12.935Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server not responding" time=2024-07-15T03:40:13.186Z level=INFO source=server.go:594 msg="waiting for server to become available" status="llm server loading model" time=2024-07-15T03:40:13.186Z level=DEBUG source=server.go:605 msg="model load progress 1.00" time=2024-07-15T03:40:13.437Z level=DEBUG source=server.go:608 msg="model load completed, waiting for server to become available" status="llm server loading model" time=2024-07-15T03:40:13.940Z level=INFO source=server.go:599 msg="llama runner started in 4.42 seconds" time=2024-07-15T03:40:13.940Z level=DEBUG source=sched.go:395 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-f2296999531d6120801529a45b1d103f7370c5970be939ebfc2ba5d0833e9e1e ```
GiteaMirror commented 2026-04-12 14:15:10 -05:00
Author
Owner
Copy Link

@lorenzodimauro97 commented on GitHub (Jul 15, 2024):

@LuisMalhadas I do not think these errors are related, I solved my issue by reinstalling my AMD Graphics drivers on Windows

<!-- gh-comment-id:2228118459 --> @lorenzodimauro97 commented on GitHub (Jul 15, 2024): @LuisMalhadas I do not think these errors are related, I solved my issue by reinstalling my AMD Graphics drivers on Windows
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3535
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?