github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7956] Low GPU usage on second GPU #67150

New Issue
Open
opened 2026-05-04 09:33:11 -05:00 by GiteaMirror · 25 comments
GiteaMirror commented 2026-05-04 09:33:11 -05:00
Owner
Copy Link

Originally created by @frenzybiscuit on GitHub (Dec 5, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7956

What is the issue?

I am on the 0.5.0 release (which links to 0.4.8-rc0) and using Qwen 2.5 32b Q5 with 32k context and flash attention with q8_0 KV cache.

I have a 3090 and 2080ti.

Ollama is putting 22GB on the 3090 and 5.3GB on the 2080ti.

When running a prompt the 3090 is at 80%-90% GPU usage while the 2080ti is only at 10%.

When using llama.cpp directly with split row, the VRAM on the 2080ti is mostly maxed and the GPU usage on both GPU is in the 50%-65% range.

My question: Why is the 3090 doing most of the work on Ollama?

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.4.8-rc0

Originally created by @frenzybiscuit on GitHub (Dec 5, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7956 ### What is the issue? I am on the 0.5.0 release (which links to 0.4.8-rc0) and using Qwen 2.5 32b Q5 with 32k context and flash attention with q8_0 KV cache. I have a 3090 and 2080ti. Ollama is putting 22GB on the 3090 and 5.3GB on the 2080ti. When running a prompt the 3090 is at 80%-90% GPU usage while the 2080ti is only at 10%. When using llama.cpp directly with split row, the VRAM on the 2080ti is mostly maxed and the GPU usage on both GPU is in the 50%-65% range. ---- My question: Why is the 3090 doing most of the work on Ollama? ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.4.8-rc0
GiteaMirror added the bug label 2026-05-04 09:33:11 -05:00
GiteaMirror commented 2026-05-04 09:33:12 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 5, 2024):

Ollama has probably done the tensor split sub-optimally. Server logs will aid in debugging. What are the parameters you use when you run llama.cpp directly?

<!-- gh-comment-id:2521423984 --> @rick-github commented on GitHub (Dec 5, 2024): Ollama has probably done the tensor split sub-optimally. [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging. What are the parameters you use when you run llama.cpp directly?
GiteaMirror commented 2026-05-04 09:33:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 5, 2024):

What's the token generation rate for both configurations?

<!-- gh-comment-id:2521432200 --> @rick-github commented on GitHub (Dec 5, 2024): What's the token generation rate for both configurations?
GiteaMirror commented 2026-05-04 09:33:15 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 5, 2024):

I will try getting some benchmarks and statistics for both setups when I have the free time.

Llamacpp is faster though.

<!-- gh-comment-id:2521478865 --> @frenzybiscuit commented on GitHub (Dec 5, 2024): I will try getting some benchmarks and statistics for both setups when I have the free time. Llamacpp is faster though.
GiteaMirror commented 2026-05-04 09:33:16 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 5, 2024):

And yes you’re probably right. Iirc the tensor split on llamacpp is 2.3,1.

I will post the commandline when I’m back home tonight.

<!-- gh-comment-id:2521482107 --> @frenzybiscuit commented on GitHub (Dec 5, 2024): And yes you’re probably right. Iirc the tensor split on llamacpp is 2.3,1. I will post the commandline when I’m back home tonight.
GiteaMirror commented 2026-05-04 09:33:16 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 5, 2024):

Okay, irregardless of that. When using a Q6 GGUF it crashes on Ollama but works on llamacpp. I am not sure if it is using quant K,V cache after all.

Llamacpp command (which works):

GGML_CUDA_NO_PINNED=1 ./temp/build/bin/llama-server -m models/qwen2.5-32b-instruct-q6_k.gguf -c 32768 -ngl 500 -ts 2.3,1 -sm row -mg 0 -fa -ctv q8_0 -ctk q8_0 --threads 16 --host 192.168.0.2 --port 8080 --api-key-file apikeys.txt --log-file server.log --log-timestamps --log-colors --log-prefix

Ollama is launched via systemd with the following:

Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="CUDA_VISIBLE_DEVICES=0,1"
Environment="ROCR_VISIBLE_DEVICES=55" (fake number)

Ollama error:

Dec 05 15:49:33 blank.url ollama[891906]: ggml_cuda_init: found 2 CUDA devices:
Dec 05 15:49:33 blank.url ollama[891906]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Dec 05 15:49:33 blank.url ollama[891906]:   Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
Dec 05 15:49:33 blank.url ollama[891906]: llm_load_tensors: ggml ctx size =    1.01 MiB
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloading 64 repeating layers to GPU
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloading non-repeating layers to GPU
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloaded 65/65 layers to GPU
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors:        CPU buffer size =   609.08 MiB
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors:      CUDA0 buffer size = 20601.63 MiB
Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors:      CUDA1 buffer size =  4424.22 MiB
Dec 05 15:49:35 blank.url ollama[891906]: warning: failed to mlock 644648960-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Dec 05 15:49:35 blank.url ollama[891906]: Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_ctx      = 32768
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_batch    = 1024
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_ubatch   = 512
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: flash_attn = 1
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: freq_base  = 1000000.0
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: freq_scale = 1
Dec 05 15:49:38 blank.url ollama[891906]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 3672.00 MiB on device 0: cudaMalloc failed: out of memory
Dec 05 15:49:38 blank.url ollama[891906]: llama_kv_cache_init: failed to allocate buffer for kv cache
Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache
Dec 05 15:49:38 blank.url ollama[891906]: panic: unable to create llama context
Dec 05 15:49:38 blank.url ollama[891906]: goroutine 6 [running]:
Dec 05 15:49:38 blank.url ollama[891906]: main.(*Server).loadModel(0xc0000ee120, {0x100, 0x0, 0x1, 0x1, {0xc000014280, 0x2, 0x2}, 0xc0000281c0, 0x0}, ...)
Dec 05 15:49:38 blank.url ollama[891906]:         github.com/ollama/ollama/llama/runner/runner.go:869 +0x39c
Dec 05 15:49:38 blank.url ollama[891906]: created by main.main in goroutine 1
Dec 05 15:49:38 blank.url ollama[891906]:         github.com/ollama/ollama/llama/runner/runner.go:975 +0xc6c
Dec 05 15:49:38 blank.url ollama[891906]: time=2024-12-05T15:49:38.771-08:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nllama_kv_cache_init: failed to allocate buffer for kv cache\nllama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache"
Dec 05 15:49:38 blank.url ollama[891906]: [GIN] 2024/12/05 - 15:49:38 | 500 |   8.83010873s |       127.0.0.1 | POST     "/api/chat"
Dec 05 15:49:39 blank.url ollama[891906]: [GIN] 2024/12/05 - 15:49:39 | 200 |     484.521µs |       127.0.0.1 | GET      "/api/tags"
Dec 05 15:49:43 blank.url ollama[891906]: time=2024-12-05T15:49:43.974-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.201993782 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249
Dec 05 15:49:44 blank.url ollama[891906]: time=2024-12-05T15:49:44.284-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.512411514 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249
Dec 05 15:49:44 blank.url ollama[891906]: time=2024-12-05T15:49:44.618-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.846634021 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249
<!-- gh-comment-id:2521750994 --> @frenzybiscuit commented on GitHub (Dec 5, 2024): Okay, irregardless of that. When using a Q6 GGUF it crashes on Ollama but works on llamacpp. I am not sure if it is using quant K,V cache after all. Llamacpp command (which works): `GGML_CUDA_NO_PINNED=1 ./temp/build/bin/llama-server -m models/qwen2.5-32b-instruct-q6_k.gguf -c 32768 -ngl 500 -ts 2.3,1 -sm row -mg 0 -fa -ctv q8_0 -ctk q8_0 --threads 16 --host 192.168.0.2 --port 8080 --api-key-file apikeys.txt --log-file server.log --log-timestamps --log-colors --log-prefix` Ollama is launched via systemd with the following: ``` Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="CUDA_VISIBLE_DEVICES=0,1" Environment="ROCR_VISIBLE_DEVICES=55" (fake number) ``` Ollama error: ``` Dec 05 15:49:33 blank.url ollama[891906]: ggml_cuda_init: found 2 CUDA devices: Dec 05 15:49:33 blank.url ollama[891906]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Dec 05 15:49:33 blank.url ollama[891906]: Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes Dec 05 15:49:33 blank.url ollama[891906]: llm_load_tensors: ggml ctx size = 1.01 MiB Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloading 64 repeating layers to GPU Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloading non-repeating layers to GPU Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: offloaded 65/65 layers to GPU Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: CPU buffer size = 609.08 MiB Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: CUDA0 buffer size = 20601.63 MiB Dec 05 15:49:35 blank.url ollama[891906]: llm_load_tensors: CUDA1 buffer size = 4424.22 MiB Dec 05 15:49:35 blank.url ollama[891906]: warning: failed to mlock 644648960-byte buffer (after previously locking 0 bytes): Cannot allocate memory Dec 05 15:49:35 blank.url ollama[891906]: Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root). Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_ctx = 32768 Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_batch = 1024 Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: n_ubatch = 512 Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: flash_attn = 1 Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: freq_base = 1000000.0 Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: freq_scale = 1 Dec 05 15:49:38 blank.url ollama[891906]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 3672.00 MiB on device 0: cudaMalloc failed: out of memory Dec 05 15:49:38 blank.url ollama[891906]: llama_kv_cache_init: failed to allocate buffer for kv cache Dec 05 15:49:38 blank.url ollama[891906]: llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache Dec 05 15:49:38 blank.url ollama[891906]: panic: unable to create llama context Dec 05 15:49:38 blank.url ollama[891906]: goroutine 6 [running]: Dec 05 15:49:38 blank.url ollama[891906]: main.(*Server).loadModel(0xc0000ee120, {0x100, 0x0, 0x1, 0x1, {0xc000014280, 0x2, 0x2}, 0xc0000281c0, 0x0}, ...) Dec 05 15:49:38 blank.url ollama[891906]: github.com/ollama/ollama/llama/runner/runner.go:869 +0x39c Dec 05 15:49:38 blank.url ollama[891906]: created by main.main in goroutine 1 Dec 05 15:49:38 blank.url ollama[891906]: github.com/ollama/ollama/llama/runner/runner.go:975 +0xc6c Dec 05 15:49:38 blank.url ollama[891906]: time=2024-12-05T15:49:38.771-08:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nllama_kv_cache_init: failed to allocate buffer for kv cache\nllama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache" Dec 05 15:49:38 blank.url ollama[891906]: [GIN] 2024/12/05 - 15:49:38 | 500 | 8.83010873s | 127.0.0.1 | POST "/api/chat" Dec 05 15:49:39 blank.url ollama[891906]: [GIN] 2024/12/05 - 15:49:39 | 200 | 484.521µs | 127.0.0.1 | GET "/api/tags" Dec 05 15:49:43 blank.url ollama[891906]: time=2024-12-05T15:49:43.974-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.201993782 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249 Dec 05 15:49:44 blank.url ollama[891906]: time=2024-12-05T15:49:44.284-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.512411514 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249 Dec 05 15:49:44 blank.url ollama[891906]: time=2024-12-05T15:49:44.618-08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.846634021 model=/usr/share/ollama/.ollama/models/blobs/sha256-39c10067265fdead55f9987ea44be2a5c76457dff7b85218e69edf429f84e249 ```
GiteaMirror commented 2026-05-04 09:33:16 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 5, 2024):

Nope it is using quant k,v cache as q4 works. It's just putting too much memory onto the first GPU and not enough onto the second.

<!-- gh-comment-id:2521757664 --> @frenzybiscuit commented on GitHub (Dec 5, 2024): Nope it is using quant k,v cache as q4 works. It's just putting too much memory onto the first GPU and not enough onto the second.
GiteaMirror commented 2026-05-04 09:33:17 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 6, 2024):

Logs from earlier in the run will show ollama calculations.

<!-- gh-comment-id:2521758839 --> @rick-github commented on GitHub (Dec 6, 2024): Logs from earlier in the run will show ollama calculations.
GiteaMirror commented 2026-05-04 09:33:17 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 6, 2024):

Dec 05 16:05:11 blank.url ollama[903431]: time=2024-12-05T16:05:11.302-08:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-884a8f95-c93b-627f-cf33-0d96d887005e library=cuda total="23.6 GiB" available="338.8 MiB"
Dec 05 16:05:11 blank.url ollama[903431]: time=2024-12-05T16:05:11.302-08:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-e6b34a2a-1f14-42f9-519c-27ff284c7d2f library=cuda total="10.6 GiB" available="328.3 MiB"
Dec 05 16:05:16 blank.url ollama[903431]: time=2024-12-05T16:05:16.813-08:00 level=INFO source=server.go:105 msg="system memory" total="117.5 GiB" free="106.5 GiB" free_swap="18.8 GiB"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.135-08:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=256 layers.model=65 layers.offload=52 layers.split=43,9 memory.available="[23.3 GiB 10.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="38.5 GiB" memory.required.partial="33.1 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[23.1 GiB 10.1 GiB]" memory.weights.total="24.6 GiB" memory.weights.repeating="24.0 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="5.8 GiB" memory.graph.partial="5.8 GiB"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:231 msg="enabling flash attention"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:397 msg="starting llama server" cmd="/tmp/ollama4158640610/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-d5bb40d1e5f6d13a45b0b73c1f28ffdcc44b825ec05a06a109457e2b5c8b5ba3 --ctx-size 32768 --batch-size 1024 --n-gpu-layers 256 --threads 4 --flash-attn --kv-cache-type q8_0 --mlock --parallel 1 --tensor-split 43,9 --port 38485"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:576 msg="waiting for llama runner to start responding"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server error"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=runner.go:941 msg="starting go runner"
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=runner.go:942 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(gcc)" threads=4
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:38485"
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: loaded meta data with 29 key-value pairs and 771 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-d5bb40d1e5f6d13a45b0b73c1f28ffdcc44b825ec05a06a109457e2b5c8b5ba3 (version GGUF V3 (latest))
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 0: general.architecture str = qwen2
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 1: general.type str = model
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 2: general.name str = qwen2.5-32b-instruct
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 3: general.version str = v0.1
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 4: general.finetune str = qwen2.5-32b-instruct
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 5: general.size_label str = 33B
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 6: qwen2.block_count u32 = 64
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 7: qwen2.context_length u32 = 131072
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 8: qwen2.embedding_length u32 = 5120
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 9: qwen2.feed_forward_length u32 = 27648
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 10: qwen2.attention.head_count u32 = 40
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 11: qwen2.attention.head_count_kv u32 = 8
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 12: qwen2.rope.freq_base f32 = 1000000.000000
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 13: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 14: general.file_type u32 = 17
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 25: general.quantization_version u32 = 2
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 26: split.no u16 = 0
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 27: split.count u16 = 0
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 28: split.tensors.count i32 = 771
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type f32: 321 tensors
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type q5_K: 385 tensors
Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type q6_K: 65 tensors
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_vocab: special tokens cache size = 22
Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.388-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server loading model"
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_vocab: token to piece cache size = 0.9310 MB
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: format = GGUF V3 (latest)
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: arch = qwen2
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: vocab type = BPE
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_vocab = 152064
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_merges = 151387
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: vocab_only = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ctx_train = 131072
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd = 5120
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_layer = 64
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_head = 40
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_head_kv = 8
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_rot = 128
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_swa = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_head_k = 128
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_head_v = 128
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_gqa = 5
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_k_gqa = 1024
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_v_gqa = 1024
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_norm_eps = 0.0e+00
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_clamp_kqv = 0.0e+00
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_logit_scale = 0.0e+00
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ff = 27648
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_expert = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_expert_used = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: causal attn = 1
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: pooling type = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope type = 2
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope scaling = linear
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: freq_base_train = 1000000.0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: freq_scale_train = 1
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ctx_orig_yarn = 131072
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope_finetuned = unknown
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_conv = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_inner = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_state = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_dt_rank = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_dt_b_c_rms = 0
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model type = ?B
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model ftype = Q5_K - Medium
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model params = 32.76 B
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model size = 21.66 GiB (5.68 BPW)
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: general.name = qwen2.5-32b-instruct
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOS token = 151645 '<|im_end|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: LF token = 148848 'ÄĬ'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOT token = 151645 '<|im_end|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOG token = 151643 '<|endoftext|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOG token = 151645 '<|im_end|>'
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: max token length = 256
Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: found 2 CUDA devices:
Dec 05 16:05:17 blank.url ollama[903431]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Dec 05 16:05:17 blank.url ollama[903431]: Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
Dec 05 16:05:17 blank.url ollama[903431]: llm_load_tensors: ggml ctx size = 1.01 MiB
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloading 64 repeating layers to GPU
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloading non-repeating layers to GPU
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloaded 65/65 layers to GPU
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CPU buffer size = 510.47 MiB
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CUDA0 buffer size = 17694.37 MiB
Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CUDA1 buffer size = 3973.98 MiB
Dec 05 16:05:19 blank.url ollama[903431]: warning: failed to mlock 541245440-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Dec 05 16:05:19 blank.url ollama[903431]: Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
Dec 05 16:05:21 blank.url ollama[903431]: time=2024-12-05T16:05:21.851-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server not responding"
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_ctx = 32768
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_batch = 1024
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_ubatch = 512
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: flash_attn = 1
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: freq_base = 1000000.0
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: freq_scale = 1
Dec 05 16:05:21 blank.url ollama[903431]: llama_kv_cache_init: CUDA0 KV buffer size = 3672.00 MiB
Dec 05 16:05:21 blank.url ollama[903431]: llama_kv_cache_init: CUDA1 KV buffer size = 680.00 MiB
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: KV self size = 4352.00 MiB, K (q8_0): 2176.00 MiB, V (q8_0): 2176.00 MiB
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: CUDA_Host output buffer size = 0.60 MiB
Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA0 compute buffer size = 466.01 MiB
Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA1 compute buffer size = 475.02 MiB
Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA_Host compute buffer size = 266.02 MiB
Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: graph nodes = 1991
Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: graph splits = 3
Dec 05 16:05:22 blank.url ollama[903431]: time=2024-12-05T16:05:22.102-08:00 level=INFO source=server.go:615 msg="llama runner started in 4.97 seconds"
Dec 05 16:05:22 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:22 | 200 | 12.002863967s | 127.0.0.1 | POST "/api/chat"
Dec 05 16:05:23 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:23 | 200 | 467.151µs | 127.0.0.1 | GET "/api/tags"
Dec 05 16:05:24 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:24 | 200 | 1.169256994s | 127.0.0.1 | POST "/api/chat"

<!-- gh-comment-id:2521776645 --> @frenzybiscuit commented on GitHub (Dec 6, 2024): > Dec 05 16:05:11 blank.url ollama[903431]: time=2024-12-05T16:05:11.302-08:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-884a8f95-c93b-627f-cf33-0d96d887005e library=cuda total="23.6 GiB" available="338.8 MiB" > Dec 05 16:05:11 blank.url ollama[903431]: time=2024-12-05T16:05:11.302-08:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-e6b34a2a-1f14-42f9-519c-27ff284c7d2f library=cuda total="10.6 GiB" available="328.3 MiB" > Dec 05 16:05:16 blank.url ollama[903431]: time=2024-12-05T16:05:16.813-08:00 level=INFO source=server.go:105 msg="system memory" total="117.5 GiB" free="106.5 GiB" free_swap="18.8 GiB" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.135-08:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=256 layers.model=65 layers.offload=52 layers.split=43,9 memory.available="[23.3 GiB 10.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="38.5 GiB" memory.required.partial="33.1 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[23.1 GiB 10.1 GiB]" memory.weights.total="24.6 GiB" memory.weights.repeating="24.0 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="5.8 GiB" memory.graph.partial="5.8 GiB" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:231 msg="enabling flash attention" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:397 msg="starting llama server" cmd="/tmp/ollama4158640610/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-d5bb40d1e5f6d13a45b0b73c1f28ffdcc44b825ec05a06a109457e2b5c8b5ba3 --ctx-size 32768 --batch-size 1024 --n-gpu-layers 256 --threads 4 --flash-attn --kv-cache-type q8_0 --mlock --parallel 1 --tensor-split 43,9 --port 38485" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:576 msg="waiting for llama runner to start responding" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.136-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server error" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=runner.go:941 msg="starting go runner" > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=runner.go:942 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(gcc)" threads=4 > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.171-08:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:38485" > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: loaded meta data with 29 key-value pairs and 771 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-d5bb40d1e5f6d13a45b0b73c1f28ffdcc44b825ec05a06a109457e2b5c8b5ba3 (version GGUF V3 (latest)) > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 0: general.architecture str = qwen2 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 1: general.type str = model > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 2: general.name str = qwen2.5-32b-instruct > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 3: general.version str = v0.1 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 4: general.finetune str = qwen2.5-32b-instruct > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 5: general.size_label str = 33B > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 6: qwen2.block_count u32 = 64 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 7: qwen2.context_length u32 = 131072 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 8: qwen2.embedding_length u32 = 5120 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 9: qwen2.feed_forward_length u32 = 27648 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 10: qwen2.attention.head_count u32 = 40 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 11: qwen2.attention.head_count_kv u32 = 8 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 12: qwen2.rope.freq_base f32 = 1000000.000000 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 13: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 14: general.file_type u32 = 17 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 25: general.quantization_version u32 = 2 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 26: split.no u16 = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 27: split.count u16 = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - kv 28: split.tensors.count i32 = 771 > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type f32: 321 tensors > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type q5_K: 385 tensors > Dec 05 16:05:17 blank.url ollama[903431]: llama_model_loader: - type q6_K: 65 tensors > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_vocab: special tokens cache size = 22 > Dec 05 16:05:17 blank.url ollama[903431]: time=2024-12-05T16:05:17.388-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server loading model" > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_vocab: token to piece cache size = 0.9310 MB > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: format = GGUF V3 (latest) > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: arch = qwen2 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: vocab type = BPE > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_vocab = 152064 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_merges = 151387 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: vocab_only = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ctx_train = 131072 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd = 5120 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_layer = 64 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_head = 40 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_head_kv = 8 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_rot = 128 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_swa = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_head_k = 128 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_head_v = 128 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_gqa = 5 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_k_gqa = 1024 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_embd_v_gqa = 1024 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_norm_eps = 0.0e+00 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: f_logit_scale = 0.0e+00 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ff = 27648 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_expert = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_expert_used = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: causal attn = 1 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: pooling type = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope type = 2 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope scaling = linear > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: freq_base_train = 1000000.0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: freq_scale_train = 1 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: n_ctx_orig_yarn = 131072 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: rope_finetuned = unknown > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_conv = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_inner = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_d_state = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_dt_rank = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: ssm_dt_b_c_rms = 0 > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model type = ?B > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model ftype = Q5_K - Medium > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model params = 32.76 B > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: model size = 21.66 GiB (5.68 BPW) > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: general.name = qwen2.5-32b-instruct > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: BOS token = 151643 '<|endoftext|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOS token = 151645 '<|im_end|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: PAD token = 151643 '<|endoftext|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: LF token = 148848 'ÄĬ' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOT token = 151645 '<|im_end|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOG token = 151643 '<|endoftext|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: EOG token = 151645 '<|im_end|>' > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_print_meta: max token length = 256 > Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no > Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no > Dec 05 16:05:17 blank.url ollama[903431]: ggml_cuda_init: found 2 CUDA devices: > Dec 05 16:05:17 blank.url ollama[903431]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes > Dec 05 16:05:17 blank.url ollama[903431]: Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes > Dec 05 16:05:17 blank.url ollama[903431]: llm_load_tensors: ggml ctx size = 1.01 MiB > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloading 64 repeating layers to GPU > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloading non-repeating layers to GPU > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: offloaded 65/65 layers to GPU > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CPU buffer size = 510.47 MiB > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CUDA0 buffer size = 17694.37 MiB > Dec 05 16:05:19 blank.url ollama[903431]: llm_load_tensors: CUDA1 buffer size = 3973.98 MiB > Dec 05 16:05:19 blank.url ollama[903431]: warning: failed to mlock 541245440-byte buffer (after previously locking 0 bytes): Cannot allocate memory > Dec 05 16:05:19 blank.url ollama[903431]: Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root). > Dec 05 16:05:21 blank.url ollama[903431]: time=2024-12-05T16:05:21.851-08:00 level=INFO source=server.go:610 msg="waiting for server to become available" status="llm server not responding" > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_ctx = 32768 > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_batch = 1024 > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: n_ubatch = 512 > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: flash_attn = 1 > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: freq_base = 1000000.0 > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: freq_scale = 1 > Dec 05 16:05:21 blank.url ollama[903431]: llama_kv_cache_init: CUDA0 KV buffer size = 3672.00 MiB > Dec 05 16:05:21 blank.url ollama[903431]: llama_kv_cache_init: CUDA1 KV buffer size = 680.00 MiB > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: KV self size = 4352.00 MiB, K (q8_0): 2176.00 MiB, V (q8_0): 2176.00 MiB > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: CUDA_Host output buffer size = 0.60 MiB > Dec 05 16:05:21 blank.url ollama[903431]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4) > Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA0 compute buffer size = 466.01 MiB > Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA1 compute buffer size = 475.02 MiB > Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: CUDA_Host compute buffer size = 266.02 MiB > Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: graph nodes = 1991 > Dec 05 16:05:22 blank.url ollama[903431]: llama_new_context_with_model: graph splits = 3 > Dec 05 16:05:22 blank.url ollama[903431]: time=2024-12-05T16:05:22.102-08:00 level=INFO source=server.go:615 msg="llama runner started in 4.97 seconds" > Dec 05 16:05:22 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:22 | 200 | 12.002863967s | 127.0.0.1 | POST "/api/chat" > Dec 05 16:05:23 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:23 | 200 | 467.151µs | 127.0.0.1 | GET "/api/tags" > Dec 05 16:05:24 blank.url ollama[903431]: [GIN] 2024/12/05 - 16:05:24 | 200 | 1.169256994s | 127.0.0.1 | POST "/api/chat"
GiteaMirror commented 2026-05-04 09:33:17 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 6, 2024):

Is there a way to tell Ollama how to split the tensors per model? That may be the better idea.

<!-- gh-comment-id:2521809074 --> @frenzybiscuit commented on GitHub (Dec 6, 2024): Is there a way to tell Ollama how to split the tensors per model? That may be the better idea.
GiteaMirror commented 2026-05-04 09:33:17 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 10, 2024):

Yeah, it's pretty bad at this. See attached screenshot.

Screenshot From 2024-12-10 09-21-30

<!-- gh-comment-id:2532343410 --> @frenzybiscuit commented on GitHub (Dec 10, 2024): Yeah, it's pretty bad at this. See attached screenshot. ![Screenshot From 2024-12-10 09-21-30](https://github.com/user-attachments/assets/cee4b75b-cffb-47c6-80e4-51a8ae486fc3)
GiteaMirror commented 2026-05-04 09:33:17 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 10, 2024):

Do you have the logs for this allocation? The logs you posted earlier were from two different runs and it's difficult to piece together the flow.

<!-- gh-comment-id:2532352393 --> @rick-github commented on GitHub (Dec 10, 2024): Do you have the logs for this allocation? The logs you posted earlier were from two different runs and it's difficult to piece together the flow.
GiteaMirror commented 2026-05-04 09:33:18 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 10, 2024):

I will get logs and attach later tonight when I have free time.

I think it's important to note that every time the model restarts after its unloaded that there is different vram allocations.

Sometimes it allocates it decently (around 8GB on the 2080ti) but other times the above happens.

This is with the same model and config.

<!-- gh-comment-id:2532445960 --> @frenzybiscuit commented on GitHub (Dec 10, 2024): I will get logs and attach later tonight when I have free time. I think it's important to note that every time the model restarts after its unloaded that there is different vram allocations. Sometimes it allocates it decently (around 8GB on the 2080ti) but other times the above happens. This is with the same model and config.
GiteaMirror commented 2026-05-04 09:33:18 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 10, 2024):

I need to clean the logs of my domain name, but the tensor split on a 32b model is 26,6 which seems bad for a dual gpu setup.

<!-- gh-comment-id:2532991820 --> @frenzybiscuit commented on GitHub (Dec 10, 2024): I need to clean the logs of my domain name, but the tensor split on a 32b model is 26,6 which seems bad for a dual gpu setup.
GiteaMirror commented 2026-05-04 09:33:18 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 10, 2024):

More interesting is what ollama thought the state of the GPU was before it tried to allocate layers. The distribution algorithm tries to equalize across GPUs, but doesn't account for the relative performance of the GPUs or the size of the VRAM. If it's allocating 26 layers to one GPU it means that ollama thinks there wasn't enough space on the other to evenly distribute layers.

<!-- gh-comment-id:2533005510 --> @rick-github commented on GitHub (Dec 10, 2024): More interesting is what ollama thought the state of the GPU was before it tried to allocate layers. The distribution algorithm tries to equalize across GPUs, but doesn't account for the relative performance of the GPUs or the size of the VRAM. If it's allocating 26 layers to one GPU it means that ollama thinks there wasn't enough space on the other to evenly distribute layers.
GiteaMirror commented 2026-05-04 09:33:19 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 10, 2024):

More interesting is what ollama thought the state of the GPU was before it tried to allocate layers. The distribution algorithm tries to equalize across GPUs, but doesn't account for the relative performance of the GPUs or the size of the VRAM. If it's allocating 26 layers to one GPU it means that ollama thinks there wasn't enough space on the other to evenly distribute layers.

This could be the issue. When loading a model (any model) it ping-pongs back and forth between both GPU loading and unloading from VRAM. It will eventually (usually 3-4 seconds of this) load and stay loaded into VRAM.

I can take a screen recording of this behavior if needed.

<!-- gh-comment-id:2533238239 --> @frenzybiscuit commented on GitHub (Dec 10, 2024): > More interesting is what ollama thought the state of the GPU was before it tried to allocate layers. The distribution algorithm tries to equalize across GPUs, but doesn't account for the relative performance of the GPUs or the size of the VRAM. If it's allocating 26 layers to one GPU it means that ollama thinks there wasn't enough space on the other to evenly distribute layers. This could be the issue. When loading a model (any model) it ping-pongs back and forth between both GPU loading and unloading from VRAM. It will eventually (usually 3-4 seconds of this) load and stay loaded into VRAM. I can take a screen recording of this behavior if needed.
GiteaMirror commented 2026-05-04 09:33:19 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 11, 2024):

If the model is loading and unloading from VRAM it will be recorded in the logs. But this behaviour isn't normal, a screen recording may shed light.

<!-- gh-comment-id:2533262765 --> @rick-github commented on GitHub (Dec 11, 2024): If the model is loading and unloading from VRAM it will be recorded in the logs. But this behaviour isn't normal, a screen recording may shed light.
GiteaMirror commented 2026-05-04 09:33:19 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 11, 2024):

Alright, here you go. I don't particularly care if the domain is out there.

I am not able to repeat the prior results, but the tensor split is still 26,6 and the vram on the 2080ti is still lower then it should be.

https://github.com/user-attachments/assets/a46e0679-fc82-4d9c-8295-5bd28d03bff8

<!-- gh-comment-id:2533386637 --> @frenzybiscuit commented on GitHub (Dec 11, 2024): Alright, here you go. I don't particularly care if the domain is out there. I am not able to repeat the prior results, but the tensor split is still 26,6 and the vram on the 2080ti is still lower then it should be. https://github.com/user-attachments/assets/a46e0679-fc82-4d9c-8295-5bd28d03bff8
GiteaMirror commented 2026-05-04 09:33:20 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 11, 2024):

Also increasing context length to use the free vram on the 2080ti doesn't work. The model doesn't load and errors out.

<!-- gh-comment-id:2533389038 --> @frenzybiscuit commented on GitHub (Dec 11, 2024): Also increasing context length to use the free vram on the 2080ti doesn't work. The model doesn't load and errors out.
GiteaMirror commented 2026-05-04 09:33:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 11, 2024):

Can you add the logs for this period?

<!-- gh-comment-id:2533394050 --> @rick-github commented on GitHub (Dec 11, 2024): Can you add the logs for this period?
GiteaMirror commented 2026-05-04 09:33:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 11, 2024):

Ollama switched from qwen2 to command-r. It's expected that the VRAM will go down when the model is unloaded and then go up again when a new model is loaded.

<!-- gh-comment-id:2533413460 --> @rick-github commented on GitHub (Dec 11, 2024): Ollama switched from qwen2 to command-r. It's expected that the VRAM will go down when the model is unloaded and then go up again when a new model is loaded.
GiteaMirror commented 2026-05-04 09:33:20 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 11, 2024):

Yeah, but it does the same thing when not switching models. I can post another screen capture of it without one loaded if you need it.

<!-- gh-comment-id:2533414589 --> @frenzybiscuit commented on GitHub (Dec 11, 2024): Yeah, but it does the same thing when not switching models. I can post another screen capture of it without one loaded if you need it.
GiteaMirror commented 2026-05-04 09:33:21 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 11, 2024):

Interestingly, it works fine on the 72b models. The vram is split correctly on those models. (I.E. 20GB on the 3090 and 10GB on the 2080ti)

Seems to be an issue with the lower parameter models specifically?

<!-- gh-comment-id:2533423315 --> @frenzybiscuit commented on GitHub (Dec 11, 2024): Interestingly, it works fine on the 72b models. The vram is split correctly on those models. (I.E. 20GB on the 3090 and 10GB on the 2080ti) Seems to be an issue with the lower parameter models specifically?
GiteaMirror commented 2026-05-04 09:33:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 11, 2024):

If you could post logs of these events that would be great.

<!-- gh-comment-id:2533435350 --> @rick-github commented on GitHub (Dec 11, 2024): If you could post logs of these events that would be great.
GiteaMirror commented 2026-05-04 09:33:21 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 11, 2024):

What's the best way to debug Ollama when installed as a systemd service?

Anyway, I know these aren't the logs you wanted but this is Ollama compared to Llamacpp.

The actual speed is similar (Ollama actually appears faster) however, llamacpp is able to run a Q6 qwen 2.5 32b while Ollama is only able to run a Q5. Also, llamacpp has a higher context (by about 20k). So if there is speed difference, I suspect that is why.

Llamacpp:

Screenshot 2024-12-11 at 3 01 24 PM

Ollama:

Screenshot 2024-12-11 at 3 07 29 PM
<!-- gh-comment-id:2537401950 --> @frenzybiscuit commented on GitHub (Dec 11, 2024): What's the best way to debug Ollama when installed as a systemd service? Anyway, I know these aren't the logs you wanted but this is Ollama compared to Llamacpp. The actual speed is similar (Ollama actually appears faster) however, llamacpp is able to run a Q6 qwen 2.5 32b while Ollama is only able to run a Q5. Also, llamacpp has a higher context (by about 20k). So if there is speed difference, I suspect that is why. Llamacpp: <img width="942" alt="Screenshot 2024-12-11 at 3 01 24 PM" src="https://github.com/user-attachments/assets/48bb7bcb-3dd3-4f36-a573-95ebbb770e78" /> Ollama: <img width="730" alt="Screenshot 2024-12-11 at 3 07 29 PM" src="https://github.com/user-attachments/assets/98774fca-bc76-4a5e-bd19-20292008548a" />
GiteaMirror commented 2026-05-04 09:33:21 -05:00
Author
Owner
Copy Link

@frenzybiscuit commented on GitHub (Dec 14, 2024):

Can I get clarification on how ollama handles multi-gpu setups?

Is it splitting by row?

If so, is it splitting the tensors with 2,1 or with 2.5,1? In llamacpp I am splitting by 2.5.

<!-- gh-comment-id:2543363619 --> @frenzybiscuit commented on GitHub (Dec 14, 2024): Can I get clarification on how ollama handles multi-gpu setups? Is it splitting by row? If so, is it splitting the tensors with 2,1 or with 2.5,1? In llamacpp I am splitting by 2.5.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#67150
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?