github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 13:42:25 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10916] Cannot get Ollama to utilize two 3090s for processing small/large models? #7181

New Issue
Closed
opened 2026-04-12 19:10:45 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 19:10:45 -05:00
Owner
Copy Link

Originally created by @wingraver on GitHub (May 30, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10916

What is the issue?

Having tried small (8B) and large (70B) models alike, thinking &/or vision models, and currently updated to version 0.9.0, I have not been able to ever get Ollama to fully utilize the two 3090s on my machine. Even when "ollama ps" says it's using GPU 100%, task manager and process explorer (and the fans on my PC) suggest otherwise.

Sometimes there's a brief spike in usage that quickly falls away and most all of the activity is seen on the CPU.

Maybe it's me... Is there some specific steps that need to be followed to ensure maximum utilization of the GPUs? Do I need to buy an NVLink, apply some special global configuration... feels like a waste. Thanks in advance for any help in getting this right...

Relevant log output

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.9.0

Originally created by @wingraver on GitHub (May 30, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10916 ### What is the issue? Having tried small (8B) and large (70B) models alike, thinking &/or vision models, and currently updated to version 0.9.0, I have not been able to ever get Ollama to fully utilize the **two 3090s** on my machine. Even when "ollama ps" says it's using GPU 100%, task manager and process explorer (and the fans on my PC) suggest otherwise. Sometimes there's a brief spike in usage that quickly falls away and most all of the activity is seen on the CPU. Maybe it's me... Is there some specific steps that need to be followed to ensure maximum utilization of the GPUs? Do I need to buy an NVLink, apply some special global configuration... feels like a waste. Thanks in advance for any help in getting this right... ### Relevant log output ```shell ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.9.0
GiteaMirror added the bug label 2026-04-12 19:10:45 -05:00
GiteaMirror commented 2026-04-12 19:10:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 30, 2025):

Server logs may aid in debugging.

<!-- gh-comment-id:2922704283 --> @rick-github commented on GitHub (May 30, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging.
GiteaMirror commented 2026-04-12 19:10:47 -05:00
Author
Owner
Copy Link

@wingraver commented on GitHub (May 30, 2025):

[GIN] 2025/05/30 - 10:55:39 | 200 | 0s | 127.0.0.1 | HEAD "/"
[GIN] 2025/05/30 - 10:55:39 | 200 | 7.0382ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/05/30 - 10:56:02 | 200 | 0s | 127.0.0.1 | HEAD "/"
[GIN] 2025/05/30 - 10:56:02 | 200 | 47.2215ms | 127.0.0.1 | POST "/api/show"
time=2025-05-30T10:56:02.910-05:00 level=INFO source=sched.go:804 msg="new model will fit in available VRAM, loading" model=C:\Users\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 library=cuda parallel=2 required="41.2 GiB"
time=2025-05-30T10:56:02.943-05:00 level=INFO source=server.go:135 msg="system memory" total="95.8 GiB" free="72.6 GiB" free_swap="71.7 GiB"
time=2025-05-30T10:56:02.944-05:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=33,32 memory.available="[22.8 GiB 22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="41.2 GiB" memory.required.partial="41.2 GiB" memory.required.kv="2.0 GiB" memory.required.allocations="[21.0 GiB 20.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB" memory.weights.nonrepeating="788.3 MiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB"
llama_model_loader: loaded meta data with 27 key-value pairs and 707 tensors from C:\Users\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen3
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen3 32B
llama_model_loader: - kv 3: general.basename str = Qwen3
llama_model_loader: - kv 4: general.size_label str = 32B
llama_model_loader: - kv 5: qwen3.block_count u32 = 64
llama_model_loader: - kv 6: qwen3.context_length u32 = 40960
llama_model_loader: - kv 7: qwen3.embedding_length u32 = 5120
llama_model_loader: - kv 8: qwen3.feed_forward_length u32 = 25600
llama_model_loader: - kv 9: qwen3.attention.head_count u32 = 64
llama_model_loader: - kv 10: qwen3.attention.head_count_kv u32 = 8
llama_model_loader: - kv 11: qwen3.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 13: qwen3.attention.key_length u32 = 128
llama_model_loader: - kv 14: qwen3.attention.value_length u32 = 128
llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,151936] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - kv 26: general.file_type u32 = 7
llama_model_loader: - type f32: 257 tensors
llama_model_loader: - type f16: 64 tensors
llama_model_loader: - type q8_0: 386 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 32.71 GiB (8.58 BPW)
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch = qwen3
print_info: vocab_only = 1
print_info: model type = ?B
print_info: model params = 32.76 B
print_info: general.name = Qwen3 32B
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-05-30T10:56:03.111-05:00 level=INFO source=server.go:431 msg="starting llama server" cmd="C:\Users\\AppData\Local\Programs\Ollama\ollama.exe runner --model C:\Users\\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 --ctx-size 8192 --batch-size 512 --n-gpu-layers 65 --threads 32 --no-mmap --parallel 2 --tensor-split 33,32 --port 64789"
time=2025-05-30T10:56:03.116-05:00 level=INFO source=sched.go:483 msg="loaded runners" count=1
time=2025-05-30T10:56:03.116-05:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2025-05-30T10:56:03.116-05:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
time=2025-05-30T10:56:03.181-05:00 level=INFO source=runner.go:815 msg="starting go runner"
load_backend: loaded CPU backend from C:\Users\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from C:\Users\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
time=2025-05-30T10:56:03.556-05:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-05-30T10:56:03.558-05:00 level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:64789"
time=2025-05-30T10:56:03.618-05:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23306 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23306 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 707 tensors from C:\Users\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen3
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen3 32B
llama_model_loader: - kv 3: general.basename str = Qwen3
llama_model_loader: - kv 4: general.size_label str = 32B
llama_model_loader: - kv 5: qwen3.block_count u32 = 64
llama_model_loader: - kv 6: qwen3.context_length u32 = 40960
llama_model_loader: - kv 7: qwen3.embedding_length u32 = 5120
llama_model_loader: - kv 8: qwen3.feed_forward_length u32 = 25600
llama_model_loader: - kv 9: qwen3.attention.head_count u32 = 64
llama_model_loader: - kv 10: qwen3.attention.head_count_kv u32 = 8
llama_model_loader: - kv 11: qwen3.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 13: qwen3.attention.key_length u32 = 128
llama_model_loader: - kv 14: qwen3.attention.value_length u32 = 128
llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,151936] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - kv 26: general.file_type u32 = 7
llama_model_loader: - type f32: 257 tensors
llama_model_loader: - type f16: 64 tensors
llama_model_loader: - type q8_0: 386 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 32.71 GiB (8.58 BPW)
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch = qwen3
print_info: vocab_only = 0
print_info: n_ctx_train = 40960
print_info: n_embd = 5120
print_info: n_layer = 64
print_info: n_head = 64
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: n_swa_pattern = 1
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 25600
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 40960
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 32B
print_info: model params = 32.76 B
print_info: general.name = Qwen3 32B
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 64 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors: CUDA_Host model buffer size = 788.24 MiB
load_tensors: CUDA0 model buffer size = 16460.07 MiB
load_tensors: CUDA1 model buffer size = 16250.75 MiB
llama_context: constructing llama_context
llama_context: n_seq_max = 2
llama_context: n_ctx = 8192
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch = 1024
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 1000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 1.20 MiB
llama_kv_cache_unified: kv_size = 8192, type_k = 'f16', type_v = 'f16', n_layer = 64, can_shift = 1, padding = 32
llama_kv_cache_unified: CUDA0 KV buffer size = 1056.00 MiB
llama_kv_cache_unified: CUDA1 KV buffer size = 992.00 MiB
llama_kv_cache_unified: KV self size = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: CUDA0 compute buffer size = 1180.01 MiB
llama_context: CUDA1 compute buffer size = 1180.02 MiB
llama_context: CUDA_Host compute buffer size = 74.02 MiB
llama_context: graph nodes = 2438
llama_context: graph splits = 3
time=2025-05-30T10:56:16.401-05:00 level=INFO source=server.go:630 msg="llama runner started in 13.28 seconds"
[GIN] 2025/05/30 - 10:56:16 | 200 | 13.5785235s | 127.0.0.1 | POST "/api/generate"
[GIN] 2025/05/30 - 10:57:57 | 200 | 1m2s | 127.0.0.1 | POST "/api/chat"

@rick-github this it?

<!-- gh-comment-id:2922843510 --> @wingraver commented on GitHub (May 30, 2025): [GIN] 2025/05/30 - 10:55:39 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/05/30 - 10:55:39 | 200 | 7.0382ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/05/30 - 10:56:02 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/05/30 - 10:56:02 | 200 | 47.2215ms | 127.0.0.1 | POST "/api/show" time=2025-05-30T10:56:02.910-05:00 level=INFO source=sched.go:804 msg="new model will fit in available VRAM, loading" model=C:\Users\\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 library=cuda parallel=2 required="41.2 GiB" time=2025-05-30T10:56:02.943-05:00 level=INFO source=server.go:135 msg="system memory" total="95.8 GiB" free="72.6 GiB" free_swap="71.7 GiB" time=2025-05-30T10:56:02.944-05:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=33,32 memory.available="[22.8 GiB 22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="41.2 GiB" memory.required.partial="41.2 GiB" memory.required.kv="2.0 GiB" memory.required.allocations="[21.0 GiB 20.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB" memory.weights.nonrepeating="788.3 MiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB" llama_model_loader: loaded meta data with 27 key-value pairs and 707 tensors from C:\Users\\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen3 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen3 32B llama_model_loader: - kv 3: general.basename str = Qwen3 llama_model_loader: - kv 4: general.size_label str = 32B llama_model_loader: - kv 5: qwen3.block_count u32 = 64 llama_model_loader: - kv 6: qwen3.context_length u32 = 40960 llama_model_loader: - kv 7: qwen3.embedding_length u32 = 5120 llama_model_loader: - kv 8: qwen3.feed_forward_length u32 = 25600 llama_model_loader: - kv 9: qwen3.attention.head_count u32 = 64 llama_model_loader: - kv 10: qwen3.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: qwen3.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 12: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 13: qwen3.attention.key_length u32 = 128 llama_model_loader: - kv 14: qwen3.attention.value_length u32 = 128 llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645 llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... llama_model_loader: - kv 25: general.quantization_version u32 = 2 llama_model_loader: - kv 26: general.file_type u32 = 7 llama_model_loader: - type f32: 257 tensors llama_model_loader: - type f16: 64 tensors llama_model_loader: - type q8_0: 386 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q8_0 print_info: file size = 32.71 GiB (8.58 BPW) load: special tokens cache size = 26 load: token to piece cache size = 0.9311 MB print_info: arch = qwen3 print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 32.76 B print_info: general.name = Qwen3 32B print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = 151643 '<|endoftext|>' print_info: EOS token = 151645 '<|im_end|>' print_info: EOT token = 151645 '<|im_end|>' print_info: PAD token = 151643 '<|endoftext|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151659 '<|fim_prefix|>' print_info: FIM SUF token = 151661 '<|fim_suffix|>' print_info: FIM MID token = 151660 '<|fim_middle|>' print_info: FIM PAD token = 151662 '<|fim_pad|>' print_info: FIM REP token = 151663 '<|repo_name|>' print_info: FIM SEP token = 151664 '<|file_sep|>' print_info: EOG token = 151643 '<|endoftext|>' print_info: EOG token = 151645 '<|im_end|>' print_info: EOG token = 151662 '<|fim_pad|>' print_info: EOG token = 151663 '<|repo_name|>' print_info: EOG token = 151664 '<|file_sep|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-05-30T10:56:03.111-05:00 level=INFO source=server.go:431 msg="starting llama server" cmd="C:\\Users\\\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\\\.ollama\\models\\blobs\\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 --ctx-size 8192 --batch-size 512 --n-gpu-layers 65 --threads 32 --no-mmap --parallel 2 --tensor-split 33,32 --port 64789" time=2025-05-30T10:56:03.116-05:00 level=INFO source=sched.go:483 msg="loaded runners" count=1 time=2025-05-30T10:56:03.116-05:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2025-05-30T10:56:03.116-05:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" time=2025-05-30T10:56:03.181-05:00 level=INFO source=runner.go:815 msg="starting go runner" load_backend: loaded CPU backend from C:\Users\\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from C:\Users\\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll time=2025-05-30T10:56:03.556-05:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-05-30T10:56:03.558-05:00 level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:64789" time=2025-05-30T10:56:03.618-05:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23306 MiB free llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23306 MiB free llama_model_loader: loaded meta data with 27 key-value pairs and 707 tensors from C:\Users\\.ollama\models\blobs\sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen3 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen3 32B llama_model_loader: - kv 3: general.basename str = Qwen3 llama_model_loader: - kv 4: general.size_label str = 32B llama_model_loader: - kv 5: qwen3.block_count u32 = 64 llama_model_loader: - kv 6: qwen3.context_length u32 = 40960 llama_model_loader: - kv 7: qwen3.embedding_length u32 = 5120 llama_model_loader: - kv 8: qwen3.feed_forward_length u32 = 25600 llama_model_loader: - kv 9: qwen3.attention.head_count u32 = 64 llama_model_loader: - kv 10: qwen3.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: qwen3.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 12: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 13: qwen3.attention.key_length u32 = 128 llama_model_loader: - kv 14: qwen3.attention.value_length u32 = 128 llama_model_loader: - kv 15: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 16: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 17: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 19: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 151645 llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... llama_model_loader: - kv 25: general.quantization_version u32 = 2 llama_model_loader: - kv 26: general.file_type u32 = 7 llama_model_loader: - type f32: 257 tensors llama_model_loader: - type f16: 64 tensors llama_model_loader: - type q8_0: 386 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q8_0 print_info: file size = 32.71 GiB (8.58 BPW) load: special tokens cache size = 26 load: token to piece cache size = 0.9311 MB print_info: arch = qwen3 print_info: vocab_only = 0 print_info: n_ctx_train = 40960 print_info: n_embd = 5120 print_info: n_layer = 64 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: n_swa_pattern = 1 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 25600 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 2 print_info: rope scaling = linear print_info: freq_base_train = 1000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 40960 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 32B print_info: model params = 32.76 B print_info: general.name = Qwen3 32B print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = 151643 '<|endoftext|>' print_info: EOS token = 151645 '<|im_end|>' print_info: EOT token = 151645 '<|im_end|>' print_info: PAD token = 151643 '<|endoftext|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151659 '<|fim_prefix|>' print_info: FIM SUF token = 151661 '<|fim_suffix|>' print_info: FIM MID token = 151660 '<|fim_middle|>' print_info: FIM PAD token = 151662 '<|fim_pad|>' print_info: FIM REP token = 151663 '<|repo_name|>' print_info: FIM SEP token = 151664 '<|file_sep|>' print_info: EOG token = 151643 '<|endoftext|>' print_info: EOG token = 151645 '<|im_end|>' print_info: EOG token = 151662 '<|fim_pad|>' print_info: EOG token = 151663 '<|repo_name|>' print_info: EOG token = 151664 '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = false) load_tensors: offloading 64 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 65/65 layers to GPU load_tensors: CUDA_Host model buffer size = 788.24 MiB load_tensors: CUDA0 model buffer size = 16460.07 MiB load_tensors: CUDA1 model buffer size = 16250.75 MiB llama_context: constructing llama_context llama_context: n_seq_max = 2 llama_context: n_ctx = 8192 llama_context: n_ctx_per_seq = 4096 llama_context: n_batch = 1024 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = 0 llama_context: freq_base = 1000000.0 llama_context: freq_scale = 1 llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized llama_context: CUDA_Host output buffer size = 1.20 MiB llama_kv_cache_unified: kv_size = 8192, type_k = 'f16', type_v = 'f16', n_layer = 64, can_shift = 1, padding = 32 llama_kv_cache_unified: CUDA0 KV buffer size = 1056.00 MiB llama_kv_cache_unified: CUDA1 KV buffer size = 992.00 MiB llama_kv_cache_unified: KV self size = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB llama_context: pipeline parallelism enabled (n_copies=4) llama_context: CUDA0 compute buffer size = 1180.01 MiB llama_context: CUDA1 compute buffer size = 1180.02 MiB llama_context: CUDA_Host compute buffer size = 74.02 MiB llama_context: graph nodes = 2438 llama_context: graph splits = 3 time=2025-05-30T10:56:16.401-05:00 level=INFO source=server.go:630 msg="llama runner started in 13.28 seconds" [GIN] 2025/05/30 - 10:56:16 | 200 | 13.5785235s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/05/30 - 10:57:57 | 200 | 1m2s | 127.0.0.1 | POST "/api/chat" @rick-github this it?
GiteaMirror commented 2026-04-12 19:10:48 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 30, 2025):

time=2025-05-30T10:56:02.944-05:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1
 layers.model=65 layers.offload=65 layers.split=33,32 memory.available="[22.8 GiB 22.7 GiB]" memory.gpu_overhead="0 B"
 memory.required.full="41.2 GiB" memory.required.partial="41.2 GiB" memory.required.kv="2.0 GiB"
 memory.required.allocations="[21.0 GiB 20.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB"
 memory.weights.nonrepeating="788.3 MiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB"

ollama has estimated using 41.G of 45.5G to load the model in the GPU, so has calculated that it can offload all 65 of the models 65 layers.

load_tensors: offloading 64 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors: CUDA0 model buffer size = 16460.07 MiB
load_tensors: CUDA1 model buffer size = 16250.75 MiB

The runner has loaded all layers into the GPU. All indications are that the model is running 100% on the GPU.

Even when "ollama ps" says it's using GPU 100%, task manager and process explorer (and the fans on my PC) suggest otherwise.

Note that while the GPUs are doing inference, the CPU is the controller - it is busy sending instructions to the GPUs on what kernels to launch, moving intermediate results between the GPUs, etc. So the CPU will still be busy even when the GPUs are the only devices processing the model weights.

<!-- gh-comment-id:2922857855 --> @rick-github commented on GitHub (May 30, 2025): ``` time=2025-05-30T10:56:02.944-05:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=33,32 memory.available="[22.8 GiB 22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="41.2 GiB" memory.required.partial="41.2 GiB" memory.required.kv="2.0 GiB" memory.required.allocations="[21.0 GiB 20.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB" memory.weights.nonrepeating="788.3 MiB" memory.graph.full="2.7 GiB" memory.graph.partial="2.7 GiB" ``` ollama has estimated using 41.G of 45.5G to load the model in the GPU, so has calculated that it can offload all 65 of the models 65 layers. ``` load_tensors: offloading 64 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 65/65 layers to GPU load_tensors: CUDA0 model buffer size = 16460.07 MiB load_tensors: CUDA1 model buffer size = 16250.75 MiB ``` The runner has loaded all layers into the GPU. All indications are that the model is running 100% on the GPU. > Even when "ollama ps" says it's using GPU 100%, task manager and process explorer (and the fans on my PC) suggest otherwise. Note that while the GPUs are doing inference, the CPU is the controller - it is busy sending instructions to the GPUs on what kernels to launch, moving intermediate results between the GPUs, etc. So the CPU will still be busy even when the GPUs are the only devices processing the model weights.
GiteaMirror commented 2026-04-12 19:10:49 -05:00
Author
Owner
Copy Link

@wingraver commented on GitHub (May 30, 2025):

Thanks for this additional info @rick-github, what I don't get though is when processing any amount of time, task manager shows the second GPU with 0% utilization and first GPU has a low percentage (less than 10 in most cases) while the CPU is humming away at over 10% depending on model size and question asked. I always figured it would be more along the line of 80% or more for both GPUs while CPU utilization is less than 10%.

What scenarios/models effectively utilize over 80% of two 3090s where Ollama is concerned?

<!-- gh-comment-id:2922873726 --> @wingraver commented on GitHub (May 30, 2025): Thanks for this additional info @rick-github, what I don't get though is when processing any amount of time, task manager shows the second GPU with 0% utilization and first GPU has a low percentage (less than 10 in most cases) while the CPU is humming away at over 10% depending on model size and question asked. I always figured it would be more along the line of 80% or more for both GPUs while CPU utilization is less than 10%. What scenarios/models effectively utilize over 80% of two 3090s where Ollama is concerned?
GiteaMirror commented 2026-04-12 19:10:49 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 30, 2025):

You won't get 80% on both GPUs at the same time for a single inference: https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990.

But 10% is low. 3090s have a PCIe 4.0 x16 interface, what does your motherboard have?

<!-- gh-comment-id:2922888896 --> @rick-github commented on GitHub (May 30, 2025): You won't get 80% on both GPUs at the same time for a single inference: https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990. But 10% is low. 3090s have a PCIe 4.0 x16 interface, what does your motherboard have?
GiteaMirror commented 2026-04-12 19:10:51 -05:00
Author
Owner
Copy Link

@wingraver commented on GitHub (Jun 4, 2025):

Nope can't be right! Just now running gemma3 27b model, the estimated size is 50GB (via ollama ps) with a 9%/91% split CPU/GPU. Switched over to Task Manager and Process Explorer to see again all this activity is taking place between the CPU and only one of the GPU. I set the environment variable to OLLAMA_GPU_COUNT = 2 and OLLAMA_GPU_MEMORY_LIMIT = 48 (two GPUs @ 24GB each). It just doesn't make sense that the application cannot leverage multiple gpus to serve LLMs.

<!-- gh-comment-id:2941775683 --> @wingraver commented on GitHub (Jun 4, 2025): Nope can't be right! Just now running gemma3 27b model, the estimated size is 50GB (via ollama ps) with a 9%/91% split CPU/GPU. Switched over to Task Manager and Process Explorer to see again all this activity is taking place between the CPU and **only one of the GPU**. I set the environment variable to OLLAMA_GPU_COUNT = 2 and OLLAMA_GPU_MEMORY_LIMIT = 48 (two GPUs @ 24GB each). It just doesn't make sense that the application cannot leverage multiple gpus to serve LLMs.
GiteaMirror commented 2026-04-12 19:10:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 4, 2025):

Server logs may aid in debugging.

OLLAMA_GPU_COUNT and OLLAMA_GPU_MEMORY_LIMIT are not ollama configuration variables.

<!-- gh-comment-id:2941887548 --> @rick-github commented on GitHub (Jun 4, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging. `OLLAMA_GPU_COUNT` and `OLLAMA_GPU_MEMORY_LIMIT` are not ollama configuration variables.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7181
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?