github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-13 17:34:40 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8394] The same model could load all onto the GPU last year. Today, after upgrading to ollama, I found that it cannot be loaded onto the GPU all at once. #83075

New Issue
Closed
opened 2026-05-09 17:08:16 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-05-09 17:08:16 -05:00
Owner
Copy Link
Copy Source

Originally created by @21307369 on GitHub (Jan 12, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8394

What is the issue?

The situation is as follows, version 0.2.1ollama can fully load the GPU with codegeex4:latest. However, after upgrading to 0.5.4 today, I am unable to fully load the GPU. My GPU is a 6750GRE with 12GB of VRAM. I found something wrong with this... The size of this model is 5.5G. Observing the resource usage, the VRAM usage is around 3G, with the rest in memory.
I am using a smaller model, hhao/qwen2.5-coder-tools:3b, which is 1.19GB in size. It will indeed occupy all the GPU memory. The speed is also normal.
hhao/qwen2.5-coder-tools:3b
image
image
codegeex4:
image

OS

Windows

GPU

AMD

CPU

Intel

Ollama version

0.5.4-0-g08b8916

Originally created by @21307369 on GitHub (Jan 12, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8394 ### What is the issue? The situation is as follows, version 0.2.1ollama can fully load the GPU with codegeex4:latest. However, after upgrading to 0.5.4 today, I am unable to fully load the GPU. My GPU is a 6750GRE with 12GB of VRAM. I found something wrong with this... The size of this model is 5.5G. Observing the resource usage, the VRAM usage is around 3G, with the rest in memory. I am using a smaller model, hhao/qwen2.5-coder-tools:3b, which is 1.19GB in size. It will indeed occupy all the GPU memory. The speed is also normal. hhao/qwen2.5-coder-tools:3b ![image](https://github.com/user-attachments/assets/14f93224-af6e-4a1a-916b-9ee5ec268f3b) ![image](https://github.com/user-attachments/assets/b89c3b08-c186-4428-8e3a-2013fbf259b8) codegeex4: ![image](https://github.com/user-attachments/assets/4f2fd3c6-36b6-44d4-a252-db8dc6ee2aae) ### OS Windows ### GPU AMD ### CPU Intel ### Ollama version 0.5.4-0-g08b8916
GiteaMirror added the bug label 2026-05-09 17:08:16 -05:00
GiteaMirror commented 2026-05-09 17:08:20 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Jan 12, 2025):

Server logs will aid in debugging.

<!-- gh-comment-id:2585624292 --> @rick-github commented on GitHub (Jan 12, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
GiteaMirror commented 2026-05-09 17:08:21 -05:00
Author
Owner
Copy Link
Copy Source

@21307369 commented on GitHub (Jan 12, 2025):

2025/01/12 15:43:13 routes.go:1259: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:E:\.ollama\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-01-12T15:43:13.841+08:00 level=INFO source=images.go:757 msg="total blobs: 15"
time=2025-01-12T15:43:13.842+08:00 level=INFO source=images.go:764 msg="total unused blobs removed: 0"
time=2025-01-12T15:43:13.846+08:00 level=INFO source=routes.go:1310 msg="Listening on 127.0.0.1:11434 (version 0.5.4-0-g08b8916)"
time=2025-01-12T15:43:13.847+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm_avx]"
time=2025-01-12T15:43:13.847+08:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-01-12T15:43:13.848+08:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-01-12T15:43:13.848+08:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=6 efficiency=0 threads=6
time=2025-01-12T15:43:14.380+08:00 level=INFO source=types.go:131 msg="inference compute" id=0 library=rocm variant="" compute=gfx1031 driver=6.2 name="AMD Radeon RX 6750 GRE 12GB" total="12.0 GiB" available="11.8 GiB"
time=2025-01-12T15:43:38.314+08:00 level=INFO source=sched.go:185 msg="one or more GPUs detected that are unable to accurately report free memory - disabling default concurrency"
time=2025-01-12T15:43:38.348+08:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=E:.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 gpu=0 parallel=4 available=12573671424 required="6.2 GiB"
time=2025-01-12T15:43:38.934+08:00 level=INFO source=server.go:104 msg="system memory" total="31.9 GiB" free="22.7 GiB" free_swap="16.6 GiB"
time=2025-01-12T15:43:38.934+08:00 level=INFO source=memory.go:356 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[11.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="320.0 MiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.6 GiB" memory.weights.repeating="4.1 GiB" memory.weights.nonrepeating="485.6 MiB" memory.graph.full="561.0 MiB" memory.graph.partial="789.6 MiB"time=2025-01-12T15:43:38.942+08:00 level=INFO source=server.go:376 msg="starting llama server" cmd="E:\DOWN\OLLAMA\windows-amd64\lib\ollama\runners\rocm_avx\ollama_llama_server.exe runner --model E:\.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 --ctx-size 8192 --batch-size 512 --n-gpu-layers 41 --threads 6 --parallel 4 --port 20214"
time=2025-01-12T15:43:38.943+08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-01-12T15:43:38.943+08:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-01-12T15:43:38.944+08:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-01-12T15:43:38.968+08:00 level=INFO source=runner.go:938 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 6750 GRE 12GB, compute capability 10.3, VMM: no
time=2025-01-12T15:43:40.648+08:00 level=INFO source=runner.go:939 msg=system info="ROCm : NO_PEER_COPY = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(clang)" threads=6
llama_load_model_from_file: using device ROCm0 (AMD Radeon RX 6750 GRE 12GB) - 12036 MiB free
time=2025-01-12T15:43:40.648+08:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:20214"
llama_model_loader: loaded meta data with 23 key-value pairs and 283 tensors from E:.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = chatglm
llama_model_loader: - kv 1: general.name str = codegeex4-all-9b
llama_model_loader: - kv 2: chatglm.context_length u32 = 131072
llama_model_loader: - kv 3: chatglm.embedding_length u32 = 4096
llama_model_loader: - kv 4: chatglm.feed_forward_length u32 = 13696
llama_model_loader: - kv 5: chatglm.block_count u32 = 40
llama_model_loader: - kv 6: chatglm.attention.head_count u32 = 32
llama_model_loader: - kv 7: chatglm.attention.head_count_kv u32 = 2
llama_model_loader: - kv 8: chatglm.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 9: general.file_type u32 = 2
llama_model_loader: - kv 10: chatglm.rope.dimension_count u32 = 64
llama_model_loader: - kv 11: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 12: chatglm.rope.freq_base f32 = 5000000.000000
llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 14: tokenizer.ggml.pre str = chatglm-bpe
time=2025-01-12T15:43:40.701+08:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,151552] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,151552] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,151073] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 18: tokenizer.ggml.padding_token_id u32 = 151329
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 151329
llama_model_loader: - kv 20: tokenizer.ggml.eot_token_id u32 = 151336
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 151329
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q4_0: 161 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 223
llm_load_vocab: token to piece cache size = 0.9732 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = chatglm
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 151552
llm_load_print_meta: n_merges = 151073
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 2
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 16
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 13696
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 9B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 9.40 B
llm_load_print_meta: model size = 5.08 GiB (4.64 BPW)
llm_load_print_meta: general.name = codegeex4-all-9b
llm_load_print_meta: EOS token = 151329 '<|endoftext|>'
llm_load_print_meta: EOT token = 151336 '<|user|>'
llm_load_print_meta: UNK token = 151329 '<|endoftext|>'
llm_load_print_meta: PAD token = 151329 '<|endoftext|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOG token = 151329 '<|endoftext|>'
llm_load_print_meta: EOG token = 151336 '<|user|>'
llm_load_print_meta: max token length = 1024
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors: CPU_Mapped model buffer size = 333.00 MiB
llm_load_tensors: ROCm0 model buffer size = 4863.85 MiB
llama_new_context_with_model: n_seq_max = 4
llama_new_context_with_model: n_ctx = 8192
llama_new_context_with_model: n_ctx_per_seq = 2048
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 5000000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: ROCm0 KV buffer size = 320.00 MiB
llama_new_context_with_model: KV self size = 320.00 MiB, K (f16): 160.00 MiB, V (f16): 160.00 MiB
llama_new_context_with_model: ROCm_Host output buffer size = 2.38 MiB
llama_new_context_with_model: ROCm0 compute buffer size = 561.00 MiB
llama_new_context_with_model: ROCm_Host compute buffer size = 24.01 MiB
llama_new_context_with_model: graph nodes = 1606
llama_new_context_with_model: graph splits = 2
time=2025-01-12T15:43:44.460+08:00 level=INFO source=server.go:594 msg="llama runner started in 5.52 seconds"
llama_model_loader: loaded meta data with 23 key-value pairs and 283 tensors from E:.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = chatglm
llama_model_loader: - kv 1: general.name str = codegeex4-all-9b
llama_model_loader: - kv 2: chatglm.context_length u32 = 131072
llama_model_loader: - kv 3: chatglm.embedding_length u32 = 4096
llama_model_loader: - kv 4: chatglm.feed_forward_length u32 = 13696
llama_model_loader: - kv 5: chatglm.block_count u32 = 40
llama_model_loader: - kv 6: chatglm.attention.head_count u32 = 32
llama_model_loader: - kv 7: chatglm.attention.head_count_kv u32 = 2
llama_model_loader: - kv 8: chatglm.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 9: general.file_type u32 = 2
llama_model_loader: - kv 10: chatglm.rope.dimension_count u32 = 64
llama_model_loader: - kv 11: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 12: chatglm.rope.freq_base f32 = 5000000.000000
llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 14: tokenizer.ggml.pre str = chatglm-bpe
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,151552] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,151552] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,151073] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 18: tokenizer.ggml.padding_token_id u32 = 151329
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 151329
llama_model_loader: - kv 20: tokenizer.ggml.eot_token_id u32 = 151336
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 151329
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q4_0: 161 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 223
llm_load_vocab: token to piece cache size = 0.9732 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = chatglm
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 151552
llm_load_print_meta: n_merges = 151073
llm_load_print_meta: vocab_only = 1
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = all F32
llm_load_print_meta: model params = 9.40 B
llm_load_print_meta: model size = 5.08 GiB (4.64 BPW)
llm_load_print_meta: general.name = codegeex4-all-9b
llm_load_print_meta: EOS token = 151329 '<|endoftext|>'
llm_load_print_meta: EOT token = 151336 '<|user|>'
llm_load_print_meta: UNK token = 151329 '<|endoftext|>'
llm_load_print_meta: PAD token = 151329 '<|endoftext|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOG token = 151329 '<|endoftext|>'
llm_load_print_meta: EOG token = 151336 '<|user|>'
llm_load_print_meta: max token length = 1024
llama_model_load: vocab only - skipping tensors

<!-- gh-comment-id:2585626924 --> @21307369 commented on GitHub (Jan 12, 2025): 2025/01/12 15:43:13 routes.go:1259: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:E:\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-01-12T15:43:13.841+08:00 level=INFO source=images.go:757 msg="total blobs: 15" time=2025-01-12T15:43:13.842+08:00 level=INFO source=images.go:764 msg="total unused blobs removed: 0" time=2025-01-12T15:43:13.846+08:00 level=INFO source=routes.go:1310 msg="Listening on 127.0.0.1:11434 (version 0.5.4-0-g08b8916)" time=2025-01-12T15:43:13.847+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm_avx]" time=2025-01-12T15:43:13.847+08:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-01-12T15:43:13.848+08:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-01-12T15:43:13.848+08:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=6 efficiency=0 threads=6 time=2025-01-12T15:43:14.380+08:00 level=INFO source=types.go:131 msg="inference compute" id=0 library=rocm variant="" compute=gfx1031 driver=6.2 name="AMD Radeon RX 6750 GRE 12GB" total="12.0 GiB" available="11.8 GiB" time=2025-01-12T15:43:38.314+08:00 level=INFO source=sched.go:185 msg="one or more GPUs detected that are unable to accurately report free memory - disabling default concurrency" time=2025-01-12T15:43:38.348+08:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=E:\.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 gpu=0 parallel=4 available=12573671424 required="6.2 GiB" time=2025-01-12T15:43:38.934+08:00 level=INFO source=server.go:104 msg="system memory" total="31.9 GiB" free="22.7 GiB" free_swap="16.6 GiB" time=2025-01-12T15:43:38.934+08:00 level=INFO source=memory.go:356 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[11.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="320.0 MiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.6 GiB" memory.weights.repeating="4.1 GiB" memory.weights.nonrepeating="485.6 MiB" memory.graph.full="561.0 MiB" memory.graph.partial="789.6 MiB"time=2025-01-12T15:43:38.942+08:00 level=INFO source=server.go:376 msg="starting llama server" cmd="E:\\DOWN\\OLLAMA\\windows-amd64\\lib\\ollama\\runners\\rocm_avx\\ollama_llama_server.exe runner --model E:\\.ollama\\models\\blobs\\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 --ctx-size 8192 --batch-size 512 --n-gpu-layers 41 --threads 6 --parallel 4 --port 20214" time=2025-01-12T15:43:38.943+08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-01-12T15:43:38.943+08:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" time=2025-01-12T15:43:38.944+08:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" time=2025-01-12T15:43:38.968+08:00 level=INFO source=runner.go:938 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 6750 GRE 12GB, compute capability 10.3, VMM: no time=2025-01-12T15:43:40.648+08:00 level=INFO source=runner.go:939 msg=system info="ROCm : NO_PEER_COPY = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(clang)" threads=6 llama_load_model_from_file: using device ROCm0 (AMD Radeon RX 6750 GRE 12GB) - 12036 MiB free time=2025-01-12T15:43:40.648+08:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:20214" llama_model_loader: loaded meta data with 23 key-value pairs and 283 tensors from E:\.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = chatglm llama_model_loader: - kv 1: general.name str = codegeex4-all-9b llama_model_loader: - kv 2: chatglm.context_length u32 = 131072 llama_model_loader: - kv 3: chatglm.embedding_length u32 = 4096 llama_model_loader: - kv 4: chatglm.feed_forward_length u32 = 13696 llama_model_loader: - kv 5: chatglm.block_count u32 = 40 llama_model_loader: - kv 6: chatglm.attention.head_count u32 = 32 llama_model_loader: - kv 7: chatglm.attention.head_count_kv u32 = 2 llama_model_loader: - kv 8: chatglm.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 9: general.file_type u32 = 2 llama_model_loader: - kv 10: chatglm.rope.dimension_count u32 = 64 llama_model_loader: - kv 11: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 12: chatglm.rope.freq_base f32 = 5000000.000000 llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 14: tokenizer.ggml.pre str = chatglm-bpe time=2025-01-12T15:43:40.701+08:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,151552] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,151552] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,151073] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 18: tokenizer.ggml.padding_token_id u32 = 151329 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 151329 llama_model_loader: - kv 20: tokenizer.ggml.eot_token_id u32 = 151336 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 151329 llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 121 tensors llama_model_loader: - type q4_0: 161 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 223 llm_load_vocab: token to piece cache size = 0.9732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = chatglm llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 151552 llm_load_print_meta: n_merges = 151073 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 2 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 16 llm_load_print_meta: n_embd_k_gqa = 256 llm_load_print_meta: n_embd_v_gqa = 256 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 13696 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 5000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 9B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 9.40 B llm_load_print_meta: model size = 5.08 GiB (4.64 BPW) llm_load_print_meta: general.name = codegeex4-all-9b llm_load_print_meta: EOS token = 151329 '<|endoftext|>' llm_load_print_meta: EOT token = 151336 '<|user|>' llm_load_print_meta: UNK token = 151329 '<|endoftext|>' llm_load_print_meta: PAD token = 151329 '<|endoftext|>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 151329 '<|endoftext|>' llm_load_print_meta: EOG token = 151336 '<|user|>' llm_load_print_meta: max token length = 1024 llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloading output layer to GPU llm_load_tensors: offloaded 41/41 layers to GPU llm_load_tensors: CPU_Mapped model buffer size = 333.00 MiB llm_load_tensors: ROCm0 model buffer size = 4863.85 MiB llama_new_context_with_model: n_seq_max = 4 llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: n_ctx_per_seq = 2048 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 5000000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_kv_cache_init: ROCm0 KV buffer size = 320.00 MiB llama_new_context_with_model: KV self size = 320.00 MiB, K (f16): 160.00 MiB, V (f16): 160.00 MiB llama_new_context_with_model: ROCm_Host output buffer size = 2.38 MiB llama_new_context_with_model: ROCm0 compute buffer size = 561.00 MiB llama_new_context_with_model: ROCm_Host compute buffer size = 24.01 MiB llama_new_context_with_model: graph nodes = 1606 llama_new_context_with_model: graph splits = 2 time=2025-01-12T15:43:44.460+08:00 level=INFO source=server.go:594 msg="llama runner started in 5.52 seconds" llama_model_loader: loaded meta data with 23 key-value pairs and 283 tensors from E:\.ollama\models\blobs\sha256-816441b33390807d429fbdb1de7e33bb4d569ac68e2203bdbca5d8d79b5c7266 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = chatglm llama_model_loader: - kv 1: general.name str = codegeex4-all-9b llama_model_loader: - kv 2: chatglm.context_length u32 = 131072 llama_model_loader: - kv 3: chatglm.embedding_length u32 = 4096 llama_model_loader: - kv 4: chatglm.feed_forward_length u32 = 13696 llama_model_loader: - kv 5: chatglm.block_count u32 = 40 llama_model_loader: - kv 6: chatglm.attention.head_count u32 = 32 llama_model_loader: - kv 7: chatglm.attention.head_count_kv u32 = 2 llama_model_loader: - kv 8: chatglm.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 9: general.file_type u32 = 2 llama_model_loader: - kv 10: chatglm.rope.dimension_count u32 = 64 llama_model_loader: - kv 11: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 12: chatglm.rope.freq_base f32 = 5000000.000000 llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 14: tokenizer.ggml.pre str = chatglm-bpe llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,151552] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,151552] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,151073] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 18: tokenizer.ggml.padding_token_id u32 = 151329 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 151329 llama_model_loader: - kv 20: tokenizer.ggml.eot_token_id u32 = 151336 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 151329 llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 121 tensors llama_model_loader: - type q4_0: 161 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 223 llm_load_vocab: token to piece cache size = 0.9732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = chatglm llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 151552 llm_load_print_meta: n_merges = 151073 llm_load_print_meta: vocab_only = 1 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = all F32 llm_load_print_meta: model params = 9.40 B llm_load_print_meta: model size = 5.08 GiB (4.64 BPW) llm_load_print_meta: general.name = codegeex4-all-9b llm_load_print_meta: EOS token = 151329 '<|endoftext|>' llm_load_print_meta: EOT token = 151336 '<|user|>' llm_load_print_meta: UNK token = 151329 '<|endoftext|>' llm_load_print_meta: PAD token = 151329 '<|endoftext|>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 151329 '<|endoftext|>' llm_load_print_meta: EOG token = 151336 '<|user|>' llm_load_print_meta: max token length = 1024 llama_model_load: vocab only - skipping tensors
GiteaMirror commented 2026-05-09 17:08:21 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Jan 12, 2025):

time=2025-01-12T15:43:38.934+08:00 level=INFO source=memory.go:356 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[11.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="320.0 MiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.6 GiB" memory.weights.repeating="4.1 GiB" memory.weights.nonrepeating="485.6 MiB" memory.graph.full="561.0 MiB" memory.graph.partial="789.6 MiB"
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors: CPU_Mapped model buffer size = 333.00 MiB
llm_load_tensors: ROCm0 model buffer size = 4863.85 MiB

ollama thinks it's offloading everything to the GPU. Do you have logs for 0.2.1 to compare?

<!-- gh-comment-id:2585733814 --> @rick-github commented on GitHub (Jan 12, 2025): ``` time=2025-01-12T15:43:38.934+08:00 level=INFO source=memory.go:356 msg="offload to rocm" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[11.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="320.0 MiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.6 GiB" memory.weights.repeating="4.1 GiB" memory.weights.nonrepeating="485.6 MiB" memory.graph.full="561.0 MiB" memory.graph.partial="789.6 MiB" llm_load_tensors: offloaded 41/41 layers to GPU llm_load_tensors: CPU_Mapped model buffer size = 333.00 MiB llm_load_tensors: ROCm0 model buffer size = 4863.85 MiB ``` ollama thinks it's offloading everything to the GPU. Do you have logs for 0.2.1 to compare?
GiteaMirror commented 2026-05-09 17:08:22 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Jan 12, 2025):

Might be a problem with the ROCm driver: https://github.com/ggerganov/llama.cpp/issues/9964

<!-- gh-comment-id:2585761918 --> @rick-github commented on GitHub (Jan 12, 2025): Might be a problem with the ROCm driver: https://github.com/ggerganov/llama.cpp/issues/9964
GiteaMirror commented 2026-05-09 17:08:23 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Jan 12, 2025):

From 24.12.1 release notes:

  • Fixed Issues and Improvements
    • Lower than expected performance may be observed while using llama.cpp. This may also impact Ollama and LM Studio.
<!-- gh-comment-id:2585763132 --> @rick-github commented on GitHub (Jan 12, 2025): From 24.12.1 release notes: - Fixed Issues and Improvements - Lower than expected performance may be observed while using llama.cpp. This may also impact Ollama and LM Studio.
GiteaMirror commented 2026-05-09 17:08:23 -05:00
Author
Owner
Copy Link
Copy Source

@21307369 commented on GitHub (Jan 14, 2025):

You are right, it's this issue, the driver upgrade to 24.12.1 has resolved the problem

<!-- gh-comment-id:2591134143 --> @21307369 commented on GitHub (Jan 14, 2025): You are right, it's this issue, the driver upgrade to 24.12.1 has resolved the problem
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#83075