[GH-ISSUE #9595] deploys multiple models at the same time, the VRAM will be consumed more #52770

New Issue

Closed

opened 2026-04-29 00:48:43 -05:00 by GiteaMirror · 3 comments

GiteaMirror commented 2026-04-29 00:48:43 -05:00

Owner

Copy Link

Originally created by @NGC13009 on GitHub (Mar 8, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9595

What is the issue?

Running qwq alone consumes 77G VRAM, running qwen2.5-coder:32b alone consumes 41G VRAM

But when running simultaneously, why does it require 99+175GB VRAM?

i do nothing.

my server have 8 * A40

ps. the log is very long, i delete some not important, such as my server's username, the timestamp

Relevant log output

systemd[1]: Started Ollama Service.
ollama[2033163]: 2025/03/08 21:17:57 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:2,3,4,5,6,7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/media/data/ollama_model OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:3 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
ollama[2033163]: time=2025-03-08T21:17:57.994+08:00 level=INFO source=images.go:432 msg="total blobs: 35"
ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=routes.go:1277 msg="Listening on 127.0.0.1:11434 (version 0.5.13)"
ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-d170e984-8eec-a189-188d-b0c0284f8e4b library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-c2509db4-bad8-74f4-979d-f6d85b4c2ab7 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-51439b0b-a6e3-0362-dee5-03491e2d0cc2 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f8220244-52f7-0126-22b2-ba8e54c17a6a library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-5b8fe053-4948-782d-aba4-51655ea16364 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-3aa57143-2ba4-fe81-bf63-53936141ddfa library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:05 | 200 |      396.55µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:05 | 200 |   59.479075ms |       127.0.0.1 | POST     "/api/show"
ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:15.993+08:00 level=INFO source=server.go:97 msg="system memory" total="1007.5 GiB" free="981.5 GiB" free_swap="2.0 GiB"
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=999 layers.model=65 layers.offload=65 layers.split=11,11,11,11,11,10 memory.available="[44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="163.6 GiB" memory.required.partial="163.6 GiB" memory.required.kv="24.0 GiB" memory.required.allocations="[27.3 GiB 27.6 GiB 27.3 GiB 27.3 GiB 27.5 GiB 26.6 GiB]" memory.weights.total="41.5 GiB" memory.weights.repeating="40.9 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="19.1 GiB" memory.graph.partial="19.1 GiB"
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=INFO source=server.go:182 msg="enabling flash attention"
ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/data/ollama_model/blobs/sha256-c62ccde5630c20c8a9cf601861d31977d07450cad6dfdf1c661aab307107bddb --ctx-size 196608 --batch-size 512 --n-gpu-layers 999 --threads 80 --flash-attn --kv-cache-type q8_0 --parallel 3 --tensor-split 11,11,11,11,11,10 --port 44385"
ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
ollama[2033163]: time=2025-03-08T21:18:17.094+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
ollama[2033163]: time=2025-03-08T21:18:17.114+08:00 level=INFO source=runner.go:931 msg="starting go runner"
ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ollama[2033163]: ggml_cuda_init: found 6 CUDA devices:
ollama[2033163]:   Device 0: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 1: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 2: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 3: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 4: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 5: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
ollama[2033163]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so
ollama[2033163]: time=2025-03-08T21:18:17.869+08:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=80
ollama[2033163]: time=2025-03-08T21:18:17.870+08:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:44385"
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA3 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA4 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA5 (NVIDIA A40) - 45134 MiB free
ollama[2033163]: llama_model_loader: loaded meta data with 33 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-c62ccde5630c20c8a9cf601861d31977d07450cad6dfdf1c661aab307107bddb (version GGUF V3 (latest))
ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama[2033163]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
ollama[2033163]: llama_model_loader: - kv   1:                               general.type str              = model
ollama[2033163]: llama_model_loader: - kv   2:                               general.name str              = QwQ 32B
ollama[2033163]: llama_model_loader: - kv   3:                           general.basename str              = QwQ
ollama[2033163]: llama_model_loader: - kv   4:                         general.size_label str              = 32B
ollama[2033163]: llama_model_loader: - kv   5:                            general.license str              = apache-2.0
ollama[2033163]: llama_model_loader: - kv   6:                       general.license.link str              = https://huggingface.co/Qwen/QWQ-32B/b...
ollama[2033163]: llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
ollama[2033163]: llama_model_loader: - kv   8:                  general.base_model.0.name str              = Qwen2.5 32B
ollama[2033163]: llama_model_loader: - kv   9:          general.base_model.0.organization str              = Qwen
ollama[2033163]: llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-32B
ollama[2033163]: llama_model_loader: - kv  11:                               general.tags arr[str,2]       = ["chat", "text-generation"]
ollama[2033163]: llama_model_loader: - kv  12:                          general.languages arr[str,1]       = ["en"]
ollama[2033163]: llama_model_loader: - kv  13:                          qwen2.block_count u32              = 64
ollama[2033163]: llama_model_loader: - kv  14:                       qwen2.context_length u32              = 131072
ollama[2033163]: llama_model_loader: - kv  15:                     qwen2.embedding_length u32              = 5120
ollama[2033163]: llama_model_loader: - kv  16:                  qwen2.feed_forward_length u32              = 27648
ollama[2033163]: llama_model_loader: - kv  17:                 qwen2.attention.head_count u32              = 40
ollama[2033163]: llama_model_loader: - kv  18:              qwen2.attention.head_count_kv u32              = 8
ollama[2033163]: llama_model_loader: - kv  19:                       qwen2.rope.freq_base f32              = 1000000.000000
ollama[2033163]: llama_model_loader: - kv  20:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000010
ollama[2033163]: llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = gpt2
ollama[2033163]: llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = qwen2
ollama[2033163]: llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
ollama[2033163]: llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama[2033163]: time=2025-03-08T21:18:18.100+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
ollama[2033163]: llama_model_loader: - kv  25:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
ollama[2033163]: llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 151645
ollama[2033163]: llama_model_loader: - kv  27:            tokenizer.ggml.padding_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = false
ollama[2033163]: llama_model_loader: - kv  30:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
ollama[2033163]: llama_model_loader: - kv  31:               general.quantization_version u32              = 2
ollama[2033163]: llama_model_loader: - kv  32:                          general.file_type u32              = 15
ollama[2033163]: llama_model_loader: - type  f32:  321 tensors
ollama[2033163]: llama_model_loader: - type q4_K:  385 tensors
ollama[2033163]: llama_model_loader: - type q6_K:   65 tensors
ollama[2033163]: print_info: file format = GGUF V3 (latest)
ollama[2033163]: print_info: file type   = Q4_K - Medium
ollama[2033163]: print_info: file size   = 18.48 GiB (4.85 BPW)
ollama[2033163]: load: special tokens cache size = 26
ollama[2033163]: load: token to piece cache size = 0.9311 MB
ollama[2033163]: print_info: arch             = qwen2
ollama[2033163]: print_info: vocab_only       = 0
ollama[2033163]: print_info: n_ctx_train      = 131072
ollama[2033163]: print_info: n_embd           = 5120
ollama[2033163]: print_info: n_layer          = 64
ollama[2033163]: print_info: n_head           = 40
ollama[2033163]: print_info: n_head_kv        = 8
ollama[2033163]: print_info: n_rot            = 128
ollama[2033163]: print_info: n_swa            = 0
ollama[2033163]: print_info: n_embd_head_k    = 128
ollama[2033163]: print_info: n_embd_head_v    = 128
ollama[2033163]: print_info: n_gqa            = 5
ollama[2033163]: print_info: n_embd_k_gqa     = 1024
ollama[2033163]: print_info: n_embd_v_gqa     = 1024
ollama[2033163]: print_info: f_norm_eps       = 0.0e+00
ollama[2033163]: print_info: f_norm_rms_eps   = 1.0e-05
ollama[2033163]: print_info: f_clamp_kqv      = 0.0e+00
ollama[2033163]: print_info: f_max_alibi_bias = 0.0e+00
ollama[2033163]: print_info: f_logit_scale    = 0.0e+00
ollama[2033163]: print_info: n_ff             = 27648
ollama[2033163]: print_info: n_expert         = 0
ollama[2033163]: print_info: n_expert_used    = 0
ollama[2033163]: print_info: causal attn      = 1
ollama[2033163]: print_info: pooling type     = 0
ollama[2033163]: print_info: rope type        = 2
ollama[2033163]: print_info: rope scaling     = linear
ollama[2033163]: print_info: freq_base_train  = 1000000.0
ollama[2033163]: print_info: freq_scale_train = 1
ollama[2033163]: print_info: n_ctx_orig_yarn  = 131072
ollama[2033163]: print_info: rope_finetuned   = unknown
ollama[2033163]: print_info: ssm_d_conv       = 0
ollama[2033163]: print_info: ssm_d_inner      = 0
ollama[2033163]: print_info: ssm_d_state      = 0
ollama[2033163]: print_info: ssm_dt_rank      = 0
ollama[2033163]: print_info: ssm_dt_b_c_rms   = 0
ollama[2033163]: print_info: model type       = 32B
ollama[2033163]: print_info: model params     = 32.76 B
ollama[2033163]: print_info: general.name     = QwQ 32B
ollama[2033163]: print_info: vocab type       = BPE
ollama[2033163]: print_info: n_vocab          = 152064
ollama[2033163]: print_info: n_merges         = 151387
ollama[2033163]: print_info: BOS token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOS token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOT token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: PAD token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: LF token         = 198 'Ċ'
ollama[2033163]: print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
ollama[2033163]: print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
ollama[2033163]: print_info: FIM MID token    = 151660 '<|fim_middle|>'
ollama[2033163]: print_info: FIM PAD token    = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: FIM REP token    = 151663 '<|repo_name|>'
ollama[2033163]: print_info: FIM SEP token    = 151664 '<|file_sep|>'
ollama[2033163]: print_info: EOG token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOG token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOG token        = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: EOG token        = 151663 '<|repo_name|>'
ollama[2033163]: print_info: EOG token        = 151664 '<|file_sep|>'
ollama[2033163]: print_info: max token length = 256
ollama[2033163]: load_tensors: loading model tensors, this can take a while... (mmap = true)
ollama[2033163]: load_tensors: offloading 64 repeating layers to GPU
ollama[2033163]: load_tensors: offloading output layer to GPU
ollama[2033163]: load_tensors: offloaded 65/65 layers to GPU
ollama[2033163]: load_tensors:        CUDA0 model buffer size =  3202.76 MiB
ollama[2033163]: load_tensors:        CUDA1 model buffer size =  2986.20 MiB
ollama[2033163]: load_tensors:        CUDA2 model buffer size =  3022.29 MiB
ollama[2033163]: load_tensors:        CUDA3 model buffer size =  3022.29 MiB
ollama[2033163]: load_tensors:        CUDA4 model buffer size =  2986.20 MiB
ollama[2033163]: load_tensors:        CUDA5 model buffer size =  3288.61 MiB
ollama[2033163]: load_tensors:   CPU_Mapped model buffer size =   417.66 MiB
ollama[2033163]: llama_init_from_model: n_seq_max     = 3
ollama[2033163]: llama_init_from_model: n_ctx         = 196608
ollama[2033163]: llama_init_from_model: n_ctx_per_seq = 65536
ollama[2033163]: llama_init_from_model: n_batch       = 1536
ollama[2033163]: llama_init_from_model: n_ubatch      = 512
ollama[2033163]: llama_init_from_model: flash_attn    = 1
ollama[2033163]: llama_init_from_model: freq_base     = 1000000.0
ollama[2033163]: llama_init_from_model: freq_scale    = 1
ollama[2033163]: llama_init_from_model: n_ctx_per_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
ollama[2033163]: llama_kv_cache_init: kv_size = 196608, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 64, can_shift = 1
ollama[2033163]: llama_kv_cache_init:      CUDA0 KV buffer size =  4488.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA1 KV buffer size =  4488.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA2 KV buffer size =  4488.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA3 KV buffer size =  4488.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA4 KV buffer size =  4488.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA5 KV buffer size =  3672.00 MiB
ollama[2033163]: llama_init_from_model: KV self size  = 26112.00 MiB, K (q8_0): 13056.00 MiB, V (q8_0): 13056.00 MiB
ollama[2033163]: llama_init_from_model:  CUDA_Host  output buffer size =     1.80 MiB
ollama[2033163]: llama_init_from_model: pipeline parallelism enabled (n_copies=4)
ollama[2033163]: llama_init_from_model:      CUDA0 compute buffer size =  1906.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA1 compute buffer size =   946.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA2 compute buffer size =   946.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA3 compute buffer size =   946.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA4 compute buffer size =   946.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA5 compute buffer size =  1115.02 MiB
ollama[2033163]: llama_init_from_model:  CUDA_Host compute buffer size =  1546.02 MiB
ollama[2033163]: llama_init_from_model: graph nodes  = 1991
ollama[2033163]: llama_init_from_model: graph splits = 7
ollama[2033163]: time=2025-03-08T21:18:22.870+08:00 level=INFO source=server.go:596 msg="llama runner started in 5.78 seconds"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:22 | 200 | 16.968622459s |       127.0.0.1 | POST     "/api/generate"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:29 | 200 |  704.968892ms |       127.0.0.1 | POST     "/api/chat"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:41 | 200 |      27.055µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:41 | 200 |   23.549804ms |       127.0.0.1 | POST     "/api/show"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-d170e984-8eec-a189-188d-b0c0284f8e4b library=cuda total="44.4 GiB" available="17.1 GiB"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-c2509db4-bad8-74f4-979d-f6d85b4c2ab7 library=cuda total="44.4 GiB" available="16.8 GiB"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-51439b0b-a6e3-0362-dee5-03491e2d0cc2 library=cuda total="44.4 GiB" available="17.1 GiB"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-f8220244-52f7-0126-22b2-ba8e54c17a6a library=cuda total="44.4 GiB" available="17.1 GiB"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-5b8fe053-4948-782d-aba4-51655ea16364 library=cuda total="44.4 GiB" available="16.8 GiB"
ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-3aa57143-2ba4-fe81-bf63-53936141ddfa library=cuda total="44.4 GiB" available="17.7 GiB"
ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:51.218+08:00 level=INFO source=sched.go:731 msg="new model will fit in available VRAM, loading" model=/media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 library=cuda parallel=3 required="93.1 GiB"
ollama[2033163]: time=2025-03-08T21:18:52.355+08:00 level=INFO source=server.go:97 msg="system memory" total="1007.5 GiB" free="979.1 GiB" free_swap="2.0 GiB"
ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=11,11,11,11,11,10 memory.available="[17.7 GiB 17.1 GiB 17.1 GiB 17.1 GiB 16.8 GiB 16.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="93.1 GiB" memory.required.partial="93.1 GiB" memory.required.kv="12.0 GiB" memory.required.allocations="[15.5 GiB 15.8 GiB 15.5 GiB 15.5 GiB 15.8 GiB 15.0 GiB]" memory.weights.total="29.5 GiB" memory.weights.repeating="28.9 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="9.6 GiB" memory.graph.partial="9.6 GiB"
ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128
ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:182 msg="enabling flash attention"
ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 --ctx-size 98304 --batch-size 512 --n-gpu-layers 65 --threads 80 --flash-attn --kv-cache-type q8_0 --parallel 3 --tensor-split 11,11,11,11,11,10 --port 40267"
ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=2
ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
ollama[2033163]: time=2025-03-08T21:18:53.509+08:00 level=INFO source=runner.go:931 msg="starting go runner"
ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ollama[2033163]: ggml_cuda_init: found 6 CUDA devices:
ollama[2033163]:   Device 0: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 1: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 2: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 3: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 4: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]:   Device 5: NVIDIA A40, compute capability 8.6, VMM: yes
ollama[2033163]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
ollama[2033163]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so
ollama[2033163]: time=2025-03-08T21:18:54.298+08:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=80
ollama[2033163]: time=2025-03-08T21:18:54.298+08:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:40267"
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 36727 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 35201 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 36341 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA3 (NVIDIA A40) - 36341 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA4 (NVIDIA A40) - 36377 MiB free
ollama[2033163]: llama_model_load_from_file_impl: using device CUDA5 (NVIDIA A40) - 36377 MiB free
ollama[2033163]: time=2025-03-08T21:18:54.498+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
ollama[2033163]: llama_model_loader: loaded meta data with 34 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 (version GGUF V3 (latest))
ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama[2033163]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
ollama[2033163]: llama_model_loader: - kv   1:                               general.type str              = model
ollama[2033163]: llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 32B Instruct
ollama[2033163]: llama_model_loader: - kv   3:                           general.finetune str              = Instruct
ollama[2033163]: llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-Coder
ollama[2033163]: llama_model_loader: - kv   5:                         general.size_label str              = 32B
ollama[2033163]: llama_model_loader: - kv   6:                            general.license str              = apache-2.0
ollama[2033163]: llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-C...
ollama[2033163]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
ollama[2033163]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 Coder 32B
ollama[2033163]: llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
ollama[2033163]: llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-C...
ollama[2033163]: llama_model_loader: - kv  12:                               general.tags arr[str,6]       = ["code", "codeqwen", "chat", "qwen", ...
ollama[2033163]: llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
ollama[2033163]: llama_model_loader: - kv  14:                          qwen2.block_count u32              = 64
ollama[2033163]: llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
ollama[2033163]: llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 5120
ollama[2033163]: llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 27648
ollama[2033163]: llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 40
ollama[2033163]: llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 8
ollama[2033163]: llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
ollama[2033163]: llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
ollama[2033163]: llama_model_loader: - kv  22:                          general.file_type u32              = 15
ollama[2033163]: llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
ollama[2033163]: llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
ollama[2033163]: llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
ollama[2033163]: llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama[2033163]: llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
ollama[2033163]: llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
ollama[2033163]: llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
ollama[2033163]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
ollama[2033163]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
ollama[2033163]: llama_model_loader: - type  f32:  321 tensors
ollama[2033163]: llama_model_loader: - type q4_K:  385 tensors
ollama[2033163]: llama_model_loader: - type q6_K:   65 tensors
ollama[2033163]: print_info: file format = GGUF V3 (latest)
ollama[2033163]: print_info: file type   = Q4_K - Medium
ollama[2033163]: print_info: file size   = 18.48 GiB (4.85 BPW)
ollama[2033163]: load: special tokens cache size = 22
ollama[2033163]: load: token to piece cache size = 0.9310 MB
ollama[2033163]: print_info: arch             = qwen2
ollama[2033163]: print_info: vocab_only       = 0
ollama[2033163]: print_info: n_ctx_train      = 32768
ollama[2033163]: print_info: n_embd           = 5120
ollama[2033163]: print_info: n_layer          = 64
ollama[2033163]: print_info: n_head           = 40
ollama[2033163]: print_info: n_head_kv        = 8
ollama[2033163]: print_info: n_rot            = 128
ollama[2033163]: print_info: n_swa            = 0
ollama[2033163]: print_info: n_embd_head_k    = 128
ollama[2033163]: print_info: n_embd_head_v    = 128
ollama[2033163]: print_info: n_gqa            = 5
ollama[2033163]: print_info: n_embd_k_gqa     = 1024
ollama[2033163]: print_info: n_embd_v_gqa     = 1024
ollama[2033163]: print_info: f_norm_eps       = 0.0e+00
ollama[2033163]: print_info: f_norm_rms_eps   = 1.0e-06
ollama[2033163]: print_info: f_clamp_kqv      = 0.0e+00
ollama[2033163]: print_info: f_max_alibi_bias = 0.0e+00
ollama[2033163]: print_info: f_logit_scale    = 0.0e+00
ollama[2033163]: print_info: n_ff             = 27648
ollama[2033163]: print_info: n_expert         = 0
ollama[2033163]: print_info: n_expert_used    = 0
ollama[2033163]: print_info: causal attn      = 1
ollama[2033163]: print_info: pooling type     = 0
ollama[2033163]: print_info: rope type        = 2
ollama[2033163]: print_info: rope scaling     = linear
ollama[2033163]: print_info: freq_base_train  = 1000000.0
ollama[2033163]: print_info: freq_scale_train = 1
ollama[2033163]: print_info: n_ctx_orig_yarn  = 32768
ollama[2033163]: print_info: rope_finetuned   = unknown
ollama[2033163]: print_info: ssm_d_conv       = 0
ollama[2033163]: print_info: ssm_d_inner      = 0
ollama[2033163]: print_info: ssm_d_state      = 0
ollama[2033163]: print_info: ssm_dt_rank      = 0
ollama[2033163]: print_info: ssm_dt_b_c_rms   = 0
ollama[2033163]: print_info: model type       = 32B
ollama[2033163]: print_info: model params     = 32.76 B
ollama[2033163]: print_info: general.name     = Qwen2.5 Coder 32B Instruct
ollama[2033163]: print_info: vocab type       = BPE
ollama[2033163]: print_info: n_vocab          = 152064
ollama[2033163]: print_info: n_merges         = 151387
ollama[2033163]: print_info: BOS token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOS token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOT token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: PAD token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: LF token         = 198 'Ċ'
ollama[2033163]: print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
ollama[2033163]: print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
ollama[2033163]: print_info: FIM MID token    = 151660 '<|fim_middle|>'
ollama[2033163]: print_info: FIM PAD token    = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: FIM REP token    = 151663 '<|repo_name|>'
ollama[2033163]: print_info: FIM SEP token    = 151664 '<|file_sep|>'
ollama[2033163]: print_info: EOG token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOG token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOG token        = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: EOG token        = 151663 '<|repo_name|>'
ollama[2033163]: print_info: EOG token        = 151664 '<|file_sep|>'
ollama[2033163]: print_info: max token length = 256
ollama[2033163]: load_tensors: loading model tensors, this can take a while... (mmap = true)
ollama[2033163]: load_tensors: offloading 64 repeating layers to GPU
ollama[2033163]: load_tensors: offloading output layer to GPU
ollama[2033163]: load_tensors: offloaded 65/65 layers to GPU
ollama[2033163]: load_tensors:        CUDA0 model buffer size =  3167.96 MiB
ollama[2033163]: load_tensors:        CUDA1 model buffer size =  3021.00 MiB
ollama[2033163]: load_tensors:        CUDA2 model buffer size =  3022.29 MiB
ollama[2033163]: load_tensors:        CUDA3 model buffer size =  3022.29 MiB
ollama[2033163]: load_tensors:        CUDA4 model buffer size =  2986.20 MiB
ollama[2033163]: load_tensors:        CUDA5 model buffer size =  3288.61 MiB
ollama[2033163]: load_tensors:   CPU_Mapped model buffer size =   417.66 MiB
ollama[2033163]: llama_init_from_model: n_seq_max     = 3
ollama[2033163]: llama_init_from_model: n_ctx         = 98304
ollama[2033163]: llama_init_from_model: n_ctx_per_seq = 32768
ollama[2033163]: llama_init_from_model: n_batch       = 1536
ollama[2033163]: llama_init_from_model: n_ubatch      = 512
ollama[2033163]: llama_init_from_model: flash_attn    = 1
ollama[2033163]: llama_init_from_model: freq_base     = 1000000.0
ollama[2033163]: llama_init_from_model: freq_scale    = 1
ollama[2033163]: llama_kv_cache_init: kv_size = 98304, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 64, can_shift = 1
ollama[2033163]: llama_kv_cache_init:      CUDA0 KV buffer size =  2244.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA1 KV buffer size =  2244.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA2 KV buffer size =  2244.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA3 KV buffer size =  2244.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA4 KV buffer size =  2244.00 MiB
ollama[2033163]: llama_kv_cache_init:      CUDA5 KV buffer size =  1836.00 MiB
ollama[2033163]: llama_init_from_model: KV self size  = 13056.00 MiB, K (q8_0): 6528.00 MiB, V (q8_0): 6528.00 MiB
ollama[2033163]: llama_init_from_model:  CUDA_Host  output buffer size =     1.80 MiB
ollama[2033163]: llama_init_from_model: pipeline parallelism enabled (n_copies=4)
ollama[2033163]: llama_init_from_model:      CUDA0 compute buffer size =  1042.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA1 compute buffer size =   562.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA2 compute buffer size =   562.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA3 compute buffer size =   562.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA4 compute buffer size =   562.01 MiB
ollama[2033163]: llama_init_from_model:      CUDA5 compute buffer size =   731.02 MiB
ollama[2033163]: llama_init_from_model:  CUDA_Host compute buffer size =   778.02 MiB
ollama[2033163]: llama_init_from_model: graph nodes  = 1991
ollama[2033163]: llama_init_from_model: graph splits = 7
ollama[2033163]: time=2025-03-08T21:18:58.513+08:00 level=INFO source=server.go:596 msg="llama runner started in 5.02 seconds"
ollama[2033163]: [GIN] 2025/03/08 - 21:18:58 | 200 | 16.524096975s |       127.0.0.1 | POST     "/api/generate"
ollama[2033163]: llama_model_loader: loaded meta data with 34 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 (version GGUF V3 (latest))
ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama[2033163]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
ollama[2033163]: llama_model_loader: - kv   1:                               general.type str              = model
ollama[2033163]: llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 32B Instruct
ollama[2033163]: llama_model_loader: - kv   3:                           general.finetune str              = Instruct
ollama[2033163]: llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-Coder
ollama[2033163]: llama_model_loader: - kv   5:                         general.size_label str              = 32B
ollama[2033163]: llama_model_loader: - kv   6:                            general.license str              = apache-2.0
ollama[2033163]: llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-C...
ollama[2033163]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
ollama[2033163]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 Coder 32B
ollama[2033163]: llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
ollama[2033163]: llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-C...
ollama[2033163]: llama_model_loader: - kv  12:                               general.tags arr[str,6]       = ["code", "codeqwen", "chat", "qwen", ...
ollama[2033163]: llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
ollama[2033163]: llama_model_loader: - kv  14:                          qwen2.block_count u32              = 64
ollama[2033163]: llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
ollama[2033163]: llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 5120
ollama[2033163]: llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 27648
ollama[2033163]: llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 40
ollama[2033163]: llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 8
ollama[2033163]: llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
ollama[2033163]: llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
ollama[2033163]: llama_model_loader: - kv  22:                          general.file_type u32              = 15
ollama[2033163]: llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
ollama[2033163]: llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
ollama[2033163]: llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
ollama[2033163]: llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama[2033163]: llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
ollama[2033163]: llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
ollama[2033163]: llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
ollama[2033163]: llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
ollama[2033163]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
ollama[2033163]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
ollama[2033163]: llama_model_loader: - type  f32:  321 tensors
ollama[2033163]: llama_model_loader: - type q4_K:  385 tensors
ollama[2033163]: llama_model_loader: - type q6_K:   65 tensors
ollama[2033163]: print_info: file format = GGUF V3 (latest)
ollama[2033163]: print_info: file type   = Q4_K - Medium
ollama[2033163]: print_info: file size   = 18.48 GiB (4.85 BPW)
ollama[2033163]: load: special tokens cache size = 22
ollama[2033163]: load: token to piece cache size = 0.9310 MB
ollama[2033163]: print_info: arch             = qwen2
ollama[2033163]: print_info: vocab_only       = 1
ollama[2033163]: print_info: model type       = ?B
ollama[2033163]: print_info: model params     = 32.76 B
ollama[2033163]: print_info: general.name     = Qwen2.5 Coder 32B Instruct
ollama[2033163]: print_info: vocab type       = BPE
ollama[2033163]: print_info: n_vocab          = 152064
ollama[2033163]: print_info: n_merges         = 151387
ollama[2033163]: print_info: BOS token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOS token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOT token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: PAD token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: LF token         = 198 'Ċ'
ollama[2033163]: print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
ollama[2033163]: print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
ollama[2033163]: print_info: FIM MID token    = 151660 '<|fim_middle|>'
ollama[2033163]: print_info: FIM PAD token    = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: FIM REP token    = 151663 '<|repo_name|>'
ollama[2033163]: print_info: FIM SEP token    = 151664 '<|file_sep|>'
ollama[2033163]: print_info: EOG token        = 151643 '<|endoftext|>'
ollama[2033163]: print_info: EOG token        = 151645 '<|im_end|>'
ollama[2033163]: print_info: EOG token        = 151662 '<|fim_pad|>'
ollama[2033163]: print_info: EOG token        = 151663 '<|repo_name|>'
ollama[2033163]: print_info: EOG token        = 151664 '<|file_sep|>'
ollama[2033163]: print_info: max token length = 256
ollama[2033163]: llama_model_load: vocab only - skipping tensors
ollama[2033163]: [GIN] 2025/03/08 - 21:19:04 | 200 |  840.178362ms |       127.0.0.1 | POST     "/api/chat"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:11 | 200 |      68.886µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:11 | 200 |     122.734µs |       127.0.0.1 | GET      "/api/ps"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:46 | 200 |        27.2µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:46 | 200 |      40.331µs |       127.0.0.1 | GET      "/api/ps"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:47 | 200 |      48.905µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:19:47 | 200 |      31.523µs |       127.0.0.1 | GET      "/api/ps"
ollama[2033163]: [GIN] 2025/03/08 - 21:20:48 | 200 |    3.380467ms |       127.0.0.1 | GET      "/v1/models"
ollama[2033163]: [GIN] 2025/03/08 - 21:22:01 | 200 |   38.6498034s |       127.0.0.1 | POST     "/v1/chat/completions"
ollama[2033163]: [GIN] 2025/03/08 - 21:22:38 | 200 | 36.352672616s |       127.0.0.1 | POST     "/v1/chat/completions"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:01 | 200 |   58.535137ms |       127.0.0.1 | POST     "/api/show"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:04 | 200 |      47.295µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:04 | 200 |    2.986628ms |       127.0.0.1 | GET      "/api/tags"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:28 | 200 |       62.67µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:28 | 200 |    1.106065ms |       127.0.0.1 | POST     "/api/generate"
ollama[2033163]: time=2025-03-08T21:32:28.102+08:00 level=INFO source=images.go:432 msg="total blobs: 35"
ollama[2033163]: time=2025-03-08T21:32:28.103+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
ollama[2033163]: time=2025-03-08T21:32:28.103+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=40 remote=127.0.0.1:46396 proto=HTTP/1.1 query=""
ollama[2033163]: time=2025-03-08T21:32:28.105+08:00 level=INFO source=images.go:432 msg="total blobs: 35"
ollama[2033163]: time=2025-03-08T21:32:28.799+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 2"
ollama[2033163]: time=2025-03-08T21:32:28.799+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=36 remote=127.0.0.1:46396 proto=HTTP/1.1 query=""
ollama[2033163]: [GIN] 2025/03/08 - 21:32:59 | 200 |      50.825µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:32:59 | 200 |    2.260465ms |       127.0.0.1 | POST     "/api/generate"
ollama[2033163]: time=2025-03-08T21:32:59.082+08:00 level=INFO source=images.go:432 msg="total blobs: 33"
ollama[2033163]: time=2025-03-08T21:33:00.489+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 2"
ollama[2033163]: time=2025-03-08T21:33:00.489+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=37 remote=127.0.0.1:57012 proto=HTTP/1.1 query=""
ollama[2033163]: time=2025-03-08T21:33:00.493+08:00 level=INFO source=images.go:432 msg="total blobs: 31"
ollama[2033163]: time=2025-03-08T21:33:03.671+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 5"
ollama[2033163]: time=2025-03-08T21:33:03.671+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=37 remote=127.0.0.1:57012 proto=HTTP/1.1 query=""
ollama[2033163]: time=2025-03-08T21:33:03.675+08:00 level=INFO source=images.go:432 msg="total blobs: 26"
ollama[2033163]: time=2025-03-08T21:33:04.288+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 3"
ollama[2033163]: time=2025-03-08T21:33:04.288+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=33 remote=127.0.0.1:57012 proto=HTTP/1.1 query=""
ollama[2033163]: time=2025-03-08T21:33:04.323+08:00 level=INFO source=images.go:432 msg="total blobs: 23"
ollama[2033163]: time=2025-03-08T21:33:05.727+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 3"
ollama[2033163]: time=2025-03-08T21:33:05.727+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=33 remote=127.0.0.1:57012 proto=HTTP/1.1 query=""
ollama[2033163]: time=2025-03-08T21:33:05.729+08:00 level=INFO source=images.go:432 msg="total blobs: 20"
ollama[2033163]: time=2025-03-08T21:33:12.063+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 5"
ollama[2033163]: time=2025-03-08T21:33:12.063+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=38 remote=127.0.0.1:57012 proto=HTTP/1.1 query=""
ollama[2033163]: [GIN] 2025/03/08 - 21:33:37 | 200 |      26.754µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:33:37 | 200 |     600.829µs |       127.0.0.1 | GET      "/api/tags"
ollama[2033163]: [GIN] 2025/03/08 - 21:33:49 | 200 |      60.488µs |       127.0.0.1 | HEAD     "/"
ollama[2033163]: [GIN] 2025/03/08 - 21:33:49 | 200 |      77.212µs |       127.0.0.1 | GET      "/api/ps"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.13

Originally created by @NGC13009 on GitHub (Mar 8, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9595 ### What is the issue? Running `qwq` alone consumes 77G VRAM, running `qwen2.5-coder:32b` alone consumes 41G VRAM But when running simultaneously, why does it require 99+175GB VRAM? i do nothing. my server have 8 * A40  ps. the log is very long, i delete some not important, such as my server's username, the timestamp ### Relevant log output ```shell systemd[1]: Started Ollama Service. ollama[2033163]: 2025/03/08 21:17:57 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:2,3,4,5,6,7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE:q8_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/media/data/ollama_model OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:3 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" ollama[2033163]: time=2025-03-08T21:17:57.994+08:00 level=INFO source=images.go:432 msg="total blobs: 35" ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=routes.go:1277 msg="Listening on 127.0.0.1:11434 (version 0.5.13)" ollama[2033163]: time=2025-03-08T21:17:57.995+08:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-d170e984-8eec-a189-188d-b0c0284f8e4b library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-c2509db4-bad8-74f4-979d-f6d85b4c2ab7 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-51439b0b-a6e3-0362-dee5-03491e2d0cc2 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f8220244-52f7-0126-22b2-ba8e54c17a6a library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-5b8fe053-4948-782d-aba4-51655ea16364 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: time=2025-03-08T21:17:59.385+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-3aa57143-2ba4-fe81-bf63-53936141ddfa library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA A40" total="44.4 GiB" available="44.1 GiB" ollama[2033163]: [GIN] 2025/03/08 - 21:18:05 | 200 | 396.55µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:18:05 | 200 | 59.479075ms | 127.0.0.1 | POST "/api/show" ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:08.330+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:09.429+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:10.513+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:11.609+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:12.699+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:13.793+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:14.907+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:15.993+08:00 level=INFO source=server.go:97 msg="system memory" total="1007.5 GiB" free="981.5 GiB" free_swap="2.0 GiB" ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=999 layers.model=65 layers.offload=65 layers.split=11,11,11,11,11,10 memory.available="[44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB 44.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="163.6 GiB" memory.required.partial="163.6 GiB" memory.required.kv="24.0 GiB" memory.required.allocations="[27.3 GiB 27.6 GiB 27.3 GiB 27.3 GiB 27.5 GiB 26.6 GiB]" memory.weights.total="41.5 GiB" memory.weights.repeating="40.9 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="19.1 GiB" memory.graph.partial="19.1 GiB" ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:17.092+08:00 level=INFO source=server.go:182 msg="enabling flash attention" ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/data/ollama_model/blobs/sha256-c62ccde5630c20c8a9cf601861d31977d07450cad6dfdf1c661aab307107bddb --ctx-size 196608 --batch-size 512 --n-gpu-layers 999 --threads 80 --flash-attn --kv-cache-type q8_0 --parallel 3 --tensor-split 11,11,11,11,11,10 --port 44385" ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 ollama[2033163]: time=2025-03-08T21:18:17.093+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" ollama[2033163]: time=2025-03-08T21:18:17.094+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" ollama[2033163]: time=2025-03-08T21:18:17.114+08:00 level=INFO source=runner.go:931 msg="starting go runner" ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ollama[2033163]: ggml_cuda_init: found 6 CUDA devices: ollama[2033163]: Device 0: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 1: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 2: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 3: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 4: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 5: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so ollama[2033163]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so ollama[2033163]: time=2025-03-08T21:18:17.869+08:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=80 ollama[2033163]: time=2025-03-08T21:18:17.870+08:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:44385" ollama[2033163]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA3 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA4 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA5 (NVIDIA A40) - 45134 MiB free ollama[2033163]: llama_model_loader: loaded meta data with 33 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-c62ccde5630c20c8a9cf601861d31977d07450cad6dfdf1c661aab307107bddb (version GGUF V3 (latest)) ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. ollama[2033163]: llama_model_loader: - kv 0: general.architecture str = qwen2 ollama[2033163]: llama_model_loader: - kv 1: general.type str = model ollama[2033163]: llama_model_loader: - kv 2: general.name str = QwQ 32B ollama[2033163]: llama_model_loader: - kv 3: general.basename str = QwQ ollama[2033163]: llama_model_loader: - kv 4: general.size_label str = 32B ollama[2033163]: llama_model_loader: - kv 5: general.license str = apache-2.0 ollama[2033163]: llama_model_loader: - kv 6: general.license.link str = https://huggingface.co/Qwen/QWQ-32B/b... ollama[2033163]: llama_model_loader: - kv 7: general.base_model.count u32 = 1 ollama[2033163]: llama_model_loader: - kv 8: general.base_model.0.name str = Qwen2.5 32B ollama[2033163]: llama_model_loader: - kv 9: general.base_model.0.organization str = Qwen ollama[2033163]: llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-32B ollama[2033163]: llama_model_loader: - kv 11: general.tags arr[str,2] = ["chat", "text-generation"] ollama[2033163]: llama_model_loader: - kv 12: general.languages arr[str,1] = ["en"] ollama[2033163]: llama_model_loader: - kv 13: qwen2.block_count u32 = 64 ollama[2033163]: llama_model_loader: - kv 14: qwen2.context_length u32 = 131072 ollama[2033163]: llama_model_loader: - kv 15: qwen2.embedding_length u32 = 5120 ollama[2033163]: llama_model_loader: - kv 16: qwen2.feed_forward_length u32 = 27648 ollama[2033163]: llama_model_loader: - kv 17: qwen2.attention.head_count u32 = 40 ollama[2033163]: llama_model_loader: - kv 18: qwen2.attention.head_count_kv u32 = 8 ollama[2033163]: llama_model_loader: - kv 19: qwen2.rope.freq_base f32 = 1000000.000000 ollama[2033163]: llama_model_loader: - kv 20: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000010 ollama[2033163]: llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2 ollama[2033163]: llama_model_loader: - kv 22: tokenizer.ggml.pre str = qwen2 ollama[2033163]: llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... ollama[2033163]: llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... ollama[2033163]: time=2025-03-08T21:18:18.100+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" ollama[2033163]: llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... ollama[2033163]: llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 151645 ollama[2033163]: llama_model_loader: - kv 27: tokenizer.ggml.padding_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 29: tokenizer.ggml.add_bos_token bool = false ollama[2033163]: llama_model_loader: - kv 30: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... ollama[2033163]: llama_model_loader: - kv 31: general.quantization_version u32 = 2 ollama[2033163]: llama_model_loader: - kv 32: general.file_type u32 = 15 ollama[2033163]: llama_model_loader: - type f32: 321 tensors ollama[2033163]: llama_model_loader: - type q4_K: 385 tensors ollama[2033163]: llama_model_loader: - type q6_K: 65 tensors ollama[2033163]: print_info: file format = GGUF V3 (latest) ollama[2033163]: print_info: file type = Q4_K - Medium ollama[2033163]: print_info: file size = 18.48 GiB (4.85 BPW) ollama[2033163]: load: special tokens cache size = 26 ollama[2033163]: load: token to piece cache size = 0.9311 MB ollama[2033163]: print_info: arch = qwen2 ollama[2033163]: print_info: vocab_only = 0 ollama[2033163]: print_info: n_ctx_train = 131072 ollama[2033163]: print_info: n_embd = 5120 ollama[2033163]: print_info: n_layer = 64 ollama[2033163]: print_info: n_head = 40 ollama[2033163]: print_info: n_head_kv = 8 ollama[2033163]: print_info: n_rot = 128 ollama[2033163]: print_info: n_swa = 0 ollama[2033163]: print_info: n_embd_head_k = 128 ollama[2033163]: print_info: n_embd_head_v = 128 ollama[2033163]: print_info: n_gqa = 5 ollama[2033163]: print_info: n_embd_k_gqa = 1024 ollama[2033163]: print_info: n_embd_v_gqa = 1024 ollama[2033163]: print_info: f_norm_eps = 0.0e+00 ollama[2033163]: print_info: f_norm_rms_eps = 1.0e-05 ollama[2033163]: print_info: f_clamp_kqv = 0.0e+00 ollama[2033163]: print_info: f_max_alibi_bias = 0.0e+00 ollama[2033163]: print_info: f_logit_scale = 0.0e+00 ollama[2033163]: print_info: n_ff = 27648 ollama[2033163]: print_info: n_expert = 0 ollama[2033163]: print_info: n_expert_used = 0 ollama[2033163]: print_info: causal attn = 1 ollama[2033163]: print_info: pooling type = 0 ollama[2033163]: print_info: rope type = 2 ollama[2033163]: print_info: rope scaling = linear ollama[2033163]: print_info: freq_base_train = 1000000.0 ollama[2033163]: print_info: freq_scale_train = 1 ollama[2033163]: print_info: n_ctx_orig_yarn = 131072 ollama[2033163]: print_info: rope_finetuned = unknown ollama[2033163]: print_info: ssm_d_conv = 0 ollama[2033163]: print_info: ssm_d_inner = 0 ollama[2033163]: print_info: ssm_d_state = 0 ollama[2033163]: print_info: ssm_dt_rank = 0 ollama[2033163]: print_info: ssm_dt_b_c_rms = 0 ollama[2033163]: print_info: model type = 32B ollama[2033163]: print_info: model params = 32.76 B ollama[2033163]: print_info: general.name = QwQ 32B ollama[2033163]: print_info: vocab type = BPE ollama[2033163]: print_info: n_vocab = 152064 ollama[2033163]: print_info: n_merges = 151387 ollama[2033163]: print_info: BOS token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOS token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOT token = 151645 '<|im_end|>' ollama[2033163]: print_info: PAD token = 151643 '<|endoftext|>' ollama[2033163]: print_info: LF token = 198 'Ċ' ollama[2033163]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' ollama[2033163]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' ollama[2033163]: print_info: FIM MID token = 151660 '<|fim_middle|>' ollama[2033163]: print_info: FIM PAD token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: FIM REP token = 151663 '<|repo_name|>' ollama[2033163]: print_info: FIM SEP token = 151664 '<|file_sep|>' ollama[2033163]: print_info: EOG token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOG token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOG token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: EOG token = 151663 '<|repo_name|>' ollama[2033163]: print_info: EOG token = 151664 '<|file_sep|>' ollama[2033163]: print_info: max token length = 256 ollama[2033163]: load_tensors: loading model tensors, this can take a while... (mmap = true) ollama[2033163]: load_tensors: offloading 64 repeating layers to GPU ollama[2033163]: load_tensors: offloading output layer to GPU ollama[2033163]: load_tensors: offloaded 65/65 layers to GPU ollama[2033163]: load_tensors: CUDA0 model buffer size = 3202.76 MiB ollama[2033163]: load_tensors: CUDA1 model buffer size = 2986.20 MiB ollama[2033163]: load_tensors: CUDA2 model buffer size = 3022.29 MiB ollama[2033163]: load_tensors: CUDA3 model buffer size = 3022.29 MiB ollama[2033163]: load_tensors: CUDA4 model buffer size = 2986.20 MiB ollama[2033163]: load_tensors: CUDA5 model buffer size = 3288.61 MiB ollama[2033163]: load_tensors: CPU_Mapped model buffer size = 417.66 MiB ollama[2033163]: llama_init_from_model: n_seq_max = 3 ollama[2033163]: llama_init_from_model: n_ctx = 196608 ollama[2033163]: llama_init_from_model: n_ctx_per_seq = 65536 ollama[2033163]: llama_init_from_model: n_batch = 1536 ollama[2033163]: llama_init_from_model: n_ubatch = 512 ollama[2033163]: llama_init_from_model: flash_attn = 1 ollama[2033163]: llama_init_from_model: freq_base = 1000000.0 ollama[2033163]: llama_init_from_model: freq_scale = 1 ollama[2033163]: llama_init_from_model: n_ctx_per_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized ollama[2033163]: llama_kv_cache_init: kv_size = 196608, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 64, can_shift = 1 ollama[2033163]: llama_kv_cache_init: CUDA0 KV buffer size = 4488.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA1 KV buffer size = 4488.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA2 KV buffer size = 4488.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA3 KV buffer size = 4488.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA4 KV buffer size = 4488.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA5 KV buffer size = 3672.00 MiB ollama[2033163]: llama_init_from_model: KV self size = 26112.00 MiB, K (q8_0): 13056.00 MiB, V (q8_0): 13056.00 MiB ollama[2033163]: llama_init_from_model: CUDA_Host output buffer size = 1.80 MiB ollama[2033163]: llama_init_from_model: pipeline parallelism enabled (n_copies=4) ollama[2033163]: llama_init_from_model: CUDA0 compute buffer size = 1906.01 MiB ollama[2033163]: llama_init_from_model: CUDA1 compute buffer size = 946.01 MiB ollama[2033163]: llama_init_from_model: CUDA2 compute buffer size = 946.01 MiB ollama[2033163]: llama_init_from_model: CUDA3 compute buffer size = 946.01 MiB ollama[2033163]: llama_init_from_model: CUDA4 compute buffer size = 946.01 MiB ollama[2033163]: llama_init_from_model: CUDA5 compute buffer size = 1115.02 MiB ollama[2033163]: llama_init_from_model: CUDA_Host compute buffer size = 1546.02 MiB ollama[2033163]: llama_init_from_model: graph nodes = 1991 ollama[2033163]: llama_init_from_model: graph splits = 7 ollama[2033163]: time=2025-03-08T21:18:22.870+08:00 level=INFO source=server.go:596 msg="llama runner started in 5.78 seconds" ollama[2033163]: [GIN] 2025/03/08 - 21:18:22 | 200 | 16.968622459s | 127.0.0.1 | POST "/api/generate" ollama[2033163]: [GIN] 2025/03/08 - 21:18:29 | 200 | 704.968892ms | 127.0.0.1 | POST "/api/chat" ollama[2033163]: [GIN] 2025/03/08 - 21:18:41 | 200 | 27.055µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:18:41 | 200 | 23.549804ms | 127.0.0.1 | POST "/api/show" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-d170e984-8eec-a189-188d-b0c0284f8e4b library=cuda total="44.4 GiB" available="17.1 GiB" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-c2509db4-bad8-74f4-979d-f6d85b4c2ab7 library=cuda total="44.4 GiB" available="16.8 GiB" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-51439b0b-a6e3-0362-dee5-03491e2d0cc2 library=cuda total="44.4 GiB" available="17.1 GiB" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-f8220244-52f7-0126-22b2-ba8e54c17a6a library=cuda total="44.4 GiB" available="17.1 GiB" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-5b8fe053-4948-782d-aba4-51655ea16364 library=cuda total="44.4 GiB" available="16.8 GiB" ollama[2033163]: time=2025-03-08T21:18:43.243+08:00 level=INFO source=sched.go:508 msg="updated VRAM based on existing loaded models" gpu=GPU-3aa57143-2ba4-fe81-bf63-53936141ddfa library=cuda total="44.4 GiB" available="17.7 GiB" ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:44.397+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:45.550+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:46.686+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:47.825+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:48.964+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:50.092+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:51.217+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:51.218+08:00 level=INFO source=sched.go:731 msg="new model will fit in available VRAM, loading" model=/media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 library=cuda parallel=3 required="93.1 GiB" ollama[2033163]: time=2025-03-08T21:18:52.355+08:00 level=INFO source=server.go:97 msg="system memory" total="1007.5 GiB" free="979.1 GiB" free_swap="2.0 GiB" ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.489+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=11,11,11,11,11,10 memory.available="[17.7 GiB 17.1 GiB 17.1 GiB 17.1 GiB 16.8 GiB 16.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="93.1 GiB" memory.required.partial="93.1 GiB" memory.required.kv="12.0 GiB" memory.required.allocations="[15.5 GiB 15.8 GiB 15.5 GiB 15.5 GiB 15.8 GiB 15.0 GiB]" memory.weights.total="29.5 GiB" memory.weights.repeating="28.9 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="9.6 GiB" memory.graph.partial="9.6 GiB" ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.key_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=WARN source=ggml.go:136 msg="key not found" key=qwen2.attention.value_length default=128 ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:182 msg="enabling flash attention" ollama[2033163]: time=2025-03-08T21:18:53.490+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 --ctx-size 98304 --batch-size 512 --n-gpu-layers 65 --threads 80 --flash-attn --kv-cache-type q8_0 --parallel 3 --tensor-split 11,11,11,11,11,10 --port 40267" ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=2 ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" ollama[2033163]: time=2025-03-08T21:18:53.491+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" ollama[2033163]: time=2025-03-08T21:18:53.509+08:00 level=INFO source=runner.go:931 msg="starting go runner" ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ollama[2033163]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ollama[2033163]: ggml_cuda_init: found 6 CUDA devices: ollama[2033163]: Device 0: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 1: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 2: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 3: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 4: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: Device 5: NVIDIA A40, compute capability 8.6, VMM: yes ollama[2033163]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so ollama[2033163]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so ollama[2033163]: time=2025-03-08T21:18:54.298+08:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=80 ollama[2033163]: time=2025-03-08T21:18:54.298+08:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:40267" ollama[2033163]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 36727 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 35201 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 36341 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA3 (NVIDIA A40) - 36341 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA4 (NVIDIA A40) - 36377 MiB free ollama[2033163]: llama_model_load_from_file_impl: using device CUDA5 (NVIDIA A40) - 36377 MiB free ollama[2033163]: time=2025-03-08T21:18:54.498+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" ollama[2033163]: llama_model_loader: loaded meta data with 34 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 (version GGUF V3 (latest)) ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. ollama[2033163]: llama_model_loader: - kv 0: general.architecture str = qwen2 ollama[2033163]: llama_model_loader: - kv 1: general.type str = model ollama[2033163]: llama_model_loader: - kv 2: general.name str = Qwen2.5 Coder 32B Instruct ollama[2033163]: llama_model_loader: - kv 3: general.finetune str = Instruct ollama[2033163]: llama_model_loader: - kv 4: general.basename str = Qwen2.5-Coder ollama[2033163]: llama_model_loader: - kv 5: general.size_label str = 32B ollama[2033163]: llama_model_loader: - kv 6: general.license str = apache-2.0 ollama[2033163]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-C... ollama[2033163]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 ollama[2033163]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 Coder 32B ollama[2033163]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen ollama[2033163]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-C... ollama[2033163]: llama_model_loader: - kv 12: general.tags arr[str,6] = ["code", "codeqwen", "chat", "qwen", ... ollama[2033163]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"] ollama[2033163]: llama_model_loader: - kv 14: qwen2.block_count u32 = 64 ollama[2033163]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768 ollama[2033163]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 5120 ollama[2033163]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 27648 ollama[2033163]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 40 ollama[2033163]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 8 ollama[2033163]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000 ollama[2033163]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 ollama[2033163]: llama_model_loader: - kv 22: general.file_type u32 = 15 ollama[2033163]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 ollama[2033163]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 ollama[2033163]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... ollama[2033163]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... ollama[2033163]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... ollama[2033163]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 ollama[2033163]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false ollama[2033163]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... ollama[2033163]: llama_model_loader: - kv 33: general.quantization_version u32 = 2 ollama[2033163]: llama_model_loader: - type f32: 321 tensors ollama[2033163]: llama_model_loader: - type q4_K: 385 tensors ollama[2033163]: llama_model_loader: - type q6_K: 65 tensors ollama[2033163]: print_info: file format = GGUF V3 (latest) ollama[2033163]: print_info: file type = Q4_K - Medium ollama[2033163]: print_info: file size = 18.48 GiB (4.85 BPW) ollama[2033163]: load: special tokens cache size = 22 ollama[2033163]: load: token to piece cache size = 0.9310 MB ollama[2033163]: print_info: arch = qwen2 ollama[2033163]: print_info: vocab_only = 0 ollama[2033163]: print_info: n_ctx_train = 32768 ollama[2033163]: print_info: n_embd = 5120 ollama[2033163]: print_info: n_layer = 64 ollama[2033163]: print_info: n_head = 40 ollama[2033163]: print_info: n_head_kv = 8 ollama[2033163]: print_info: n_rot = 128 ollama[2033163]: print_info: n_swa = 0 ollama[2033163]: print_info: n_embd_head_k = 128 ollama[2033163]: print_info: n_embd_head_v = 128 ollama[2033163]: print_info: n_gqa = 5 ollama[2033163]: print_info: n_embd_k_gqa = 1024 ollama[2033163]: print_info: n_embd_v_gqa = 1024 ollama[2033163]: print_info: f_norm_eps = 0.0e+00 ollama[2033163]: print_info: f_norm_rms_eps = 1.0e-06 ollama[2033163]: print_info: f_clamp_kqv = 0.0e+00 ollama[2033163]: print_info: f_max_alibi_bias = 0.0e+00 ollama[2033163]: print_info: f_logit_scale = 0.0e+00 ollama[2033163]: print_info: n_ff = 27648 ollama[2033163]: print_info: n_expert = 0 ollama[2033163]: print_info: n_expert_used = 0 ollama[2033163]: print_info: causal attn = 1 ollama[2033163]: print_info: pooling type = 0 ollama[2033163]: print_info: rope type = 2 ollama[2033163]: print_info: rope scaling = linear ollama[2033163]: print_info: freq_base_train = 1000000.0 ollama[2033163]: print_info: freq_scale_train = 1 ollama[2033163]: print_info: n_ctx_orig_yarn = 32768 ollama[2033163]: print_info: rope_finetuned = unknown ollama[2033163]: print_info: ssm_d_conv = 0 ollama[2033163]: print_info: ssm_d_inner = 0 ollama[2033163]: print_info: ssm_d_state = 0 ollama[2033163]: print_info: ssm_dt_rank = 0 ollama[2033163]: print_info: ssm_dt_b_c_rms = 0 ollama[2033163]: print_info: model type = 32B ollama[2033163]: print_info: model params = 32.76 B ollama[2033163]: print_info: general.name = Qwen2.5 Coder 32B Instruct ollama[2033163]: print_info: vocab type = BPE ollama[2033163]: print_info: n_vocab = 152064 ollama[2033163]: print_info: n_merges = 151387 ollama[2033163]: print_info: BOS token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOS token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOT token = 151645 '<|im_end|>' ollama[2033163]: print_info: PAD token = 151643 '<|endoftext|>' ollama[2033163]: print_info: LF token = 198 'Ċ' ollama[2033163]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' ollama[2033163]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' ollama[2033163]: print_info: FIM MID token = 151660 '<|fim_middle|>' ollama[2033163]: print_info: FIM PAD token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: FIM REP token = 151663 '<|repo_name|>' ollama[2033163]: print_info: FIM SEP token = 151664 '<|file_sep|>' ollama[2033163]: print_info: EOG token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOG token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOG token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: EOG token = 151663 '<|repo_name|>' ollama[2033163]: print_info: EOG token = 151664 '<|file_sep|>' ollama[2033163]: print_info: max token length = 256 ollama[2033163]: load_tensors: loading model tensors, this can take a while... (mmap = true) ollama[2033163]: load_tensors: offloading 64 repeating layers to GPU ollama[2033163]: load_tensors: offloading output layer to GPU ollama[2033163]: load_tensors: offloaded 65/65 layers to GPU ollama[2033163]: load_tensors: CUDA0 model buffer size = 3167.96 MiB ollama[2033163]: load_tensors: CUDA1 model buffer size = 3021.00 MiB ollama[2033163]: load_tensors: CUDA2 model buffer size = 3022.29 MiB ollama[2033163]: load_tensors: CUDA3 model buffer size = 3022.29 MiB ollama[2033163]: load_tensors: CUDA4 model buffer size = 2986.20 MiB ollama[2033163]: load_tensors: CUDA5 model buffer size = 3288.61 MiB ollama[2033163]: load_tensors: CPU_Mapped model buffer size = 417.66 MiB ollama[2033163]: llama_init_from_model: n_seq_max = 3 ollama[2033163]: llama_init_from_model: n_ctx = 98304 ollama[2033163]: llama_init_from_model: n_ctx_per_seq = 32768 ollama[2033163]: llama_init_from_model: n_batch = 1536 ollama[2033163]: llama_init_from_model: n_ubatch = 512 ollama[2033163]: llama_init_from_model: flash_attn = 1 ollama[2033163]: llama_init_from_model: freq_base = 1000000.0 ollama[2033163]: llama_init_from_model: freq_scale = 1 ollama[2033163]: llama_kv_cache_init: kv_size = 98304, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 64, can_shift = 1 ollama[2033163]: llama_kv_cache_init: CUDA0 KV buffer size = 2244.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA1 KV buffer size = 2244.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA2 KV buffer size = 2244.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA3 KV buffer size = 2244.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA4 KV buffer size = 2244.00 MiB ollama[2033163]: llama_kv_cache_init: CUDA5 KV buffer size = 1836.00 MiB ollama[2033163]: llama_init_from_model: KV self size = 13056.00 MiB, K (q8_0): 6528.00 MiB, V (q8_0): 6528.00 MiB ollama[2033163]: llama_init_from_model: CUDA_Host output buffer size = 1.80 MiB ollama[2033163]: llama_init_from_model: pipeline parallelism enabled (n_copies=4) ollama[2033163]: llama_init_from_model: CUDA0 compute buffer size = 1042.01 MiB ollama[2033163]: llama_init_from_model: CUDA1 compute buffer size = 562.01 MiB ollama[2033163]: llama_init_from_model: CUDA2 compute buffer size = 562.01 MiB ollama[2033163]: llama_init_from_model: CUDA3 compute buffer size = 562.01 MiB ollama[2033163]: llama_init_from_model: CUDA4 compute buffer size = 562.01 MiB ollama[2033163]: llama_init_from_model: CUDA5 compute buffer size = 731.02 MiB ollama[2033163]: llama_init_from_model: CUDA_Host compute buffer size = 778.02 MiB ollama[2033163]: llama_init_from_model: graph nodes = 1991 ollama[2033163]: llama_init_from_model: graph splits = 7 ollama[2033163]: time=2025-03-08T21:18:58.513+08:00 level=INFO source=server.go:596 msg="llama runner started in 5.02 seconds" ollama[2033163]: [GIN] 2025/03/08 - 21:18:58 | 200 | 16.524096975s | 127.0.0.1 | POST "/api/generate" ollama[2033163]: llama_model_loader: loaded meta data with 34 key-value pairs and 771 tensors from /media/data/ollama_model/blobs/sha256-ac3d1ba8aa77755dab3806d9024e9c385ea0d5b412d6bdf9157f8a4a7e9fc0d9 (version GGUF V3 (latest)) ollama[2033163]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. ollama[2033163]: llama_model_loader: - kv 0: general.architecture str = qwen2 ollama[2033163]: llama_model_loader: - kv 1: general.type str = model ollama[2033163]: llama_model_loader: - kv 2: general.name str = Qwen2.5 Coder 32B Instruct ollama[2033163]: llama_model_loader: - kv 3: general.finetune str = Instruct ollama[2033163]: llama_model_loader: - kv 4: general.basename str = Qwen2.5-Coder ollama[2033163]: llama_model_loader: - kv 5: general.size_label str = 32B ollama[2033163]: llama_model_loader: - kv 6: general.license str = apache-2.0 ollama[2033163]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-C... ollama[2033163]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 ollama[2033163]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 Coder 32B ollama[2033163]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen ollama[2033163]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-C... ollama[2033163]: llama_model_loader: - kv 12: general.tags arr[str,6] = ["code", "codeqwen", "chat", "qwen", ... ollama[2033163]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"] ollama[2033163]: llama_model_loader: - kv 14: qwen2.block_count u32 = 64 ollama[2033163]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768 ollama[2033163]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 5120 ollama[2033163]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 27648 ollama[2033163]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 40 ollama[2033163]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 8 ollama[2033163]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000 ollama[2033163]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 ollama[2033163]: llama_model_loader: - kv 22: general.file_type u32 = 15 ollama[2033163]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 ollama[2033163]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 ollama[2033163]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... ollama[2033163]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... ollama[2033163]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... ollama[2033163]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 ollama[2033163]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 ollama[2033163]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false ollama[2033163]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... ollama[2033163]: llama_model_loader: - kv 33: general.quantization_version u32 = 2 ollama[2033163]: llama_model_loader: - type f32: 321 tensors ollama[2033163]: llama_model_loader: - type q4_K: 385 tensors ollama[2033163]: llama_model_loader: - type q6_K: 65 tensors ollama[2033163]: print_info: file format = GGUF V3 (latest) ollama[2033163]: print_info: file type = Q4_K - Medium ollama[2033163]: print_info: file size = 18.48 GiB (4.85 BPW) ollama[2033163]: load: special tokens cache size = 22 ollama[2033163]: load: token to piece cache size = 0.9310 MB ollama[2033163]: print_info: arch = qwen2 ollama[2033163]: print_info: vocab_only = 1 ollama[2033163]: print_info: model type = ?B ollama[2033163]: print_info: model params = 32.76 B ollama[2033163]: print_info: general.name = Qwen2.5 Coder 32B Instruct ollama[2033163]: print_info: vocab type = BPE ollama[2033163]: print_info: n_vocab = 152064 ollama[2033163]: print_info: n_merges = 151387 ollama[2033163]: print_info: BOS token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOS token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOT token = 151645 '<|im_end|>' ollama[2033163]: print_info: PAD token = 151643 '<|endoftext|>' ollama[2033163]: print_info: LF token = 198 'Ċ' ollama[2033163]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' ollama[2033163]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' ollama[2033163]: print_info: FIM MID token = 151660 '<|fim_middle|>' ollama[2033163]: print_info: FIM PAD token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: FIM REP token = 151663 '<|repo_name|>' ollama[2033163]: print_info: FIM SEP token = 151664 '<|file_sep|>' ollama[2033163]: print_info: EOG token = 151643 '<|endoftext|>' ollama[2033163]: print_info: EOG token = 151645 '<|im_end|>' ollama[2033163]: print_info: EOG token = 151662 '<|fim_pad|>' ollama[2033163]: print_info: EOG token = 151663 '<|repo_name|>' ollama[2033163]: print_info: EOG token = 151664 '<|file_sep|>' ollama[2033163]: print_info: max token length = 256 ollama[2033163]: llama_model_load: vocab only - skipping tensors ollama[2033163]: [GIN] 2025/03/08 - 21:19:04 | 200 | 840.178362ms | 127.0.0.1 | POST "/api/chat" ollama[2033163]: [GIN] 2025/03/08 - 21:19:11 | 200 | 68.886µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:19:11 | 200 | 122.734µs | 127.0.0.1 | GET "/api/ps" ollama[2033163]: [GIN] 2025/03/08 - 21:19:46 | 200 | 27.2µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:19:46 | 200 | 40.331µs | 127.0.0.1 | GET "/api/ps" ollama[2033163]: [GIN] 2025/03/08 - 21:19:47 | 200 | 48.905µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:19:47 | 200 | 31.523µs | 127.0.0.1 | GET "/api/ps" ollama[2033163]: [GIN] 2025/03/08 - 21:20:48 | 200 | 3.380467ms | 127.0.0.1 | GET "/v1/models" ollama[2033163]: [GIN] 2025/03/08 - 21:22:01 | 200 | 38.6498034s | 127.0.0.1 | POST "/v1/chat/completions" ollama[2033163]: [GIN] 2025/03/08 - 21:22:38 | 200 | 36.352672616s | 127.0.0.1 | POST "/v1/chat/completions" ollama[2033163]: [GIN] 2025/03/08 - 21:32:01 | 200 | 58.535137ms | 127.0.0.1 | POST "/api/show" ollama[2033163]: [GIN] 2025/03/08 - 21:32:04 | 200 | 47.295µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:32:04 | 200 | 2.986628ms | 127.0.0.1 | GET "/api/tags" ollama[2033163]: [GIN] 2025/03/08 - 21:32:28 | 200 | 62.67µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:32:28 | 200 | 1.106065ms | 127.0.0.1 | POST "/api/generate" ollama[2033163]: time=2025-03-08T21:32:28.102+08:00 level=INFO source=images.go:432 msg="total blobs: 35" ollama[2033163]: time=2025-03-08T21:32:28.103+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" ollama[2033163]: time=2025-03-08T21:32:28.103+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=40 remote=127.0.0.1:46396 proto=HTTP/1.1 query="" ollama[2033163]: time=2025-03-08T21:32:28.105+08:00 level=INFO source=images.go:432 msg="total blobs: 35" ollama[2033163]: time=2025-03-08T21:32:28.799+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 2" ollama[2033163]: time=2025-03-08T21:32:28.799+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=36 remote=127.0.0.1:46396 proto=HTTP/1.1 query="" ollama[2033163]: [GIN] 2025/03/08 - 21:32:59 | 200 | 50.825µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:32:59 | 200 | 2.260465ms | 127.0.0.1 | POST "/api/generate" ollama[2033163]: time=2025-03-08T21:32:59.082+08:00 level=INFO source=images.go:432 msg="total blobs: 33" ollama[2033163]: time=2025-03-08T21:33:00.489+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 2" ollama[2033163]: time=2025-03-08T21:33:00.489+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=37 remote=127.0.0.1:57012 proto=HTTP/1.1 query="" ollama[2033163]: time=2025-03-08T21:33:00.493+08:00 level=INFO source=images.go:432 msg="total blobs: 31" ollama[2033163]: time=2025-03-08T21:33:03.671+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 5" ollama[2033163]: time=2025-03-08T21:33:03.671+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=37 remote=127.0.0.1:57012 proto=HTTP/1.1 query="" ollama[2033163]: time=2025-03-08T21:33:03.675+08:00 level=INFO source=images.go:432 msg="total blobs: 26" ollama[2033163]: time=2025-03-08T21:33:04.288+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 3" ollama[2033163]: time=2025-03-08T21:33:04.288+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=33 remote=127.0.0.1:57012 proto=HTTP/1.1 query="" ollama[2033163]: time=2025-03-08T21:33:04.323+08:00 level=INFO source=images.go:432 msg="total blobs: 23" ollama[2033163]: time=2025-03-08T21:33:05.727+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 3" ollama[2033163]: time=2025-03-08T21:33:05.727+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=33 remote=127.0.0.1:57012 proto=HTTP/1.1 query="" ollama[2033163]: time=2025-03-08T21:33:05.729+08:00 level=INFO source=images.go:432 msg="total blobs: 20" ollama[2033163]: time=2025-03-08T21:33:12.063+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 5" ollama[2033163]: time=2025-03-08T21:33:12.063+08:00 level=INFO source=server.go:154 msg=http status=200 method=DELETE path=/api/delete content-length=38 remote=127.0.0.1:57012 proto=HTTP/1.1 query="" ollama[2033163]: [GIN] 2025/03/08 - 21:33:37 | 200 | 26.754µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:33:37 | 200 | 600.829µs | 127.0.0.1 | GET "/api/tags" ollama[2033163]: [GIN] 2025/03/08 - 21:33:49 | 200 | 60.488µs | 127.0.0.1 | HEAD "/" ollama[2033163]: [GIN] 2025/03/08 - 21:33:49 | 200 | 77.212µs | 127.0.0.1 | GET "/api/ps" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.13

GiteaMirror added the bug label 2026-04-29 00:48:43 -05:00

GiteaMirror closed this issue 2026-04-29 00:48:45 -05:00

GiteaMirror commented 2026-04-29 00:48:47 -05:00

Author

Owner

Copy Link

@rick-github commented on GitHub (Mar 8, 2025):

The output of ollama ps is not accurate because num_gpu has been overridden and flash attention has been enabled. The values from ollama ps are calculated by the server before those options take effect in the runner.

qwen2.5-coder:32b would normally fit on one GPU. Because it's loaded after qwq, there is not enough VRAM on a single GPU to host the model, so ollama has to spread it across the GPUs. This results in duplicating data structures that would normally only occur on a single GPU, consuming more VRAM.

<!-- gh-comment-id:2708364720 --> @rick-github commented on GitHub (Mar 8, 2025): The output of `ollama ps` is not accurate because `num_gpu` has been overridden and flash attention has been enabled. The values from `ollama ps` are calculated by the server before those options take effect in the runner. qwen2.5-coder:32b would normally fit on one GPU. Because it's loaded after qwq, there is not enough VRAM on a single GPU to host the model, so ollama has to spread it across the GPUs. This results in duplicating data structures that would normally only occur on a single GPU, consuming more VRAM.

GiteaMirror commented 2026-04-29 00:48:52 -05:00

Author

Owner

Copy Link

@NGC13009 commented on GitHub (Mar 8, 2025):

The output of ollama ps is not accurate because num_gpu has been overridden and flash attention has been enabled. The values from ollama ps are calculated by the server before those options take effect in the runner.

qwen2.5-coder:32b would normally fit on one GPU. Because it's loaded after qwq, there is not enough VRAM on a single GPU to host the model, so ollama has to spread it across the GPUs. This results in duplicating data structures that would normally only occur on a single GPU, consuming more VRAM.

Is there any way to save on video memory overhead? For example, qwen2.5 is loaded on one GPU, while qwq is automatically distributed on top of other GPUs after loading. Currently both models are equally distributed across all GPUs regardless of the startup order

But the actual video memory overhead is not that big, when viewed with nvidia-smi

<!-- gh-comment-id:2708368033 --> @NGC13009 commented on GitHub (Mar 8, 2025): > The output of `ollama ps` is not accurate because `num_gpu` has been overridden and flash attention has been enabled. The values from `ollama ps` are calculated by the server before those options take effect in the runner. > > qwen2.5-coder:32b would normally fit on one GPU. Because it's loaded after qwq, there is not enough VRAM on a single GPU to host the model, so ollama has to spread it across the GPUs. This results in duplicating data structures that would normally only occur on a single GPU, consuming more VRAM. Is there any way to save on video memory overhead? For example, qwen2.5 is loaded on one GPU, while qwq is automatically distributed on top of other GPUs after loading. Currently both models are equally distributed across all GPUs regardless of the startup order But the actual video memory overhead is not that big, when viewed with `nvidia-smi`

GiteaMirror commented 2026-04-29 00:48:54 -05:00

Author

Owner

Copy Link

@rick-github commented on GitHub (Mar 8, 2025):

Load qwen2.5-coder:32b first.

<!-- gh-comment-id:2708368771 --> @rick-github commented on GitHub (Mar 8, 2025): Load qwen2.5-coder:32b first.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

hoyyeva/fix-claude-channels-env

parth-update-hermes-launch

hoyyeva/vscode-extension-docs-update

parth-gemma4-chat-template-renderer

parth-api-status-context-length

hoyyeva/wire-up-context-length

hoyyeva/claude-code-context-doc

jmorganca/investigate-issue-17046

hoyyeva/hermes-docs

jmorganca/agent-loop-style

hoyyeva/openclaw

parth-agent-loop

hoyyeva/ollama-vscode-extension

brucemacd/cache-metrics

brucemacd/hermes-desktop

hoyyeva/docs-vscode

parth-input-style-experiment

brucemacd/docs-glm52

hoyyeva/poc-docs

Parth/mlx-launch-recommendations

parth-first-time-app-cli-experience

test/darwin-xcode-pin

improve-cloud-model-recommendations

hoyyeva/goose-docs

jmorganca/context-limit-fixes

hoyyeva/qwen-doc

hoyyeva/vscode-docs

jmorganca/remove-mlx-imagegen-code

parth-copilot-token-length-defaults

hoyyeva/poolside-windows

laguna-support

jmorganca/harden-markdown-rendering

laguna-renderer-parser

laguna-llamacpp

codex/make-integration-hidden-and-lunchable

brucemacd/omp-docs

pdevine/gguf-mtp-oldstyle

hoyyeva/migrate-pi

hoyyeva/anthropic-local-image-path

parth-launch-codex-app

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth/hide-claude-desktop-till-release

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

v0.32.1-rc0

v0.32.0

v0.32.0-rc0

v0.31.2

v0.31.2-rc2

v0.31.2-rc1

v0.31.2-rc0

v0.31.1

v0.31.0

v0.30.12-rc0

v0.30.11

v0.30.11-rc1

v0.30.11-rc0

v0.30.10

v0.30.10-rc1

v0.30.10-rc0

v0.30.9

v0.30.9-rc2

v0.30.9-rc1

v0.30.9-rc0

v0.30.8

v0.30.8-rc0

v0.30.7

v0.30.7-rc1

v0.30.7-rc0

v0.30.6

v0.30.6-rc0

v0.30.5

v0.30.5-rc0

v0.30.4

v0.30.4-rc1

v0.30.4-rc0

v0.30.3

v0.30.2

v0.30.2-rc0

v0.30.1

v0.30.1-rc0

v0.30.0

v0.30.0-rc32

v0.30.0-rc31

v0.30.0-rc30

v0.30.0-rc29

v0.30.0-rc28

v0.30.0-rc27

v0.30.0-rc26

v0.30.0-rc25

v0.30.0-rc24

v0.30.0-rc23

v0.30.0-rc22

v0.30.0-rc21

v0.30.0-rc20

v0.30.0-rc19

v0.30.0-rc18

v0.25.0-rc0

v0.30.0-rc17

v0.30.0-rc16

v0.24.0-rc1

v0.24.0

v0.24.0-rc0

v0.23.4

v0.23.4-rc0

v0.30.0-rc15

v0.23.3

v0.23.3-rc1

v0.30.0-rc14

v0.23.3-rc0

v0.30.0-rc13

v0.30.0-rc12

v0.30.0-rc11

v0.30.0-rc10

v0.30.0-rc9

v0.30.0-rc8

v0.30.0-rc7

v0.30.0-rc6

v0.30.0-rc5

v0.23.2

v0.23.2-rc0

v0.30.0-rc4

v0.30.0-rc3

v0.30.0-rc2

v0.30.0-rc1

v0.30.0-rc0

v0.23.1

v0.23.1-rc0

v0.23.0

v0.23.0-rc0

v0.22.1

v0.22.1-rc1

v0.22.1-rc0

v0.22.0

v0.22.0-rc1

v0.21.3-rc0

v0.21.2-rc1

v0.21.2

v0.21.2-rc0

v0.21.1

v0.21.1-rc1

v0.21.1-rc0

v0.21.0

v0.21.0-rc1

v0.21.0-rc0

v0.20.8-rc0

v0.20.7

v0.20.7-rc1

v0.20.7-rc0

v0.20.6

v0.20.6-rc1

v0.20.6-rc0

v0.20.5

v0.20.5-rc2

v0.20.5-rc1

v0.20.5-rc0

v0.20.4

v0.20.4-rc2

v0.20.4-rc1

v0.20.4-rc0

v0.20.3

v0.20.3-rc0

v0.20.2

v0.20.1

v0.20.1-rc2

v0.20.1-rc1

v0.20.1-rc0

v0.20.0

v0.20.0-rc1

v0.20.0-rc0

v0.19.0

v0.19.0-rc2

v0.19.0-rc1

v0.19.0-rc0

v0.18.4-rc1

v0.18.4-rc0

v0.18.3

v0.18.3-rc2

v0.18.3-rc1

v0.18.3-rc0

v0.18.2

v0.18.2-rc1

v0.18.2-rc0

v0.18.1

v0.18.1-rc1

v0.18.1-rc0

v0.18.0

v0.18.0-rc2

v0.18.0-rc1

v0.18.0-rc0

v0.17.8-rc4

v0.17.8-rc3

v0.17.8-rc2

v0.17.8-rc1

v0.17.8-rc0

v0.17.7

v0.17.7-rc2

v0.17.7-rc1

v0.17.7-rc0

v0.17.6

v0.17.5

v0.17.4

v0.17.3

v0.17.2

v0.17.1

v0.17.1-rc2

v0.17.1-rc1

v0.17.1-rc0

v0.17.0

v0.17.0-rc2

v0.17.0-rc1

v0.17.0-rc0

v0.16.3

v0.16.3-rc2

v0.16.3-rc1

v0.16.3-rc0

v0.16.2

v0.16.2-rc0

v0.16.1

v0.16.0

v0.16.0-rc2

v0.16.0-rc0

v0.16.0-rc1

v0.15.6

v0.15.5

v0.15.5-rc5

v0.15.5-rc4

v0.15.5-rc3

v0.15.5-rc2

v0.15.5-rc1

v0.15.5-rc0

v0.15.4

v0.15.3

v0.15.2

v0.15.1

v0.15.1-rc1

v0.15.1-rc0

v0.15.0-rc6

v0.15.0

v0.15.0-rc5

v0.15.0-rc4

v0.15.0-rc3

v0.15.0-rc2

v0.15.0-rc1

v0.15.0-rc0

v0.14.3

v0.14.3-rc3

v0.14.3-rc2

v0.14.3-rc1

v0.14.3-rc0

v0.14.2

v0.14.2-rc1

v0.14.2-rc0

v0.14.1

v0.14.0-rc11

v0.14.0

v0.14.0-rc10

v0.14.0-rc9

v0.14.0-rc8

v0.14.0-rc7

v0.14.0-rc6

v0.14.0-rc5

v0.14.0-rc4

v0.14.0-rc3

v0.14.0-rc2

v0.14.0-rc1

v0.14.0-rc0

v0.13.5

v0.13.5-rc1

v0.13.5-rc0

v0.13.4-rc2

v0.13.4

v0.13.4-rc1

v0.13.4-rc0

v0.13.3

v0.13.3-rc1

v0.13.3-rc0

v0.13.2

v0.13.2-rc2

v0.13.2-rc1

v0.13.2-rc0

v0.13.1

v0.13.1-rc2

v0.13.1-rc1

v0.13.1-rc0

v0.13.0

v0.13.0-rc0

v0.12.11

v0.12.11-rc1

v0.12.11-rc0

v0.12.10

v0.12.10-rc1

v0.12.10-rc0

v0.12.9-rc0

v0.12.9

v0.12.8

v0.12.8-rc0

v0.12.7

v0.12.7-rc1

v0.12.7-rc0

v0.12.7-citest0

v0.12.6

v0.12.6-rc1

v0.12.6-rc0

v0.12.5

v0.12.5-rc0

v0.12.4

v0.12.4-rc7

v0.12.4-rc6

v0.12.4-rc5

v0.12.4-rc4

v0.12.4-rc3

v0.12.4-rc2

v0.12.4-rc1

v0.12.4-rc0

v0.12.3

v0.12.2

v0.12.2-rc0

v0.12.1

v0.12.1-rc1

v0.12.1-rc2

v0.12.1-rc0

v0.12.0

v0.12.0-rc1

v0.12.0-rc0

v0.11.11

v0.11.11-rc3

v0.11.11-rc2

v0.11.11-rc1

v0.11.11-rc0

v0.11.10

v0.11.9

v0.11.9-rc0

v0.11.8

v0.11.8-rc0

v0.11.7-rc1

v0.11.7-rc0

v0.11.7

v0.11.6

v0.11.6-rc0

v0.11.5-rc4

v0.11.5-rc3

v0.11.5

v0.11.5-rc5

v0.11.5-rc2

v0.11.5-rc1

v0.11.5-rc0

v0.11.4

v0.11.4-rc0

v0.11.3

v0.11.3-rc0

v0.11.2

v0.11.1

v0.11.0-rc0

v0.11.0-rc1

v0.11.0-rc2

v0.11.0

v0.10.2-int1

v0.10.1

v0.10.0

v0.10.0-rc4

v0.10.0-rc3

v0.10.0-rc2

v0.10.0-rc1

v0.10.0-rc0

v0.9.7-rc1

v0.9.7-rc0

v0.9.6

v0.9.6-rc0

v0.9.6-ci0

v0.9.5

v0.9.4-rc5

v0.9.4-rc6

v0.9.4

v0.9.4-rc3

v0.9.4-rc4

v0.9.4-rc1

v0.9.4-rc2

v0.9.4-rc0

v0.9.3

v0.9.3-rc5

v0.9.4-citest0

v0.9.3-rc4

v0.9.3-rc3

v0.9.3-rc2

v0.9.3-rc1

v0.9.3-rc0

v0.9.2

v0.9.1

v0.9.1-rc1

v0.9.1-rc0

v0.9.1-ci1

v0.9.1-ci0

v0.9.0

v0.9.0-rc0

v0.8.0

v0.8.0-rc0

v0.7.1-rc2

v0.7.1

v0.7.1-rc1

v0.7.1-rc0

v0.7.0

v0.7.0-rc1

v0.7.0-rc0

v0.6.9-rc0

v0.6.8

v0.6.8-rc0

v0.6.7

v0.6.7-rc2

v0.6.7-rc1

v0.6.7-rc0

v0.6.6

v0.6.6-rc2

v0.6.6-rc1

v0.6.6-rc0

v0.6.5-rc1

v0.6.5

v0.6.5-rc0

v0.6.4-rc0

v0.6.4

v0.6.3-rc1

v0.6.3

v0.6.3-rc0

v0.6.2

v0.6.2-rc0

v0.6.1

v0.6.1-rc0

v0.6.0-rc0

v0.6.0

v0.5.14-rc0

v0.5.13

v0.5.13-rc6

v0.5.13-rc5

v0.5.13-rc4

v0.5.13-rc3

v0.5.13-rc2

v0.5.13-rc1

v0.5.13-rc0

v0.5.12

v0.5.12-rc1

v0.5.12-rc0

v0.5.11

v0.5.10

v0.5.9

v0.5.9-rc0

v0.5.8-rc13

v0.5.8

v0.5.8-rc12

v0.5.8-rc11

v0.5.8-rc10

v0.5.8-rc9

v0.5.8-rc8

v0.5.8-rc7

v0.5.8-rc6

v0.5.8-rc5

v0.5.8-rc4

v0.5.8-rc3

v0.5.8-rc2

v0.5.8-rc1

v0.5.8-rc0

v0.5.7

v0.5.6

v0.5.5

v0.5.5-rc0

v0.5.4

v0.5.3

v0.5.3-rc0

v0.5.2

v0.5.2-rc3

v0.5.2-rc2

v0.5.2-rc1

v0.5.2-rc0

v0.5.1

v0.5.0

v0.5.0-rc1

v0.4.8-rc0

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.3-rc0

v0.4.2

v0.4.2-rc1

v0.4.2-rc0

v0.4.1

v0.4.1-rc0

v0.4.0

v0.4.0-rc8

v0.4.0-rc7

v0.4.0-rc6

v0.4.0-rc5

v0.4.0-rc4

v0.4.0-rc3

v0.4.0-rc2

v0.4.0-rc1

v0.4.0-rc0

v0.4.0-ci3

v0.3.14

v0.3.14-rc0

v0.3.13

v0.3.12

v0.3.12-rc5

v0.3.12-rc4

v0.3.12-rc3

v0.3.12-rc2

v0.3.12-rc1

v0.3.11

v0.3.11-rc4

v0.3.11-rc3

v0.3.11-rc2

v0.3.11-rc1

v0.3.10

v0.3.10-rc1

v0.3.9

v0.3.8

v0.3.7

v0.3.7-rc6

v0.3.7-rc5

v0.3.7-rc4

v0.3.7-rc3

v0.3.7-rc2

v0.3.7-rc1

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.8

v0.2.8-rc2

v0.2.8-rc1

v0.2.7

v0.2.6

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.2-rc2

v0.2.2-rc1

v0.2.1

v0.2.0

v0.1.49-rc14

v0.1.49-rc13

v0.1.49-rc12

v0.1.49-rc11

v0.1.49-rc10

v0.1.49-rc9

v0.1.49-rc8

v0.1.49-rc7

v0.1.49-rc6

v0.1.49-rc4

v0.1.49-rc5

v0.1.49-rc3

v0.1.49-rc2

v0.1.49-rc1

v0.1.48

v0.1.47

v0.1.46

v0.1.45-rc5

v0.1.45

v0.1.45-rc4

v0.1.45-rc3

v0.1.45-rc2

v0.1.45-rc1

v0.1.44

v0.1.43

v0.1.42

v0.1.41

v0.1.40

v0.1.40-rc1

v0.1.39

v0.1.39-rc2

v0.1.39-rc1

v0.1.38

v0.1.37

v0.1.36

v0.1.35

v0.1.35-rc1

v0.1.34

v0.1.34-rc1

v0.1.33

v0.1.33-rc7

v0.1.33-rc6

v0.1.33-rc5

v0.1.33-rc4

v0.1.33-rc3

v0.1.33-rc2

v0.1.33-rc1

v0.1.32

v0.1.32-rc2

v0.1.32-rc1

v0.1.31

v0.1.30

v0.1.29

v0.1.28

v0.1.27

v0.1.26

v0.1.25

v0.1.24

v0.1.23

v0.1.22

v0.1.21

v0.1.20

v0.1.19

v0.1.18

v0.1.17

v0.1.16

v0.1.15

v0.1.14

v0.1.13

v0.1.12

v0.1.11

v0.1.10

v0.1.9

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

amd

api

app

bug

build

cli

cloud

compatibility

context-length

create

docker

documentation

embeddings

feature request

feedback wanted

good first issue

gpt-oss

gpu

harmony

help wanted

image

install

intel

js

launch

linux

macos

memory

mlx

model

needs more info

networking

nvidia

ollama.com

performance

pull-request

Mirrored from GitHub Pull Request

python

question

registry

rendering

thinking

tools

top

vulkan

windows

wsl

No Label bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#52770