github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-13 17:34:40 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9016] gpu VRAM usage didn't recover within timeout - OLLAMA_LOAD_TIMEOUT:10m0s #5863

New Issue
Closed
opened 2026-04-12 17:12:11 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 17:12:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @rwlove on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9016

What is the issue?

I'm a homelab'er, running ollama in a kubernetes cluster.

Everything was working fine until I moved my models directory to persistent storage. Since that change I hit the gpu VRAM usage didn't recover within timeout error. I found an issue here that stated to update OLLAMA_LOAD_TIMEOUT:10m0s, so I made that change, but it did not seem to make a difference. In the ollama output below you can clearly see that the variable is set, but ollama seems to timeout very quickly, it's certainly not waiting 10m.

Relevant log output

➜  home-ops git:(main) ✗ kubectl -n ai logs ollama-0 -f
Defaulted container "main" out of: main, k8tz (init)
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE0VK31uUIovZf/5Ix0DhFmhw2KL33Hv7GL5yObBu/Lr

2025/02/11 09:23:02 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:10m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-11T09:23:02.559-05:00 level=INFO source=images.go:432 msg="total blobs: 15"
time=2025-02-11T09:23:02.560-05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-11T09:23:02.560-05:00 level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.8)"
time=2025-02-11T09:23:02.561-05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-11T09:23:02.821-05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-525f2a40-7b8d-39c6-019b-50e65fe48cae library=cuda variant=v12 compute=6.1 driver=12.7 name="Tesla P40" total="23.9 GiB" available="21.0 GiB"
time=2025-02-11T09:24:11.492-05:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-525f2a40-7b8d-39c6-019b-50e65fe48cae parallel=4 available=22528262144 required="6.5 GiB"
time=2025-02-11T09:24:11.736-05:00 level=INFO source=server.go:100 msg="system memory" total="55.3 GiB" free="43.1 GiB" free_swap="0 B"
time=2025-02-11T09:24:11.737-05:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[21.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.9 GiB" memory.weights.repeating="4.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
time=2025-02-11T09:24:11.738-05:00 level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 35837"
time=2025-02-11T09:24:11.739-05:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-11T09:24:11.739-05:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-11T09:24:11.740-05:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-11T09:24:11.763-05:00 level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-11T09:24:11.763-05:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=10
time=2025-02-11T09:24:11.764-05:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35837"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1, VMM: yes
load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so
time=2025-02-11T09:24:11.992-05:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model"
llama_load_model_from_file: using device CUDA0 (Tesla P40) - 21484 MiB free
llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 Distill Llama 8B
llama_model_loader: - kv   3:                           general.basename str              = DeepSeek-R1-Distill-Llama
llama_model_loader: - kv   4:                         general.size_label str              = 8B
llama_model_loader: - kv   5:                          llama.block_count u32              = 32
llama_model_loader: - kv   6:                       llama.context_length u32              = 131072
llama_model_loader: - kv   7:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   8:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   9:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  10:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  12:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  13:                          general.file_type u32              = 15
llama_model_loader: - kv  14:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  15:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  19:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  20:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  21:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  22:                tokenizer.ggml.eos_token_id u32              = 128001
llama_model_loader: - kv  23:            tokenizer.ggml.padding_token_id u32              = 128001
llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  25:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  26:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  27:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 8.03 B
llm_load_print_meta: model size       = 4.58 GiB (4.89 BPW) 
llm_load_print_meta: general.name     = DeepSeek R1 Distill Llama 8B
llm_load_print_meta: BOS token        = 128000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token        = 128008 '<|eom_id|>'
llm_load_print_meta: PAD token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOG token        = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256


time=2025-02-11T09:25:11.021-05:00 level=WARN source=server.go:565 msg="client connection closed before server finished loading, aborting load"
time=2025-02-11T09:25:11.022-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled"
[GIN] 2025/02/11 - 09:25:11 | 499 |          1m0s |     192.168.1.1 | POST     "/api/chat"
time=2025-02-11T09:25:16.109-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.087267396 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
time=2025-02-11T09:25:16.359-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.336854305 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
time=2025-02-11T09:25:16.609-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58699622 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be

OS

Linux, Docker

GPU

Nvidia

CPU

Intel

Ollama version

0.5.8@sha256:1526317a5050bf58dae1101caea8820edd9d78892b3d008ef9995b2e3c77537f

Originally created by @rwlove on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9016 ### What is the issue? I'm a homelab'er, running ollama in a kubernetes cluster. Everything was working fine until I moved my models directory to persistent storage. Since that change I hit the `gpu VRAM usage didn't recover within timeout` error. I found an issue here that stated to update `OLLAMA_LOAD_TIMEOUT:10m0s`, so I made that change, but it did not seem to make a difference. In the ollama output below you can clearly see that the variable is set, but ollama seems to timeout very quickly, it's certainly not waiting 10m. ### Relevant log output ```shell ➜ home-ops git:(main) ✗ kubectl -n ai logs ollama-0 -f Defaulted container "main" out of: main, k8tz (init) Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE0VK31uUIovZf/5Ix0DhFmhw2KL33Hv7GL5yObBu/Lr 2025/02/11 09:23:02 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:10m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-02-11T09:23:02.559-05:00 level=INFO source=images.go:432 msg="total blobs: 15" time=2025-02-11T09:23:02.560-05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-11T09:23:02.560-05:00 level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.8)" time=2025-02-11T09:23:02.561-05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-02-11T09:23:02.821-05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-525f2a40-7b8d-39c6-019b-50e65fe48cae library=cuda variant=v12 compute=6.1 driver=12.7 name="Tesla P40" total="23.9 GiB" available="21.0 GiB" time=2025-02-11T09:24:11.492-05:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-525f2a40-7b8d-39c6-019b-50e65fe48cae parallel=4 available=22528262144 required="6.5 GiB" time=2025-02-11T09:24:11.736-05:00 level=INFO source=server.go:100 msg="system memory" total="55.3 GiB" free="43.1 GiB" free_swap="0 B" time=2025-02-11T09:24:11.737-05:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[21.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.9 GiB" memory.weights.repeating="4.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" time=2025-02-11T09:24:11.738-05:00 level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 35837" time=2025-02-11T09:24:11.739-05:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-11T09:24:11.739-05:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-11T09:24:11.740-05:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-11T09:24:11.763-05:00 level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-11T09:24:11.763-05:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=10 time=2025-02-11T09:24:11.764-05:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35837" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: Tesla P40, compute capability 6.1, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so time=2025-02-11T09:24:11.992-05:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model" llama_load_model_from_file: using device CUDA0 (Tesla P40) - 21484 MiB free llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 8B llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama llama_model_loader: - kv 4: general.size_label str = 8B llama_model_loader: - kv 5: llama.block_count u32 = 32 llama_model_loader: - kv 6: llama.context_length u32 = 131072 llama_model_loader: - kv 7: llama.embedding_length u32 = 4096 llama_model_loader: - kv 8: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 9: llama.attention.head_count u32 = 32 llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: llama.vocab_size u32 = 128256 llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 17: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 128001 llama_model_loader: - kv 23: tokenizer.ggml.padding_token_id u32 = 128001 llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 25: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 26: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 27: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.7999 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 8.03 B llm_load_print_meta: model size = 4.58 GiB (4.89 BPW) llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 8B llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: EOM token = 128008 '<|eom_id|>' llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOG token = 128008 '<|eom_id|>' llm_load_print_meta: EOG token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 time=2025-02-11T09:25:11.021-05:00 level=WARN source=server.go:565 msg="client connection closed before server finished loading, aborting load" time=2025-02-11T09:25:11.022-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled" [GIN] 2025/02/11 - 09:25:11 | 499 | 1m0s | 192.168.1.1 | POST "/api/chat" time=2025-02-11T09:25:16.109-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.087267396 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be time=2025-02-11T09:25:16.359-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.336854305 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be time=2025-02-11T09:25:16.609-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58699622 model=/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be ``` ### OS Linux, Docker ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.8@sha256:1526317a5050bf58dae1101caea8820edd9d78892b3d008ef9995b2e3c77537f
GiteaMirror added the bug label 2026-04-12 17:12:11 -05:00
GiteaMirror commented 2026-04-12 17:12:12 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 11, 2025):

time=2025-02-11T09:25:11.021-05:00 level=WARN source=server.go:565 msg="client connection closed before server finished loading, aborting load"
[GIN] 2025/02/11 - 09:25:11 | 499 |          1m0s |     192.168.1.1 | POST     "/api/chat"

Your client has a timeout of 1 minute and it quit the connection before the model finished loading.

<!-- gh-comment-id:2651100398 --> @rick-github commented on GitHub (Feb 11, 2025): ``` time=2025-02-11T09:25:11.021-05:00 level=WARN source=server.go:565 msg="client connection closed before server finished loading, aborting load" [GIN] 2025/02/11 - 09:25:11 | 499 | 1m0s | 192.168.1.1 | POST "/api/chat" ``` Your client has a timeout of 1 minute and it quit the connection before the model finished loading.
GiteaMirror commented 2026-04-12 17:12:12 -05:00
Author
Owner
Copy Link
Copy Source

@rwlove commented on GitHub (Feb 11, 2025):

Interesting. I tried with two different apps, oterm and frigate, and they both have this problem since moving to persistent storage. I guess I'll have to figure out how to change the client timeout period for every app using ollama if I want persistent storage for my models.

<!-- gh-comment-id:2651188525 --> @rwlove commented on GitHub (Feb 11, 2025): Interesting. I tried with two different apps, oterm and frigate, and they both have this problem since moving to persistent storage. I guess I'll have to figure out how to change the client timeout period for every app using ollama if I want persistent storage for my models.
GiteaMirror commented 2026-04-12 17:12:13 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 11, 2025):

You can pre-load a model before using a client.

ollama run deepseek-r1:8b ''

or

curl localhost:11434/api/generate -d '{"model":"deepseek-r1:8b"}'
<!-- gh-comment-id:2651200205 --> @rick-github commented on GitHub (Feb 11, 2025): You can pre-load a model before using a client. ``` ollama run deepseek-r1:8b '' ``` or ``` curl localhost:11434/api/generate -d '{"model":"deepseek-r1:8b"}' ```
GiteaMirror commented 2026-04-12 17:12:13 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 11, 2025):

You might also want to set OLLAMA_KEEP_ALIVE so a model is not unloaded after 5 minutes.

<!-- gh-comment-id:2651209070 --> @rick-github commented on GitHub (Feb 11, 2025): You might also want to set [`OLLAMA_KEEP_ALIVE`](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately) so a model is not unloaded after 5 minutes.
GiteaMirror commented 2026-04-12 17:12:14 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 11, 2025):

I also see that kubernetes has built-in proxies. Since two different clients timed out, it might be a configuration option for your cluster rather than for the individual clients.

<!-- gh-comment-id:2651234284 --> @rick-github commented on GitHub (Feb 11, 2025): I also see that kubernetes has [built-in proxies](https://kubernetes.io/docs/concepts/cluster-administration/proxies/). Since two different clients timed out, it might be a configuration option for your cluster rather than for the individual clients.
GiteaMirror commented 2026-04-12 17:12:14 -05:00
Author
Owner
Copy Link
Copy Source

@rwlove commented on GitHub (Feb 11, 2025):

Thanks. This is all a bit rough though. I can't just use Ollama as I did before. I can't seem to find a way to add a longer timeout to either Frigate of Oterm natively and I have other apps that I haven't looked into yet.

The persistent storage is an NVMe drive, but I guess it has some abstraction on top of it that might slow it down a bit. I'm surprised that this slows things down so much that it's causing these timeouts. It's Longhorn CSI backed by NVMe drives, one of the replicas is local on the Ollama/GPU node, but I suppose that is beyond the scope of this community.

<!-- gh-comment-id:2651234854 --> @rwlove commented on GitHub (Feb 11, 2025): Thanks. This is all a bit rough though. I can't just use Ollama as I did before. I can't seem to find a way to add a longer timeout to either Frigate of Oterm natively and I have other apps that I haven't looked into yet. The persistent storage is an NVMe drive, but I guess it has some abstraction on top of it that might slow it down a bit. I'm surprised that this slows things down so much that it's causing these timeouts. It's Longhorn CSI backed by NVMe drives, one of the replicas is local on the Ollama/GPU node, but I suppose that is beyond the scope of this community.
GiteaMirror commented 2026-04-12 17:12:14 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 11, 2025):

The model is 5G and is taking more than 60 seconds to load, which means a transfer rate of at least 83MB/s. If your home lab is just GigE, it might just be a matter of network bandwidth.

<!-- gh-comment-id:2651245984 --> @rick-github commented on GitHub (Feb 11, 2025): The model is 5G and is taking more than 60 seconds to load, which means a transfer rate of at least 83MB/s. If your home lab is just GigE, it might just be a matter of network bandwidth.
GiteaMirror commented 2026-04-12 17:12:15 -05:00
Author
Owner
Copy Link
Copy Source

@rwlove commented on GitHub (Feb 11, 2025):

Thanks for the help. I reduced my models storage volume to 1 replica and made it's locality strict-local, so there's not chance of the volume being anywhere other than on the node Ollama is running on. So far, so good - I haven't seen a timeout yet.

Thanks a ton, @rick-github!

<!-- gh-comment-id:2651822683 --> @rwlove commented on GitHub (Feb 11, 2025): Thanks for the help. I reduced my models storage volume to 1 replica and made it's locality `strict-local`, so there's not chance of the volume being anywhere other than on the node Ollama is running on. So far, so good - I haven't seen a timeout yet. Thanks a ton, @rick-github!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5863