github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #4861] Jetson - "ollama run" command loads until timeout #3075

New Issue
Closed
opened 2026-04-12 13:30:49 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-04-12 13:30:49 -05:00
Owner
Copy Link

Originally created by @Vassar-HARPER-Project on GitHub (Jun 6, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4861

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

Upon running "ollama run gemma:2b" (though this happens for all tested models: llama3, phi, tinyllama), the loading animation appears and after ~5 minutes (estimate, untimed), the response / result of the command is:
Error: timed out waiting for llama runner to start - progress 1.00 -

the server shows this log for this command:

2024/06/06 11:21:53 routes.go:1007: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
time=2024-06-06T11:21:53.848-04:00 level=INFO source=images.go:729 msg="total blobs: 11"
time=2024-06-06T11:21:53.849-04:00 level=INFO source=images.go:736 msg="total unused blobs removed: 0"
time=2024-06-06T11:21:53.849-04:00 level=INFO source=routes.go:1053 msg="Listening on 127.0.0.1:11434 (version 0.1.41)"
time=2024-06-06T11:21:53.850-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3794080172/runners
time=2024-06-06T11:21:58.984-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cuda_v11]"
time=2024-06-06T11:21:59.082-04:00 level=INFO source=types.go:71 msg="inference compute" id=GPU-42638932-6929-58db-a006-34d50a6799c1 library=cuda compute=8.7 driver=11.4 name=Orin total="29.9 GiB" available="21.7 GiB"
[GIN] 2024/06/06 - 11:22:14 | 200 |      64.512µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/06/06 - 11:22:14 | 200 |    1.232036ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2024/06/06 - 11:22:14 | 200 |     717.058µs |       127.0.0.1 | POST     "/api/show"
time=2024-06-06T11:22:16.239-04:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=19 memory.available="21.7 GiB" memory.required.full="2.6 GiB" memory.required.partial="2.6 GiB" memory.required.kv="36.0 MiB" memory.weights.total="1.6 GiB" memory.weights.repeating="1.0 GiB" memory.weights.nonrepeating="531.5 MiB" memory.graph.full="504.2 MiB" memory.graph.partial="918.6 MiB"
time=2024-06-06T11:22:16.239-04:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=19 memory.available="21.7 GiB" memory.required.full="2.6 GiB" memory.required.partial="2.6 GiB" memory.required.kv="36.0 MiB" memory.weights.total="1.6 GiB" memory.weights.repeating="1.0 GiB" memory.weights.nonrepeating="531.5 MiB" memory.graph.full="504.2 MiB" memory.graph.partial="918.6 MiB"
time=2024-06-06T11:22:16.240-04:00 level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama3794080172/runners/cuda_v11/ollama_llama_server --model /home/harper/.ollama/models/blobs/sha256-c1864a5eb19305c40519da12cc543519e48a0697ecd30e15d5ac228644957d12 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 19 --parallel 1 --port 42781"
time=2024-06-06T11:22:16.240-04:00 level=INFO source=sched.go:338 msg="loaded runners" count=1
time=2024-06-06T11:22:16.240-04:00 level=INFO source=server.go:529 msg="waiting for llama runner to start responding"
time=2024-06-06T11:22:16.241-04:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="5921b8f" tid="281473278327040" timestamp=1717687336
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="281473278327040" timestamp=1717687336 total_threads=8
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="42781" tid="281473278327040" timestamp=1717687336
llama_model_loader: loaded meta data with 21 key-value pairs and 164 tensors from /home/harper/.ollama/models/blobs/sha256-c1864a5eb19305c40519da12cc543519e48a0697ecd30e15d5ac228644957d12 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma
llama_model_loader: - kv   1:                               general.name str              = gemma-2b-it
llama_model_loader: - kv   2:                       gemma.context_length u32              = 8192
llama_model_loader: - kv   3:                          gemma.block_count u32              = 18
llama_model_loader: - kv   4:                     gemma.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  gemma.feed_forward_length u32              = 16384
llama_model_loader: - kv   6:                 gemma.attention.head_count u32              = 8
llama_model_loader: - kv   7:              gemma.attention.head_count_kv u32              = 1
llama_model_loader: - kv   8:                 gemma.attention.key_length u32              = 256
llama_model_loader: - kv   9:               gemma.attention.value_length u32              = 256
llama_model_loader: - kv  10:     gemma.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  14:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  15:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,256128]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
time=2024-06-06T11:22:16.493-04:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,256128]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,256128]  = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:   37 tensors
llama_model_loader: - type q4_0:  126 tensors
llama_model_loader: - type q8_0:    1 tensors
llm_load_vocab: special tokens cache size = 388
llm_load_vocab: token to piece cache size = 3.2028 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256128
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 8
llm_load_print_meta: n_head_kv        = 1
llm_load_print_meta: n_layer          = 18
llm_load_print_meta: n_rot            = 256
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 256
llm_load_print_meta: n_embd_v_gqa     = 256
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 16384
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 2B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 2.51 B
llm_load_print_meta: model size       = 1.56 GiB (5.34 BPW) 
llm_load_print_meta: general.name     = gemma-2b-it
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_print_meta: EOT token        = 107 '<end_of_turn>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Orin, compute capability 8.7, VMM: yes
llm_load_tensors: ggml ctx size =    0.17 MiB
llm_load_tensors: offloading 18 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 19/19 layers to GPU
llm_load_tensors:        CPU buffer size =   531.52 MiB
llm_load_tensors:      CUDA0 buffer size =  1594.93 MiB
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =    36.00 MiB
llama_new_context_with_model: KV self size  =   36.00 MiB, K (f16):   18.00 MiB, V (f16):   18.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.98 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   504.25 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =     8.01 MiB
llama_new_context_with_model: graph nodes  = 601
llama_new_context_with_model: graph splits = 2
time=2024-06-06T11:29:16.920-04:00 level=ERROR source=sched.go:344 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 1.00 - "
[GIN] 2024/06/06 - 11:29:16 | 500 |          7m2s |       127.0.0.1 | POST     "/api/chat"
time=2024-06-06T11:29:22.037-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.117139389
time=2024-06-06T11:29:22.288-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.367497274
time=2024-06-06T11:29:22.537-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.616874999

If it helps, this is running on a Jetson AGX Orin with 32GB of memory

OS

Linux

GPU

Nvidia

CPU

Other: 8-core NVIDIA Arm® Cortex A78AE v8.2 64-bit CPU 2MB L2 + 4MB L3

Ollama version

0.1.41

Originally created by @Vassar-HARPER-Project on GitHub (Jun 6, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4861 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? Upon running "ollama run gemma:2b" (though this happens for all tested models: llama3, phi, tinyllama), the loading animation appears and after ~5 minutes (estimate, untimed), the response / result of the command is: `Error: timed out waiting for llama runner to start - progress 1.00 - ` the server shows this log for this command: ``` 2024/06/06 11:21:53 routes.go:1007: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]" time=2024-06-06T11:21:53.848-04:00 level=INFO source=images.go:729 msg="total blobs: 11" time=2024-06-06T11:21:53.849-04:00 level=INFO source=images.go:736 msg="total unused blobs removed: 0" time=2024-06-06T11:21:53.849-04:00 level=INFO source=routes.go:1053 msg="Listening on 127.0.0.1:11434 (version 0.1.41)" time=2024-06-06T11:21:53.850-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3794080172/runners time=2024-06-06T11:21:58.984-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cuda_v11]" time=2024-06-06T11:21:59.082-04:00 level=INFO source=types.go:71 msg="inference compute" id=GPU-42638932-6929-58db-a006-34d50a6799c1 library=cuda compute=8.7 driver=11.4 name=Orin total="29.9 GiB" available="21.7 GiB" [GIN] 2024/06/06 - 11:22:14 | 200 | 64.512µs | 127.0.0.1 | HEAD "/" [GIN] 2024/06/06 - 11:22:14 | 200 | 1.232036ms | 127.0.0.1 | POST "/api/show" [GIN] 2024/06/06 - 11:22:14 | 200 | 717.058µs | 127.0.0.1 | POST "/api/show" time=2024-06-06T11:22:16.239-04:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=19 memory.available="21.7 GiB" memory.required.full="2.6 GiB" memory.required.partial="2.6 GiB" memory.required.kv="36.0 MiB" memory.weights.total="1.6 GiB" memory.weights.repeating="1.0 GiB" memory.weights.nonrepeating="531.5 MiB" memory.graph.full="504.2 MiB" memory.graph.partial="918.6 MiB" time=2024-06-06T11:22:16.239-04:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=19 memory.available="21.7 GiB" memory.required.full="2.6 GiB" memory.required.partial="2.6 GiB" memory.required.kv="36.0 MiB" memory.weights.total="1.6 GiB" memory.weights.repeating="1.0 GiB" memory.weights.nonrepeating="531.5 MiB" memory.graph.full="504.2 MiB" memory.graph.partial="918.6 MiB" time=2024-06-06T11:22:16.240-04:00 level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama3794080172/runners/cuda_v11/ollama_llama_server --model /home/harper/.ollama/models/blobs/sha256-c1864a5eb19305c40519da12cc543519e48a0697ecd30e15d5ac228644957d12 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 19 --parallel 1 --port 42781" time=2024-06-06T11:22:16.240-04:00 level=INFO source=sched.go:338 msg="loaded runners" count=1 time=2024-06-06T11:22:16.240-04:00 level=INFO source=server.go:529 msg="waiting for llama runner to start responding" time=2024-06-06T11:22:16.241-04:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="5921b8f" tid="281473278327040" timestamp=1717687336 INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="281473278327040" timestamp=1717687336 total_threads=8 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="42781" tid="281473278327040" timestamp=1717687336 llama_model_loader: loaded meta data with 21 key-value pairs and 164 tensors from /home/harper/.ollama/models/blobs/sha256-c1864a5eb19305c40519da12cc543519e48a0697ecd30e15d5ac228644957d12 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = gemma llama_model_loader: - kv 1: general.name str = gemma-2b-it llama_model_loader: - kv 2: gemma.context_length u32 = 8192 llama_model_loader: - kv 3: gemma.block_count u32 = 18 llama_model_loader: - kv 4: gemma.embedding_length u32 = 2048 llama_model_loader: - kv 5: gemma.feed_forward_length u32 = 16384 llama_model_loader: - kv 6: gemma.attention.head_count u32 = 8 llama_model_loader: - kv 7: gemma.attention.head_count_kv u32 = 1 llama_model_loader: - kv 8: gemma.attention.key_length u32 = 256 llama_model_loader: - kv 9: gemma.attention.value_length u32 = 256 llama_model_loader: - kv 10: gemma.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 11: tokenizer.ggml.model str = llama llama_model_loader: - kv 12: tokenizer.ggml.bos_token_id u32 = 2 llama_model_loader: - kv 13: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 14: tokenizer.ggml.padding_token_id u32 = 0 llama_model_loader: - kv 15: tokenizer.ggml.unknown_token_id u32 = 3 llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,256128] = ["<pad>", "<eos>", "<bos>", "<unk>", ... time=2024-06-06T11:22:16.493-04:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,256128] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,256128] = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 19: general.quantization_version u32 = 2 llama_model_loader: - kv 20: general.file_type u32 = 2 llama_model_loader: - type f32: 37 tensors llama_model_loader: - type q4_0: 126 tensors llama_model_loader: - type q8_0: 1 tensors llm_load_vocab: special tokens cache size = 388 llm_load_vocab: token to piece cache size = 3.2028 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = gemma llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 256128 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 8192 llm_load_print_meta: n_embd = 2048 llm_load_print_meta: n_head = 8 llm_load_print_meta: n_head_kv = 1 llm_load_print_meta: n_layer = 18 llm_load_print_meta: n_rot = 256 llm_load_print_meta: n_embd_head_k = 256 llm_load_print_meta: n_embd_head_v = 256 llm_load_print_meta: n_gqa = 8 llm_load_print_meta: n_embd_k_gqa = 256 llm_load_print_meta: n_embd_v_gqa = 256 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 8192 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 2B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 2.51 B llm_load_print_meta: model size = 1.56 GiB (5.34 BPW) llm_load_print_meta: general.name = gemma-2b-it llm_load_print_meta: BOS token = 2 '<bos>' llm_load_print_meta: EOS token = 1 '<eos>' llm_load_print_meta: UNK token = 3 '<unk>' llm_load_print_meta: PAD token = 0 '<pad>' llm_load_print_meta: LF token = 227 '<0x0A>' llm_load_print_meta: EOT token = 107 '<end_of_turn>' ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes ggml_cuda_init: CUDA_USE_TENSOR_CORES: no ggml_cuda_init: found 1 CUDA devices: Device 0: Orin, compute capability 8.7, VMM: yes llm_load_tensors: ggml ctx size = 0.17 MiB llm_load_tensors: offloading 18 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 19/19 layers to GPU llm_load_tensors: CPU buffer size = 531.52 MiB llm_load_tensors: CUDA0 buffer size = 1594.93 MiB llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 36.00 MiB llama_new_context_with_model: KV self size = 36.00 MiB, K (f16): 18.00 MiB, V (f16): 18.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.98 MiB llama_new_context_with_model: CUDA0 compute buffer size = 504.25 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 8.01 MiB llama_new_context_with_model: graph nodes = 601 llama_new_context_with_model: graph splits = 2 time=2024-06-06T11:29:16.920-04:00 level=ERROR source=sched.go:344 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 1.00 - " [GIN] 2024/06/06 - 11:29:16 | 500 | 7m2s | 127.0.0.1 | POST "/api/chat" time=2024-06-06T11:29:22.037-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.117139389 time=2024-06-06T11:29:22.288-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.367497274 time=2024-06-06T11:29:22.537-04:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.616874999 ``` If it helps, this is running on a [Jetson AGX Orin with 32GB of memory](https://www.seeedstudio.com/AGX-Orin-32GB-H01-Kit-p-5569.html) ### OS Linux ### GPU Nvidia ### CPU Other: 8-core NVIDIA Arm® Cortex A78AE v8.2 64-bit CPU 2MB L2 + 4MB L3 ### Ollama version 0.1.41
GiteaMirror added the nvidiabug labels 2026-04-12 13:30:49 -05:00
GiteaMirror commented 2026-04-12 13:30:51 -05:00
Author
Owner
Copy Link

@dingsa0210 commented on GitHub (Jun 18, 2024):

I encountered the same issue after I updated ollama image. Remove and re-pull models would be a solution, that works for me.

<!-- gh-comment-id:2174924614 --> @dingsa0210 commented on GitHub (Jun 18, 2024): I encountered the same issue after I updated ollama image. Remove and re-pull models would be a solution, that works for me.
GiteaMirror commented 2026-04-12 13:30:52 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 18, 2024):

Can you try loading with mmap disabled to see if that changes the load time?

curl http://localhost:11434/api/generate -d '{
  "model": "gemma:2b",
  "prompt": "Why is the sky blue?",
  "stream": false, "options": {"use_mmap": false}
}'

Also, PR #4741 might be relevant.

<!-- gh-comment-id:2176962566 --> @dhiltgen commented on GitHub (Jun 18, 2024): Can you try loading with mmap disabled to see if that changes the load time? ``` curl http://localhost:11434/api/generate -d '{ "model": "gemma:2b", "prompt": "Why is the sky blue?", "stream": false, "options": {"use_mmap": false} }' ``` Also, PR #4741 might be relevant.
GiteaMirror commented 2026-04-12 13:30:52 -05:00
Author
Owner
Copy Link

@Vassar-HARPER-Project commented on GitHub (Jun 19, 2024):

I encountered the same issue after I updated ollama image. Remove and re-pull models would be a solution, that works for me.

Tried exactly this. Doesn't seem to work for me.

<!-- gh-comment-id:2178718489 --> @Vassar-HARPER-Project commented on GitHub (Jun 19, 2024): > I encountered the same issue after I updated ollama image. Remove and re-pull models would be a solution, that works for me. Tried exactly this. Doesn't seem to work for me.
GiteaMirror commented 2026-04-12 13:30:53 -05:00
Author
Owner
Copy Link

@Vassar-HARPER-Project commented on GitHub (Jun 19, 2024):

Can you try loading with mmap disabled to see if that changes the load time?

curl http://localhost:11434/api/generate -d '{
  "model": "gemma:2b",
  "prompt": "Why is the sky blue?",
  "stream": false, "options": {"use_mmap": false}
}'

Also, PR #4741 might be relevant.

Running this command I get the same result. First time I run, the CUDA error. Second time, the timed-out error. Alternating. Time before the error occurs seems to be the same as before (several minutes).

<!-- gh-comment-id:2178751799 --> @Vassar-HARPER-Project commented on GitHub (Jun 19, 2024): > Can you try loading with mmap disabled to see if that changes the load time? > > ``` > curl http://localhost:11434/api/generate -d '{ > "model": "gemma:2b", > "prompt": "Why is the sky blue?", > "stream": false, "options": {"use_mmap": false} > }' > ``` > > Also, PR #4741 might be relevant. Running this command I get the same result. First time I run, the CUDA error. Second time, the timed-out error. Alternating. Time before the error occurs seems to be the same as before (several minutes).
GiteaMirror commented 2026-04-12 13:30:54 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 19, 2024):

@Vassar-HARPER-Project it sounds like you will need #4741 to move forward without building from source.

<!-- gh-comment-id:2179116962 --> @dhiltgen commented on GitHub (Jun 19, 2024): @Vassar-HARPER-Project it sounds like you will need #4741 to move forward without building from source.
GiteaMirror commented 2026-04-12 13:30:55 -05:00
Author
Owner
Copy Link

@JorgeAlberto91MS commented on GitHub (Sep 2, 2024):

Same error after deleting v0.3.8 and installing v0.3.9.
Ollama running on a Docker container works fine, but not the one installed using "curl -fsSL https://ollama.com/install.sh | sh"

Related information:
Device: SeedStudio reComputer J4012
Card: Nvidia Jetson Orin NX 16GB
Software: Jetpack 6.0
OS: Ubuntu 22.04

<!-- gh-comment-id:2325099356 --> @JorgeAlberto91MS commented on GitHub (Sep 2, 2024): Same error after deleting v0.3.8 and installing v0.3.9. Ollama running on a Docker container works fine, but not the one installed using "curl -fsSL https://ollama.com/install.sh | sh" Related information: Device: SeedStudio reComputer J4012 Card: Nvidia Jetson Orin NX 16GB Software: Jetpack 6.0 OS: Ubuntu 22.04
GiteaMirror commented 2026-04-12 13:30:55 -05:00
Author
Owner
Copy Link

@chang-1 commented on GitHub (Sep 6, 2024):

Facing the same problem with Jetson Orin AGX 32GB from seed studio.

<!-- gh-comment-id:2334881090 --> @chang-1 commented on GitHub (Sep 6, 2024): Facing the same problem with Jetson Orin AGX 32GB from seed studio.
GiteaMirror commented 2026-04-12 13:30:55 -05:00
Author
Owner
Copy Link

@soulisalmed commented on GitHub (Sep 12, 2024):

Same error Error timed out waiting for llama runner to start - progress 0.00 - with ollama v0.3.10 installed with curl

Device : Nvidia Jetson AGX Orin 64GB Developper kit
Software : Jetpack 6.0
OS: Ubuntu 22.04

<!-- gh-comment-id:2345848591 --> @soulisalmed commented on GitHub (Sep 12, 2024): Same error `Error timed out waiting for llama runner to start - progress 0.00 -` with ollama v0.3.10 installed with curl Device : Nvidia Jetson AGX Orin 64GB Developper kit Software : Jetpack 6.0 OS: Ubuntu 22.04
GiteaMirror commented 2026-04-12 13:30:56 -05:00
Author
Owner
Copy Link

@cwbjyy commented on GitHub (Sep 19, 2024):

Same error with ollama v0.3.11

<!-- gh-comment-id:2360474161 --> @cwbjyy commented on GitHub (Sep 19, 2024): Same error with ollama v0.3.11
GiteaMirror commented 2026-04-12 13:30:56 -05:00
Author
Owner
Copy Link

@Spencer90 commented on GitHub (Sep 25, 2024):

Same error running ollama v0.3.12. The circle spins for about 2 minutes and prints the following.
Error: timed out waiting for llama runner to start - progress 0.00 -

Device : Nvidia Jetson AGX Orin 64GB Developer kit
Software : Jetpack 6.0
OS: Ubuntu 22.04

<!-- gh-comment-id:2375391242 --> @Spencer90 commented on GitHub (Sep 25, 2024): Same error running ollama v0.3.12. The circle spins for about 2 minutes and prints the following. `Error: timed out waiting for llama runner to start - progress 0.00 -` Device : Nvidia Jetson AGX Orin 64GB Developer kit Software : Jetpack 6.0 OS: Ubuntu 22.04
GiteaMirror commented 2026-04-12 13:30:56 -05:00
Author
Owner
Copy Link

@whk6688 commented on GitHub (Oct 28, 2024):

the same

<!-- gh-comment-id:2441295588 --> @whk6688 commented on GitHub (Oct 28, 2024): the same
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3075
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?