[GH-ISSUE #6277] Ollama Latest (0.3.4) Will not run models #3933

Closed
opened 2026-04-12 14:48:37 -05:00 by GiteaMirror · 9 comments
Owner

Originally created by @awptechnologies on GitHub (Aug 9, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6277

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

I have tools that automatically update my containers. I use latest with ollama. after latest update to image i cant run any models. I can pull models for example llama3.1 but when i go to run they never start. I can see my memory on gpu jump to 4000mb and stop. Ushally when running these models it is closer to 6000mb. After downgrading to 0.3.3 everything works perfect again.

OS

Docker

GPU

Nvidia

CPU

Intel

Ollama version

latest 0.3.4

Originally created by @awptechnologies on GitHub (Aug 9, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6277 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? I have tools that automatically update my containers. I use latest with ollama. after latest update to image i cant run any models. I can pull models for example llama3.1 but when i go to run they never start. I can see my memory on gpu jump to 4000mb and stop. Ushally when running these models it is closer to 6000mb. After downgrading to 0.3.3 everything works perfect again. ### OS Docker ### GPU Nvidia ### CPU Intel ### Ollama version latest 0.3.4
GiteaMirror added the needs more infobug labels 2026-04-12 14:48:38 -05:00
Author
Owner

@rick-github commented on GitHub (Aug 9, 2024):

Server logs will help in debugging.

<!-- gh-comment-id:2277284614 --> @rick-github commented on GitHub (Aug 9, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will help in debugging.
Author
Owner

@awptechnologies commented on GitHub (Aug 10, 2024):

I had deleted the latest container and started up the 0.3.3 when i get some down time i will switch back to latest.

<!-- gh-comment-id:2282217492 --> @awptechnologies commented on GitHub (Aug 10, 2024): I had deleted the latest container and started up the 0.3.3 when i get some down time i will switch back to latest.
Author
Owner

@awptechnologies commented on GitHub (Aug 13, 2024):

Updated to new latest and same thing. GPU Memory stuck at 4264MIB almost like model isn't fully loaded. If i go back to 0.3.3 all is well. Here are logs.

`WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance

INFO [main] build info | build=1 commit="1e6f655" tid="139849002516480" timestamp=1723530846

INFO [main] system info | n_threads=14 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139849002516480" timestamp=1723530846 total_threads=14

INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="13" port="35375" tid="139849002516480" timestamp=1723530846

time=2024-08-13T02:34:06.516-04:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server loading model"

llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-8eeb52dfb3bb9aefdf9d1ef24b3bdbcfbe82238798c4b918278320b6fcef18fe (version GGUF V3 (latest))

llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.

llama_model_loader: - kv 0: general.architecture str = llama

llama_model_loader: - kv 1: general.type str = model

llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct

llama_model_loader: - kv 3: general.finetune str = Instruct

llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1

llama_model_loader: - kv 5: general.size_label str = 8B

llama_model_loader: - kv 6: general.license str = llama3.1

llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...

llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...

llama_model_loader: - kv 9: llama.block_count u32 = 32

llama_model_loader: - kv 10: llama.context_length u32 = 131072

llama_model_loader: - kv 11: llama.embedding_length u32 = 4096

llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336

llama_model_loader: - kv 13: llama.attention.head_count u32 = 32

llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8

llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000

llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010

llama_model_loader: - kv 17: general.file_type u32 = 2

llama_model_loader: - kv 18: llama.vocab_size u32 = 128256

llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128

llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2

llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe

llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...

llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...

llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...

llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000

llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009

llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...

llama_model_loader: - kv 28: general.quantization_version u32 = 2

llama_model_loader: - type f32: 66 tensors

llama_model_loader: - type q4_0: 225 tensors

llama_model_loader: - type q6_K: 1 tensors

llm_load_vocab: special tokens cache size = 256

llm_load_vocab: token to piece cache size = 0.7999 MB

llm_load_print_meta: format = GGUF V3 (latest)

llm_load_print_meta: arch = llama

llm_load_print_meta: vocab type = BPE

llm_load_print_meta: n_vocab = 128256

llm_load_print_meta: n_merges = 280147

llm_load_print_meta: vocab_only = 0

llm_load_print_meta: n_ctx_train = 131072

llm_load_print_meta: n_embd = 4096

llm_load_print_meta: n_layer = 32

llm_load_print_meta: n_head = 32

llm_load_print_meta: n_head_kv = 8

llm_load_print_meta: n_rot = 128

llm_load_print_meta: n_swa = 0

llm_load_print_meta: n_embd_head_k = 128

llm_load_print_meta: n_embd_head_v = 128

llm_load_print_meta: n_gqa = 4

llm_load_print_meta: n_embd_k_gqa = 1024

llm_load_print_meta: n_embd_v_gqa = 1024

llm_load_print_meta: f_norm_eps = 0.0e+00

llm_load_print_meta: f_norm_rms_eps = 1.0e-05

llm_load_print_meta: f_clamp_kqv = 0.0e+00

llm_load_print_meta: f_max_alibi_bias = 0.0e+00

llm_load_print_meta: f_logit_scale = 0.0e+00

llm_load_print_meta: n_ff = 14336

llm_load_print_meta: n_expert = 0

llm_load_print_meta: n_expert_used = 0

llm_load_print_meta: causal attn = 1

llm_load_print_meta: pooling type = 0

llm_load_print_meta: rope type = 0

llm_load_print_meta: rope scaling = linear

llm_load_print_meta: freq_base_train = 500000.0

llm_load_print_meta: freq_scale_train = 1

llm_load_print_meta: n_ctx_orig_yarn = 131072

llm_load_print_meta: rope_finetuned = unknown

llm_load_print_meta: ssm_d_conv = 0

llm_load_print_meta: ssm_d_inner = 0

llm_load_print_meta: ssm_d_state = 0

llm_load_print_meta: ssm_dt_rank = 0

llm_load_print_meta: model type = 8B

llm_load_print_meta: model ftype = Q4_0

llm_load_print_meta: model params = 8.03 B

llm_load_print_meta: model size = 4.33 GiB (4.64 BPW)

llm_load_print_meta: general.name = Meta Llama 3.1 8B Instruct

llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'

llm_load_print_meta: EOS token = 128009 '<|eot_id|>'

llm_load_print_meta: LF token = 128 'Ä'

llm_load_print_meta: EOT token = 128009 '<|eot_id|>'

llm_load_print_meta: max token length = 256

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no

ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no

ggml_cuda_init: found 1 CUDA devices:

Device 0: NVIDIA GeForce GTX 1070, compute capability 6.1, VMM: yes

llm_load_tensors: ggml ctx size = 0.27 MiB

llm_load_tensors: offloading 32 repeating layers to GPU

llm_load_tensors: offloading non-repeating layers to GPU

llm_load_tensors: offloaded 33/33 layers to GPU

llm_load_tensors: CPU buffer size = 281.81 MiB

llm_load_tensors: CUDA0 buffer size = 4156.00 MiB`

<!-- gh-comment-id:2285450333 --> @awptechnologies commented on GitHub (Aug 13, 2024): Updated to new latest and same thing. GPU Memory stuck at 4264MIB almost like model isn't fully loaded. If i go back to 0.3.3 all is well. Here are logs. `WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance INFO [main] build info | build=1 commit="1e6f655" tid="139849002516480" timestamp=1723530846 INFO [main] system info | n_threads=14 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139849002516480" timestamp=1723530846 total_threads=14 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="13" port="35375" tid="139849002516480" timestamp=1723530846 time=2024-08-13T02:34:06.516-04:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-8eeb52dfb3bb9aefdf9d1ef24b3bdbcfbe82238798c4b918278320b6fcef18fe (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 8B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 32 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 4096 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 2 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_0: 225 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.7999 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 8.03 B llm_load_print_meta: model size = 4.33 GiB (4.64 BPW) llm_load_print_meta: general.name = Meta Llama 3.1 8B Instruct llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>' llm_load_print_meta: EOS token = 128009 '<|eot_id|>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce GTX 1070, compute capability 6.1, VMM: yes llm_load_tensors: ggml ctx size = 0.27 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 281.81 MiB llm_load_tensors: CUDA0 buffer size = 4156.00 MiB`
Author
Owner

@awptechnologies commented on GitHub (Aug 13, 2024):

with 0.3.3 my memory usage on gpu jumps to 5962MIB with the same model

<!-- gh-comment-id:2285482966 --> @awptechnologies commented on GitHub (Aug 13, 2024): with 0.3.3 my memory usage on gpu jumps to 5962MIB with the same model
Author
Owner

@rick-github commented on GitHub (Aug 13, 2024):

Earlier parts of the log show runner selection and memory calculations, if you can add those it might be helpful.

<!-- gh-comment-id:2287375448 --> @rick-github commented on GitHub (Aug 13, 2024): Earlier parts of the log show runner selection and memory calculations, if you can add those it might be helpful.
Author
Owner

@awptechnologies commented on GitHub (Aug 14, 2024):

Can you explain a little further? I Copied all of the logs the container had.

<!-- gh-comment-id:2287532792 --> @awptechnologies commented on GitHub (Aug 14, 2024): Can you explain a little further? I Copied all of the logs the container had.
Author
Owner

@rick-github commented on GitHub (Aug 14, 2024):

When ollama starts up, the first thing it prints is its config:

2024/08/13 22:17:18 routes.go:1108: INFO server config env="map[CUDA_VISIBLE_DEVICES: 
GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false
OLLAMA_KEEP_ALIVE:1h40m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:512
OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1
http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*]
OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR:/tmp/ollama ROCR_VISIBLE_DEVICES:]"

That's missing from your log so it looks incomplete. It could be that you are not capturing all of the log output. Try docker logs ollama > /tmp/ollama.log 2>&1, that will capture both stdout and stderr to /tmp/ollama.log. Substitute ollama with the name of the container.

<!-- gh-comment-id:2287564546 --> @rick-github commented on GitHub (Aug 14, 2024): When ollama starts up, the first thing it prints is its config: ``` 2024/08/13 22:17:18 routes.go:1108: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:1h40m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:4 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR:/tmp/ollama ROCR_VISIBLE_DEVICES:]" ``` That's missing from your log so it looks incomplete. It could be that you are not capturing all of the log output. Try `docker logs ollama > /tmp/ollama.log 2>&1`, that will capture both stdout and stderr to /tmp/ollama.log. Substitute `ollama` with the name of the container.
Author
Owner

@dhiltgen commented on GitHub (Sep 5, 2024):

@awptechnologies can you clarify "they never start"? Does the load timeout after 5minutes, and if so, is it stalled at 0% or 100% loaded?

We've seen other users reporting that the recent addition of numa support has caused some performance problems like you describe, so you might want to try the following to see if it clears up the problem

echo 0 > /proc/sys/kernel/numa_balancing

We've also disabled numa in newer builds for the GPU runners (only enabled for CPU runners) so upgrading to the latest version may also resolve it.

<!-- gh-comment-id:2332849462 --> @dhiltgen commented on GitHub (Sep 5, 2024): @awptechnologies can you clarify "they never start"? Does the load timeout after 5minutes, and if so, is it stalled at 0% or 100% loaded? We've seen other users reporting that the recent addition of numa support has caused some performance problems like you describe, so you might want to try the following to see if it clears up the problem ``` echo 0 > /proc/sys/kernel/numa_balancing ``` We've also disabled numa in newer builds for the GPU runners (only enabled for CPU runners) so upgrading to the latest version may also resolve it.
Author
Owner

@awptechnologies commented on GitHub (Sep 9, 2024):

I have switched back to the latest release and all is well now.

<!-- gh-comment-id:2337142035 --> @awptechnologies commented on GitHub (Sep 9, 2024): I have switched back to the latest release and all is well now.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3933