github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6425] optimize numa behavior for large models with GPU and CPU inference - numa_balancing on GPU causes excessively slow load times #4040

New Issue
Open
opened 2026-04-12 14:55:48 -05:00 by GiteaMirror · 14 comments
GiteaMirror commented 2026-04-12 14:55:48 -05:00
Owner
Copy Link

Originally created by @fabiounixpi on GitHub (Aug 19, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6425

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

My setup is a 4x A100 80GB, 2TB ram, dual intel cpu. Ubuntu server 22.04.
On a previous version of ollama, the model llama3.1:405b was loaded in a reasonable amount of seconds, with latest version this is not the case anymore.
After issuing the command
ollama run llama3.1:405b
it just remain with the rotating cursor.

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.3.6

Originally created by @fabiounixpi on GitHub (Aug 19, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6425 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? My setup is a 4x A100 80GB, 2TB ram, dual intel cpu. Ubuntu server 22.04. On a previous version of ollama, the model llama3.1:405b was loaded in a reasonable amount of seconds, with latest version this is not the case anymore. After issuing the command ollama run llama3.1:405b it just remain with the rotating cursor. ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.3.6
GiteaMirror added the linuxfeature requestperformance labels 2026-04-12 14:55:48 -05:00
GiteaMirror commented 2026-04-12 14:55:49 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 19, 2024):

Server logs may aid in debugging. If possible, add OLLAMA_DEBUG=1 to the server environment to display more information on the progress of the model load.

<!-- gh-comment-id:2297357074 --> @rick-github commented on GitHub (Aug 19, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging. If possible, add `OLLAMA_DEBUG=1` to the server environment to display more information on the progress of the model load.
GiteaMirror commented 2026-04-12 14:55:50 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 20, 2024):

Thank you Rick,
here you find output from journalctl -u ollama --no-pager

Aug 19 22:32:14 llmserver systemd[1]: Started Ollama Service.
Aug 19 22:32:14 llmserver ollama[115793]: 2024/08/19 22:32:14 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.888+02:00 level=INFO source=images.go:782 msg="total blobs: 33"
Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.889+02:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0"
Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.889+02:00 level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.6)"
Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.890+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama783568570/runners
Aug 19 22:32:18 llmserver ollama[115793]: time=2024-08-19T22:32:18.459+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11 rocm_v60102 cpu cpu_avx]"
Aug 19 22:32:18 llmserver ollama[115793]: time=2024-08-19T22:32:18.459+02:00 level=INFO source=gpu.go:204 msg="looking for compatible GPUs"
Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-def8bc44-2e1d-ca3e-7e5f-babd7af9e210 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB"
Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-26e351a0-5b4c-de02-6edf-f0ccafcb7ae4 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB"
Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-d1948f90-26e7-425b-ce6b-c8b787da4298 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB"
Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-277e8b76-f8bb-a6e9-2d83-f88137fa8e44 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB"
Aug 19 22:32:55 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:32:55 | 200 | 56.856µs | 127.0.0.1 | HEAD "/"
Aug 19 22:32:55 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:32:55 | 200 | 19.924093ms | 127.0.0.1 | POST "/api/show"
Aug 19 22:32:55 llmserver ollama[115793]: time=2024-08-19T22:32:55.987+02:00 level=INFO source=sched.go:726 msg="new model will fit in available VRAM, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 library=cuda parallel=4 required="240.1 GiB"
Aug 19 22:32:55 llmserver ollama[115793]: time=2024-08-19T22:32:55.989+02:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=127 layers.offload=127 layers.split=32,32,32,31 memory.available="[78.8 GiB 78.8 GiB 78.8 GiB 78.8 GiB]" memory.required.full="240.1 GiB" memory.required.partial="240.1 GiB" memory.required.kv="7.9 GiB" memory.required.allocations="[60.5 GiB 60.5 GiB 60.4 GiB 58.7 GiB]" memory.weights.total="220.5 GiB" memory.weights.repeating="218.9 GiB" memory.weights.nonrepeating="1.6 GiB" memory.graph.full="2.3 GiB" memory.graph.partial="2.3 GiB"
Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:393 msg="starting llama server" cmd="/tmp/ollama783568570/runners/cuda_v11/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 127 --numa distribute --parallel 4 --tensor-split 32,32,32,31 --port 41343"
Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=sched.go:445 msg="loaded runners" count=1
Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:593 msg="waiting for llama runner to start responding"
Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server error"
Aug 19 22:32:56 llmserver ollama[115972]: WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance
Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] build info | build=1 commit="1e6f655" tid="128993128079360" timestamp=1724099576
Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] system info | n_threads=112 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="128993128079360" timestamp=1724099576 total_threads=224
Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="223" port="41343" tid="128993128079360" timestamp=1724099576
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: loaded meta data with 29 key-value pairs and 1137 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 (version GGUF V3 (latest))
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 0: general.architecture str = llama
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 1: general.type str = model
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 405B Instruct
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 3: general.finetune str = Instruct
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 5: general.size_label str = 405B
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 6: general.license str = llama3.1
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 9: llama.block_count u32 = 126
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 10: llama.context_length u32 = 131072
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 11: llama.embedding_length u32 = 16384
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 12: llama.feed_forward_length u32 = 53248
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 13: llama.attention.head_count u32 = 128
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 16
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 17: general.file_type u32 = 2
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 18: llama.vocab_size u32 = 128256
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 21: tokenizer.ggml.pre str = smaug-bpe
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 27: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 28: general.quantization_version u32 = 2
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type f32: 253 tensors
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type q4_0: 883 tensors
Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type q6_K: 1 tensors
Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.251+02:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server loading model"
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_vocab: special tokens cache size = 256
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_vocab: token to piece cache size = 0.7999 MB
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: format = GGUF V3 (latest)
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: arch = llama
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: vocab type = BPE
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_vocab = 128256
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_merges = 280147
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: vocab_only = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ctx_train = 131072
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd = 16384
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_layer = 126
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_head = 128
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_head_kv = 16
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_rot = 128
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_swa = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_head_k = 128
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_head_v = 128
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_gqa = 8
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_k_gqa = 2048
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_v_gqa = 2048
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_norm_eps = 0.0e+00
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_norm_rms_eps = 1.0e-05
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_clamp_kqv = 0.0e+00
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_logit_scale = 0.0e+00
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ff = 53248
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_expert = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_expert_used = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: causal attn = 1
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: pooling type = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope type = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope scaling = linear
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: freq_base_train = 500000.0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: freq_scale_train = 1
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ctx_orig_yarn = 131072
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope_finetuned = unknown
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_conv = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_inner = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_state = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_dt_rank = 0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model type = ?B
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model ftype = Q4_0
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model params = 410.08 B
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model size = 215.35 GiB (4.51 BPW)
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: general.name = Meta Llama 3.1 405B Instruct
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: LF token = 128 'Ä'
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: max token length = 256
Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: found 4 CUDA devices:
Aug 19 22:32:56 llmserver ollama[115793]: Device 0: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes
Aug 19 22:32:56 llmserver ollama[115793]: Device 1: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes
Aug 19 22:32:56 llmserver ollama[115793]: Device 2: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes
Aug 19 22:32:56 llmserver ollama[115793]: Device 3: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes
Aug 19 22:32:56 llmserver ollama[115793]: llm_load_tensors: ggml ctx size = 2.66 MiB
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloading 126 repeating layers to GPU
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloading non-repeating layers to GPU
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloaded 127/127 layers to GPU
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CPU buffer size = 1127.25 MiB
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA0 buffer size = 55300.00 MiB
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA1 buffer size = 55300.00 MiB
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA2 buffer size = 55300.00 MiB
Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA3 buffer size = 53487.72 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_ctx = 8192
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_batch = 512
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_ubatch = 512
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: flash_attn = 0
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: freq_base = 500000.0
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: freq_scale = 1
Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA0 KV buffer size = 2048.00 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA1 KV buffer size = 2048.00 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA2 KV buffer size = 2048.00 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA3 KV buffer size = 1920.00 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: KV self size = 8064.00 MiB, K (f16): 4032.00 MiB, V (f16): 4032.00 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA_Host output buffer size = 2.21 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA0 compute buffer size = 2368.01 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA1 compute buffer size = 2368.01 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA2 compute buffer size = 2368.01 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA3 compute buffer size = 2368.02 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA_Host compute buffer size = 96.02 MiB
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: graph nodes = 4038
Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: graph splits = 5
Aug 19 22:54:33 llmserver ollama[115972]: INFO [main] model loaded | tid="128993128079360" timestamp=1724100873
Aug 19 22:54:33 llmserver ollama[115793]: time=2024-08-19T22:54:33.996+02:00 level=INFO source=server.go:632 msg="llama runner started in 1298.00 seconds"
Aug 19 22:54:33 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:54:33 | 200 | 21m38s | 127.0.0.1 | POST "/api/chat"

<!-- gh-comment-id:2298045123 --> @fabiounixpi commented on GitHub (Aug 20, 2024): Thank you Rick, here you find output from journalctl -u ollama --no-pager Aug 19 22:32:14 llmserver systemd[1]: Started Ollama Service. Aug 19 22:32:14 llmserver ollama[115793]: 2024/08/19 22:32:14 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.888+02:00 level=INFO source=images.go:782 msg="total blobs: 33" Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.889+02:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0" Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.889+02:00 level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.6)" Aug 19 22:32:14 llmserver ollama[115793]: time=2024-08-19T22:32:14.890+02:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama783568570/runners Aug 19 22:32:18 llmserver ollama[115793]: time=2024-08-19T22:32:18.459+02:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11 rocm_v60102 cpu cpu_avx]" Aug 19 22:32:18 llmserver ollama[115793]: time=2024-08-19T22:32:18.459+02:00 level=INFO source=gpu.go:204 msg="looking for compatible GPUs" Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-def8bc44-2e1d-ca3e-7e5f-babd7af9e210 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB" Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-26e351a0-5b4c-de02-6edf-f0ccafcb7ae4 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB" Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-d1948f90-26e7-425b-ce6b-c8b787da4298 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB" Aug 19 22:32:19 llmserver ollama[115793]: time=2024-08-19T22:32:19.650+02:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-277e8b76-f8bb-a6e9-2d83-f88137fa8e44 library=cuda compute=8.0 driver=12.5 name="NVIDIA A100 80GB PCIe" total="79.3 GiB" available="78.8 GiB" Aug 19 22:32:55 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:32:55 | 200 | 56.856µs | 127.0.0.1 | HEAD "/" Aug 19 22:32:55 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:32:55 | 200 | 19.924093ms | 127.0.0.1 | POST "/api/show" Aug 19 22:32:55 llmserver ollama[115793]: time=2024-08-19T22:32:55.987+02:00 level=INFO source=sched.go:726 msg="new model will fit in available VRAM, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 library=cuda parallel=4 required="240.1 GiB" Aug 19 22:32:55 llmserver ollama[115793]: time=2024-08-19T22:32:55.989+02:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=127 layers.offload=127 layers.split=32,32,32,31 memory.available="[78.8 GiB 78.8 GiB 78.8 GiB 78.8 GiB]" memory.required.full="240.1 GiB" memory.required.partial="240.1 GiB" memory.required.kv="7.9 GiB" memory.required.allocations="[60.5 GiB 60.5 GiB 60.4 GiB 58.7 GiB]" memory.weights.total="220.5 GiB" memory.weights.repeating="218.9 GiB" memory.weights.nonrepeating="1.6 GiB" memory.graph.full="2.3 GiB" memory.graph.partial="2.3 GiB" Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:393 msg="starting llama server" cmd="/tmp/ollama783568570/runners/cuda_v11/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 127 --numa distribute --parallel 4 --tensor-split 32,32,32,31 --port 41343" Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=sched.go:445 msg="loaded runners" count=1 Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:593 msg="waiting for llama runner to start responding" Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.000+02:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server error" Aug 19 22:32:56 llmserver ollama[115972]: WARNING: /proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] build info | build=1 commit="1e6f655" tid="128993128079360" timestamp=1724099576 Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] system info | n_threads=112 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="128993128079360" timestamp=1724099576 total_threads=224 Aug 19 22:32:56 llmserver ollama[115972]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="223" port="41343" tid="128993128079360" timestamp=1724099576 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: loaded meta data with 29 key-value pairs and 1137 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-6fd659ca39733750d63e9dc442664c1b306ca14dab3091802b0076c91176cc68 (version GGUF V3 (latest)) Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 0: general.architecture str = llama Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 1: general.type str = model Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 405B Instruct Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 3: general.finetune str = Instruct Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 5: general.size_label str = 405B Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 6: general.license str = llama3.1 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 9: llama.block_count u32 = 126 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 10: llama.context_length u32 = 131072 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 11: llama.embedding_length u32 = 16384 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 12: llama.feed_forward_length u32 = 53248 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 13: llama.attention.head_count u32 = 128 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 16 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 17: general.file_type u32 = 2 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 21: tokenizer.ggml.pre str = smaug-bpe Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 27: tokenizer.chat_template str = {% set loop_messages = messages %}{% ... Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - kv 28: general.quantization_version u32 = 2 Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type f32: 253 tensors Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type q4_0: 883 tensors Aug 19 22:32:56 llmserver ollama[115793]: llama_model_loader: - type q6_K: 1 tensors Aug 19 22:32:56 llmserver ollama[115793]: time=2024-08-19T22:32:56.251+02:00 level=INFO source=server.go:627 msg="waiting for server to become available" status="llm server loading model" Aug 19 22:32:56 llmserver ollama[115793]: llm_load_vocab: special tokens cache size = 256 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_vocab: token to piece cache size = 0.7999 MB Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: format = GGUF V3 (latest) Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: arch = llama Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: vocab type = BPE Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_vocab = 128256 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_merges = 280147 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: vocab_only = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ctx_train = 131072 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd = 16384 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_layer = 126 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_head = 128 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_head_kv = 16 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_rot = 128 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_swa = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_head_k = 128 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_head_v = 128 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_gqa = 8 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_k_gqa = 2048 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_embd_v_gqa = 2048 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_norm_eps = 0.0e+00 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_norm_rms_eps = 1.0e-05 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: f_logit_scale = 0.0e+00 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ff = 53248 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_expert = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_expert_used = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: causal attn = 1 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: pooling type = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope type = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope scaling = linear Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: freq_base_train = 500000.0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: freq_scale_train = 1 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: n_ctx_orig_yarn = 131072 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: rope_finetuned = unknown Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_conv = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_inner = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_d_state = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: ssm_dt_rank = 0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model type = ?B Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model ftype = Q4_0 Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model params = 410.08 B Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: model size = 215.35 GiB (4.51 BPW) Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: general.name = Meta Llama 3.1 405B Instruct Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>' Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>' Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: LF token = 128 'Ä' Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>' Aug 19 22:32:56 llmserver ollama[115793]: llm_load_print_meta: max token length = 256 Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Aug 19 22:32:56 llmserver ollama[115793]: ggml_cuda_init: found 4 CUDA devices: Aug 19 22:32:56 llmserver ollama[115793]: Device 0: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes Aug 19 22:32:56 llmserver ollama[115793]: Device 1: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes Aug 19 22:32:56 llmserver ollama[115793]: Device 2: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes Aug 19 22:32:56 llmserver ollama[115793]: Device 3: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes Aug 19 22:32:56 llmserver ollama[115793]: llm_load_tensors: ggml ctx size = 2.66 MiB Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloading 126 repeating layers to GPU Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloading non-repeating layers to GPU Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: offloaded 127/127 layers to GPU Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CPU buffer size = 1127.25 MiB Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA0 buffer size = 55300.00 MiB Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA1 buffer size = 55300.00 MiB Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA2 buffer size = 55300.00 MiB Aug 19 22:32:57 llmserver ollama[115793]: llm_load_tensors: CUDA3 buffer size = 53487.72 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_ctx = 8192 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_batch = 512 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: n_ubatch = 512 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: flash_attn = 0 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: freq_base = 500000.0 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: freq_scale = 1 Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA0 KV buffer size = 2048.00 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA1 KV buffer size = 2048.00 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA2 KV buffer size = 2048.00 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_kv_cache_init: CUDA3 KV buffer size = 1920.00 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: KV self size = 8064.00 MiB, K (f16): 4032.00 MiB, V (f16): 4032.00 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA_Host output buffer size = 2.21 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4) Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA0 compute buffer size = 2368.01 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA1 compute buffer size = 2368.01 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA2 compute buffer size = 2368.01 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA3 compute buffer size = 2368.02 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: CUDA_Host compute buffer size = 96.02 MiB Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: graph nodes = 4038 Aug 19 22:54:31 llmserver ollama[115793]: llama_new_context_with_model: graph splits = 5 Aug 19 22:54:33 llmserver ollama[115972]: INFO [main] model loaded | tid="128993128079360" timestamp=1724100873 Aug 19 22:54:33 llmserver ollama[115793]: time=2024-08-19T22:54:33.996+02:00 level=INFO source=server.go:632 msg="llama runner started in 1298.00 seconds" Aug 19 22:54:33 llmserver ollama[115793]: [GIN] 2024/08/19 - 22:54:33 | 200 | 21m38s | 127.0.0.1 | POST "/api/chat"
GiteaMirror commented 2026-04-12 14:55:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 20, 2024):

Aug 19 22:54:33 llmserver ollama[115793]: time=2024-08-19T22:54:33.996+02:00 level=INFO source=server.go:632 msg="llama runner started in 1298.00 seconds"

That's a long load time. Where is the model stored, on local disk or on network disk? If the model is unloaded and reloaded, does it still take a long time to load? Do you which version of ollama was the first to be this slow?

<!-- gh-comment-id:2298124120 --> @rick-github commented on GitHub (Aug 20, 2024): ``` Aug 19 22:54:33 llmserver ollama[115793]: time=2024-08-19T22:54:33.996+02:00 level=INFO source=server.go:632 msg="llama runner started in 1298.00 seconds" ``` That's a long load time. Where is the model stored, on local disk or on network disk? If the model is unloaded and reloaded, does it still take a long time to load? Do you which version of ollama was the first to be this slow?
GiteaMirror commented 2026-04-12 14:55:51 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 20, 2024):

The model is stored on nvme local storage, the time you read is after unloading the model stopping and restarting ollama.service. I've just upgraded to 0.3.6 and suddenly noticed this behavior, if you need I can try to install an older version one by one to see at what point it becomes faster again

<!-- gh-comment-id:2298134901 --> @fabiounixpi commented on GitHub (Aug 20, 2024): The model is stored on nvme local storage, the time you read is after unloading the model stopping and restarting ollama.service. I've just upgraded to 0.3.6 and suddenly noticed this behavior, if you need I can try to install an older version one by one to see at what point it becomes faster again
GiteaMirror commented 2026-04-12 14:55:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 20, 2024):

If you could pinpoint the version at which it becomes slow that would be very helpful.

<!-- gh-comment-id:2298324112 --> @rick-github commented on GitHub (Aug 20, 2024): If you could pinpoint the version at which it becomes slow that would be very helpful.
GiteaMirror commented 2026-04-12 14:55:52 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Aug 23, 2024):

I just tried this on an 8xA100 80GB machine, and it took about 45s to load the 405b model. This was using Ubuntu 22.04 w/ a fresh install of Ollama 0.3.6.

$ ollama ps
NAME         	ID          	SIZE  	PROCESSOR	UNTIL
llama3.1:405b	da852fffc86f	269 GB	100% GPU 	4 minutes from now

@fabiounixpi can you cat /proc/cpuinfo and see if there is a line like:

physical id	: 0

The hypothesis is that you're using a multi-socket machine which somehow isn't working correctly with NUMA.

cc @dhiltgen

<!-- gh-comment-id:2307914907 --> @pdevine commented on GitHub (Aug 23, 2024): I just tried this on an 8xA100 80GB machine, and it took about 45s to load the 405b model. This was using Ubuntu 22.04 w/ a fresh install of Ollama 0.3.6. ``` $ ollama ps NAME ID SIZE PROCESSOR UNTIL llama3.1:405b da852fffc86f 269 GB 100% GPU 4 minutes from now ``` @fabiounixpi can you cat `/proc/cpuinfo` and see if there is a line like: ``` physical id : 0 ``` The hypothesis is that you're using a multi-socket machine which somehow isn't working correctly with NUMA. cc @dhiltgen
GiteaMirror commented 2026-04-12 14:55:53 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 24, 2024):

yes i read

physical id : 0
and also
physical id : 1

is a dual socket machine, 56 cores per socket, hyperthreading enabled.
but before 0.3.6 the time to load was similar to yours, 46s

<!-- gh-comment-id:2308264625 --> @fabiounixpi commented on GitHub (Aug 24, 2024): yes i read physical id : 0 and also physical id : 1 is a dual socket machine, 56 cores per socket, hyperthreading enabled. but before 0.3.6 the time to load was similar to yours, 46s
GiteaMirror commented 2026-04-12 14:55:53 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Aug 24, 2024):

@fabiounixpi there may be a simple fix here w/ #6484 . We need to test it out on our multi-socket machine.

<!-- gh-comment-id:2308473347 --> @pdevine commented on GitHub (Aug 24, 2024): @fabiounixpi there _may_ be a simple fix here w/ #6484 . We need to test it out on our multi-socket machine.
GiteaMirror commented 2026-04-12 14:55:54 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 24, 2024):

Ok, for version 0.3.4 confirm load time is ok, 46s. What can help to understand what is going wrong with my numa settings? I've not touched it and my os is an ubuntu 22.04.4. The only difference mine is not a fresh install of ollama 0.3.6 but result of a series of updates,

<!-- gh-comment-id:2308539492 --> @fabiounixpi commented on GitHub (Aug 24, 2024): Ok, for version 0.3.4 confirm load time is ok, 46s. What can help to understand what is going wrong with my numa settings? I've not touched it and my os is an ubuntu 22.04.4. The only difference mine is not a fresh install of ollama 0.3.6 but result of a series of updates,
GiteaMirror commented 2026-04-12 14:55:55 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Aug 25, 2024):

@fabiounixpi There was a change in 0.3.5 which sped up inference for NUMA based (i.e. multi socket) CPUs, but unfortunately it is making multi-GPU setups slower. We think it is now fixed in 0.3.7 which is in pre-release (although the PR should go in to rc7 I think which hasn't been released yet).

<!-- gh-comment-id:2308647690 --> @pdevine commented on GitHub (Aug 25, 2024): @fabiounixpi There was a change in 0.3.5 which sped up inference for NUMA based (i.e. multi socket) CPUs, but unfortunately it is making multi-GPU setups slower. We think it is now fixed in 0.3.7 which is in pre-release (although the PR should go in to rc7 I think which hasn't been released yet).
GiteaMirror commented 2026-04-12 14:55:56 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 28, 2024):

Version 0.3.8 is still slow, what infos may help diagnosing the root of the problem?

<!-- gh-comment-id:2315183191 --> @fabiounixpi commented on GitHub (Aug 28, 2024): Version 0.3.8 is still slow, what infos may help diagnosing the root of the problem?
GiteaMirror commented 2026-04-12 14:55:57 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 28, 2024):

It sounds like the numa flag was not the root cause then. @fabiounixpi can you share updated server logs, ideally with OLLAMA_DEBUG=1 set so we can see a bit more diagnostic information and do a full model load so we can see timestamps from start to finish. If you could share some more details about your system that might also help find the root cause. (what kind of storage do you have, what performance metrics do you observe while it's loading (CPU load, I/O load, etc.))

<!-- gh-comment-id:2315763953 --> @dhiltgen commented on GitHub (Aug 28, 2024): It sounds like the numa flag was not the root cause then. @fabiounixpi can you share updated server logs, ideally with OLLAMA_DEBUG=1 set so we can see a bit more diagnostic information and do a full model load so we can see timestamps from start to finish. If you could share some more details about your system that might also help find the root cause. (what kind of storage do you have, what performance metrics do you observe while it's loading (CPU load, I/O load, etc.))
GiteaMirror commented 2026-04-12 14:55:57 -05:00
Author
Owner
Copy Link

@fabiounixpi commented on GitHub (Aug 28, 2024):

ollama-038-debug-true-numa-balancing-false.txt
may be i've managed to solve, with
echo 0 > /proc/sys/kernel/numa_balancing
in attach the requested log
PS: looking at flags I do not uderstand why some AVX512_ flags are false, even if the cores are actually Sapphire Rapids. May be it is because we are switching to GPUs?

<!-- gh-comment-id:2316002395 --> @fabiounixpi commented on GitHub (Aug 28, 2024): [ollama-038-debug-true-numa-balancing-false.txt](https://github.com/user-attachments/files/16787522/ollama-038-debug-true-numa-balancing-false.txt) may be i've managed to solve, with echo 0 > /proc/sys/kernel/numa_balancing in attach the requested log PS: looking at flags I do not uderstand why some AVX512_ flags are false, even if the cores are actually Sapphire Rapids. May be it is because we are switching to GPUs?
GiteaMirror commented 2026-04-12 14:55:58 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Sep 3, 2024):

That's great to hear disabling numa balancing solved your slow model load problem.

My suspicion is this may be specific to loading large models onto the GPU on a numa system, and users with numa systems intending to use CPU inference may benefit from numa_balancing being enabled.

We don't currently compile the subprocess C++ code with the AVX512 vector flags enabled, as we haven't seen a significant performance improvement. We're trying to balance broad hardware support without creating too many permutations. We're working to improve the ability for users to build their own customized versions locally with exactly the compiler flags they want.

I'll keep this open as a feature enhancement request to see if we can tune this a bit more to either set the default behavior automatically, or at least log a warning if we detect a scenario that is likely to yield stalled model loads.

<!-- gh-comment-id:2327116916 --> @dhiltgen commented on GitHub (Sep 3, 2024): That's great to hear disabling numa balancing solved your slow model load problem. My suspicion is this may be specific to loading large models onto the GPU on a numa system, and users with numa systems intending to use CPU inference may benefit from numa_balancing being enabled. We don't currently compile the subprocess C++ code with the AVX512 vector flags enabled, as we haven't seen a significant performance improvement. We're trying to balance broad hardware support without creating too many permutations. We're working to improve the ability for users to build their own customized versions locally with exactly the compiler flags they want. I'll keep this open as a feature enhancement request to see if we can tune this a bit more to either set the default behavior automatically, or at least log a warning if we detect a scenario that is likely to yield stalled model loads.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request linux performance
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4040
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?