github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #6479] v0.3.7-rc5 no longer uses multiple GPUs for a single model #50589

New Issue
Closed
opened 2026-04-28 16:27:26 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-28 16:27:26 -05:00
Owner
Copy Link

Originally created by @Maltz42 on GitHub (Aug 23, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6479

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

Moving from 0.3.6 to 0.3.7-rc5, Ollama no longer uses both GPUs for a single model when the model will not fit on one card. If I load two models, though, it will use the second card to load the second model. Output of "ollama ps" and "nvidia-smi" below.

ollama ps:
NAME            ID              SIZE    PROCESSOR       UNTIL
mistral-large:latest    0ca7dfa0bf06    71 GB   29%/71% CPU/GPU 4 minutes from now

nvidia-smi:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               On  | 00000000:01:00.0 Off |                  Off |
| 37%   67C    P2             176W / 300W |  47681MiB / 49140MiB |     37%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               On  | 00000000:E1:00.0 Off |                  Off |
| 30%   49C    P8              22W / 300W |      4MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    262888      C   ...unners/cuda_v11/ollama_llama_server    47674MiB |
+---------------------------------------------------------------------------------------+

And here's with two models loaded (but inactive):

NAME                    ID              SIZE    PROCESSOR       UNTIL
gemma2:27b              53261bc9c192    20 GB   100% GPU        4 minutes from now
mistral-large:latest    0ca7dfa0bf06    71 GB   29%/71% CPU/GPU 2 minutes from now

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               On  | 00000000:01:00.0 Off |                  Off |
| 30%   51C    P8              25W / 300W |  47681MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               On  | 00000000:E1:00.0 Off |                  Off |
| 37%   57C    P8              25W / 300W |  19053MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    262888      C   ...unners/cuda_v11/ollama_llama_server    47674MiB |
|    1   N/A  N/A    278516      C   ...unners/cuda_v11/ollama_llama_server    19046MiB |
+---------------------------------------------------------------------------------------+

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.3.7-rc5

Originally created by @Maltz42 on GitHub (Aug 23, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6479 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? Moving from 0.3.6 to 0.3.7-rc5, Ollama no longer uses both GPUs for a single model when the model will not fit on one card. If I load two models, though, it will use the second card to load the second model. Output of "ollama ps" and "nvidia-smi" below. ``` ollama ps: NAME ID SIZE PROCESSOR UNTIL mistral-large:latest 0ca7dfa0bf06 71 GB 29%/71% CPU/GPU 4 minutes from now nvidia-smi: +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA RTX A6000 On | 00000000:01:00.0 Off | Off | | 37% 67C P2 176W / 300W | 47681MiB / 49140MiB | 37% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA RTX A6000 On | 00000000:E1:00.0 Off | Off | | 30% 49C P8 22W / 300W | 4MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 262888 C ...unners/cuda_v11/ollama_llama_server 47674MiB | +---------------------------------------------------------------------------------------+ ``` And here's with two models loaded (but inactive): ``` NAME ID SIZE PROCESSOR UNTIL gemma2:27b 53261bc9c192 20 GB 100% GPU 4 minutes from now mistral-large:latest 0ca7dfa0bf06 71 GB 29%/71% CPU/GPU 2 minutes from now +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA RTX A6000 On | 00000000:01:00.0 Off | Off | | 30% 51C P8 25W / 300W | 47681MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA RTX A6000 On | 00000000:E1:00.0 Off | Off | | 37% 57C P8 25W / 300W | 19053MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 262888 C ...unners/cuda_v11/ollama_llama_server 47674MiB | | 1 N/A N/A 278516 C ...unners/cuda_v11/ollama_llama_server 19046MiB | +---------------------------------------------------------------------------------------+ ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.3.7-rc5
GiteaMirror added the bug label 2026-04-28 16:27:26 -05:00
GiteaMirror commented 2026-04-28 16:27:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 23, 2024):

Server logs may help in debugging.

<!-- gh-comment-id:2307451866 --> @rick-github commented on GitHub (Aug 23, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may help in debugging.
GiteaMirror commented 2026-04-28 16:27:29 -05:00
Author
Owner
Copy Link

@Maltz42 commented on GitHub (Aug 23, 2024):

[GIN] 2024/08/23 - 17:02:19 | 200 |       23.69µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/08/23 - 17:02:19 | 200 |    3.728994ms |       127.0.0.1 | POST     "/api/show"
time=2024-08-23T17:02:20.155Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 l



















                                                                                                                                          layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB"
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama3372309948/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --parallel 1 --port 37647"
time=2024-08-23T17:02:20.157Z level=INFO source=sched.go:445 msg="loaded runners" count=1
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2024-08-23T17:02:20.157Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="1e6f655" tid="123602874314752" timestamp=1724432540
INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123602874314752" timestamp=1724432540 total_threads=48
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="37647" tid="123602874314752" timestamp=1724432540
llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Large Instruct 2407
llama_model_loader: - kv   3:                            general.version str              = 2407
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral
llama_model_loader: - kv   6:                         general.size_label str              = Large
llama_model_loader: - kv   7:                            general.license str              = other
llama_model_loader: - kv   8:                       general.license.name str              = mrl
llama_model_loader: - kv   9:                       general.license.link str              = https://mistral.ai/licenses/MRL-0.1.md
llama_model_loader: - kv  10:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv  11:                          llama.block_count u32              = 88
llama_model_loader: - kv  12:                       llama.context_length u32              = 32768
llama_model_loader: - kv  13:                     llama.embedding_length u32              = 12288
llama_model_loader: - kv  14:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  15:                 llama.attention.head_count u32              = 96
llama_model_loader: - kv  16:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  17:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                          general.file_type u32              = 2
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  26:                      tokenizer.ggml.scores arr[f32,32768]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,32768]   = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  177 tensors
llama_model_loader: - type q4_0:  617 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 771
llm_load_vocab: token to piece cache size = 0.1732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32768
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 12288
llm_load_print_meta: n_layer          = 88
llm_load_print_meta: n_head           = 96
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 12
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 122.61 B
llm_load_print_meta: model size       = 64.34 GiB (4.51 BPW)
llm_load_print_meta: general.name     = Mistral Large Instruct 2407
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 781 '<0x0A>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-08-23T17:02:20.608Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-08-23T17:02:23.565Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloaded 62/89 layers to GPU
llm_load_tensors:        CPU buffer size = 65879.30 MiB
llm_load_tensors:      CUDA0 buffer size = 46040.81 MiB
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =   208.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   496.00 MiB
llama_new_context_with_model: KV self size  =  704.00 MiB, K (f16):  352.00 MiB, V (f16):  352.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.17 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   513.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    28.01 MiB
llama_new_context_with_model: graph nodes  = 2822
llama_new_context_with_model: graph splits = 290
INFO [main] model loaded | tid="123602874314752" timestamp=1724432546
time=2024-08-23T17:02:26.576Z level=INFO source=server.go:630 msg="llama runner started in 6.42 seconds"
[GIN] 2024/08/23 - 17:02:26 | 200 |  6.603581689s |       127.0.0.1 | POST     "/api/chat"
<!-- gh-comment-id:2307479709 --> @Maltz42 commented on GitHub (Aug 23, 2024): ``` [GIN] 2024/08/23 - 17:02:19 | 200 | 23.69µs | 127.0.0.1 | HEAD "/" [GIN] 2024/08/23 - 17:02:19 | 200 | 3.728994ms | 127.0.0.1 | POST "/api/show" time=2024-08-23T17:02:20.155Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 l layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB" time=2024-08-23T17:02:20.157Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama3372309948/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --parallel 1 --port 37647" time=2024-08-23T17:02:20.157Z level=INFO source=sched.go:445 msg="loaded runners" count=1 time=2024-08-23T17:02:20.157Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2024-08-23T17:02:20.157Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="1e6f655" tid="123602874314752" timestamp=1724432540 INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="123602874314752" timestamp=1724432540 total_threads=48 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="37647" tid="123602874314752" timestamp=1724432540 llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Mistral Large Instruct 2407 llama_model_loader: - kv 3: general.version str = 2407 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Mistral llama_model_loader: - kv 6: general.size_label str = Large llama_model_loader: - kv 7: general.license str = other llama_model_loader: - kv 8: general.license.name str = mrl llama_model_loader: - kv 9: general.license.link str = https://mistral.ai/licenses/MRL-0.1.md llama_model_loader: - kv 10: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ... llama_model_loader: - kv 11: llama.block_count u32 = 88 llama_model_loader: - kv 12: llama.context_length u32 = 32768 llama_model_loader: - kv 13: llama.embedding_length u32 = 12288 llama_model_loader: - kv 14: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 15: llama.attention.head_count u32 = 96 llama_model_loader: - kv 16: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 17: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 18: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - kv 20: llama.vocab_size u32 = 32768 llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 23: tokenizer.ggml.model str = llama llama_model_loader: - kv 24: tokenizer.ggml.pre str = default llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32768] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32768] = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: general.quantization_version u32 = 2 llama_model_loader: - type f32: 177 tensors llama_model_loader: - type q4_0: 617 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 771 llm_load_vocab: token to piece cache size = 0.1732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 12288 llm_load_print_meta: n_layer = 88 llm_load_print_meta: n_head = 96 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 12 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 28672 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 122.61 B llm_load_print_meta: model size = 64.34 GiB (4.51 BPW) llm_load_print_meta: general.name = Mistral Large Instruct 2407 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes llm_load_tensors: ggml ctx size = 0.74 MiB time=2024-08-23T17:02:20.608Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2024-08-23T17:02:23.565Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" llm_load_tensors: offloading 62 repeating layers to GPU llm_load_tensors: offloaded 62/89 layers to GPU llm_load_tensors: CPU buffer size = 65879.30 MiB llm_load_tensors: CUDA0 buffer size = 46040.81 MiB llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 208.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 496.00 MiB llama_new_context_with_model: KV self size = 704.00 MiB, K (f16): 352.00 MiB, V (f16): 352.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.17 MiB llama_new_context_with_model: CUDA0 compute buffer size = 513.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 28.01 MiB llama_new_context_with_model: graph nodes = 2822 llama_new_context_with_model: graph splits = 290 INFO [main] model loaded | tid="123602874314752" timestamp=1724432546 time=2024-08-23T17:02:26.576Z level=INFO source=server.go:630 msg="llama runner started in 6.42 seconds" [GIN] 2024/08/23 - 17:02:26 | 200 | 6.603581689s | 127.0.0.1 | POST "/api/chat" ```
GiteaMirror commented 2026-04-28 16:27:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 23, 2024):

If you could add OLLAMA_DEBUG=1 to the server environment, restart and load the model and then post the full log, that will be helpful.

<!-- gh-comment-id:2307509634 --> @rick-github commented on GitHub (Aug 23, 2024): If you could add `OLLAMA_DEBUG=1` to the server environment, restart and load the model and then post the full log, that will be helpful.
GiteaMirror commented 2026-04-28 16:27:31 -05:00
Author
Owner
Copy Link

@Maltz42 commented on GitHub (Aug 23, 2024):

[GIN] 2024/08/23 - 17:58:33 | 200 |       45.17µs |       127.0.0.1 | HEAD     "/"
[GIN] 2024/08/23 - 17:58:33 | 200 |    3.617657ms |       127.0.0.1 | POST     "/api/show"
time=2024-08-23T17:58:33.381Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="251.2 GiB" before.free="247.0 G



















                                                                                                                                          GiB" before.free_swap="64.0 GiB" now.total="251.2 GiB" now.free="246.8 GiB" now.free_swap="64.0 GiB"
CUDA driver version: 12.2
time=2024-08-23T17:58:33.457Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB"
time=2024-08-23T17:58:33.528Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB"
releasing cuda driver library
time=2024-08-23T17:58:33.528Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x8161c0 gpu_count=2
time=2024-08-23T17:58:33.531Z level=DEBUG source=sched.go:219 msg="loading first model" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822
time=2024-08-23T17:58:33.531Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.533Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.534Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.537Z level=DEBUG source=server.go:101 msg="system memory" total="251.2 GiB" free="246.8 GiB" free_swap="64.0 GiB"
time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]"
time=2024-08-23T17:58:33.538Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB"
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server
time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T17:58:33.541Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --verbose --parallel 1 --port 34631"
time=2024-08-23T17:58:33.541Z level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin LD_LIBRARY_PATH=/home/dfm/ollama/lib/ollama:/tmp/ollama1191698724/runners/cuda_v11 CUDA_VISIBLE_DEVICES=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55]"
time=2024-08-23T17:58:33.541Z level=INFO source=sched.go:445 msg="loaded runners" count=1
time=2024-08-23T17:58:33.541Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
time=2024-08-23T17:58:33.542Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="1e6f655" tid="129151660158976" timestamp=1724435913
INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="129151660158976" timestamp=1724435913 total_threads=48
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="34631" tid="129151660158976" timestamp=1724435913
llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Large Instruct 2407
llama_model_loader: - kv   3:                            general.version str              = 2407
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral
llama_model_loader: - kv   6:                         general.size_label str              = Large
llama_model_loader: - kv   7:                            general.license str              = other
llama_model_loader: - kv   8:                       general.license.name str              = mrl
llama_model_loader: - kv   9:                       general.license.link str              = https://mistral.ai/licenses/MRL-0.1.md
llama_model_loader: - kv  10:                          general.languages arr[str,10]      = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv  11:                          llama.block_count u32              = 88
llama_model_loader: - kv  12:                       llama.context_length u32              = 32768
llama_model_loader: - kv  13:                     llama.embedding_length u32              = 12288
llama_model_loader: - kv  14:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  15:                 llama.attention.head_count u32              = 96
llama_model_loader: - kv  16:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  17:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                          general.file_type u32              = 2
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  26:                      tokenizer.ggml.scores arr[f32,32768]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,32768]   = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  177 tensors
llama_model_loader: - type q4_0:  617 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 771
llm_load_vocab: token to piece cache size = 0.1732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32768
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 12288
llm_load_print_meta: n_layer          = 88
llm_load_print_meta: n_head           = 96
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 12
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 122.61 B
llm_load_print_meta: model size       = 64.34 GiB (4.51 BPW)
llm_load_print_meta: general.name     = Mistral Large Instruct 2407
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 781 '<0x0A>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.74 MiB
time=2024-08-23T17:58:33.993Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
time=2024-08-23T17:58:37.143Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: offloading 62 repeating layers to GPU
llm_load_tensors: offloaded 62/89 layers to GPU
llm_load_tensors:        CPU buffer size = 65879.30 MiB
llm_load_tensors:      CUDA0 buffer size = 46040.81 MiB
time=2024-08-23T17:58:37.394Z level=DEBUG source=server.go:636 msg="model load progress 0.33"
time=2024-08-23T17:58:37.645Z level=DEBUG source=server.go:636 msg="model load progress 0.42"
time=2024-08-23T17:58:37.896Z level=DEBUG source=server.go:636 msg="model load progress 0.51"
time=2024-08-23T17:58:38.147Z level=DEBUG source=server.go:636 msg="model load progress 0.60"
time=2024-08-23T17:58:38.398Z level=DEBUG source=server.go:636 msg="model load progress 0.70"
time=2024-08-23T17:58:38.649Z level=DEBUG source=server.go:636 msg="model load progress 0.79"
time=2024-08-23T17:58:38.900Z level=DEBUG source=server.go:636 msg="model load progress 0.88"
time=2024-08-23T17:58:39.151Z level=DEBUG source=server.go:636 msg="model load progress 0.97"
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =   208.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   496.00 MiB
llama_new_context_with_model: KV self size  =  704.00 MiB, K (f16):  352.00 MiB, V (f16):  352.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.17 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   513.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    28.01 MiB
llama_new_context_with_model: graph nodes  = 2822
llama_new_context_with_model: graph splits = 290
time=2024-08-23T17:58:39.401Z level=DEBUG source=server.go:636 msg="model load progress 1.00"
time=2024-08-23T17:58:39.652Z level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server loading model"
DEBUG [initialize] initializing slots | n_slots=1 tid="129151660158976" timestamp=1724435920
DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="129151660158976" timestamp=1724435920
INFO [main] model loaded | tid="129151660158976" timestamp=1724435920
DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="129151660158976" timestamp=1724435920
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="129151660158976" timestamp=1724435920
time=2024-08-23T17:58:40.154Z level=INFO source=server.go:630 msg="llama runner started in 6.61 seconds"
time=2024-08-23T17:58:40.154Z level=DEBUG source=sched.go:458 msg="finished setting up runner" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822
[GIN] 2024/08/23 - 17:58:40 | 200 |  6.777621249s |       127.0.0.1 | POST     "/api/chat"
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:462 msg="context for request finished"
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:334 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 duration=5m0s
time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:352 msg="after processing request finished event" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 refCount=0
<!-- gh-comment-id:2307560172 --> @Maltz42 commented on GitHub (Aug 23, 2024): ``` [GIN] 2024/08/23 - 17:58:33 | 200 | 45.17µs | 127.0.0.1 | HEAD "/" [GIN] 2024/08/23 - 17:58:33 | 200 | 3.617657ms | 127.0.0.1 | POST "/api/show" time=2024-08-23T17:58:33.381Z level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="251.2 GiB" before.free="247.0 G GiB" before.free_swap="64.0 GiB" now.total="251.2 GiB" now.free="246.8 GiB" now.free_swap="64.0 GiB" CUDA driver version: 12.2 time=2024-08-23T17:58:33.457Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB" time=2024-08-23T17:58:33.528Z level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 name="NVIDIA RTX A6000" overhead="0 B" before.total="47.5 GiB" before.free="47.3 GiB" now.total="47.5 GiB" now.free="47.3 GiB" now.used="266.6 MiB" releasing cuda driver library time=2024-08-23T17:58:33.528Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x8161c0 gpu_count=2 time=2024-08-23T17:58:33.531Z level=DEBUG source=sched.go:219 msg="loading first model" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 time=2024-08-23T17:58:33.531Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.532Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.533Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.534Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.535Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.536Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.537Z level=DEBUG source=server.go:101 msg="system memory" total="251.2 GiB" free="246.8 GiB" free_swap="64.0 GiB" time=2024-08-23T17:58:33.537Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[47.3 GiB]" time=2024-08-23T17:58:33.538Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=89 layers.offload=62 layers.split="" memory.available="[47.3 GiB]" memory.required.full="66.5 GiB" memory.required.partial="47.2 GiB" memory.required.kv="704.0 MiB" memory.required.allocations="[47.2 GiB]" memory.weights.total="64.5 GiB" memory.weights.repeating="64.2 GiB" memory.weights.nonrepeating="315.0 MiB" memory.graph.full="484.0 MiB" memory.graph.partial="545.0 MiB" time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cpu_avx2/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/cuda_v12/ollama_llama_server time=2024-08-23T17:58:33.538Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1191698724/runners/rocm_v60102/ollama_llama_server time=2024-08-23T17:58:33.541Z level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama1191698724/runners/cuda_v11/ollama_llama_server --model /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 62 --verbose --parallel 1 --port 34631" time=2024-08-23T17:58:33.541Z level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin LD_LIBRARY_PATH=/home/dfm/ollama/lib/ollama:/tmp/ollama1191698724/runners/cuda_v11 CUDA_VISIBLE_DEVICES=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55]" time=2024-08-23T17:58:33.541Z level=INFO source=sched.go:445 msg="loaded runners" count=1 time=2024-08-23T17:58:33.541Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" time=2024-08-23T17:58:33.542Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="1e6f655" tid="129151660158976" timestamp=1724435913 INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="129151660158976" timestamp=1724435913 total_threads=48 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="47" port="34631" tid="129151660158976" timestamp=1724435913 llama_model_loader: loaded meta data with 34 key-value pairs and 795 tensors from /home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Mistral Large Instruct 2407 llama_model_loader: - kv 3: general.version str = 2407 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Mistral llama_model_loader: - kv 6: general.size_label str = Large llama_model_loader: - kv 7: general.license str = other llama_model_loader: - kv 8: general.license.name str = mrl llama_model_loader: - kv 9: general.license.link str = https://mistral.ai/licenses/MRL-0.1.md llama_model_loader: - kv 10: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ... llama_model_loader: - kv 11: llama.block_count u32 = 88 llama_model_loader: - kv 12: llama.context_length u32 = 32768 llama_model_loader: - kv 13: llama.embedding_length u32 = 12288 llama_model_loader: - kv 14: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 15: llama.attention.head_count u32 = 96 llama_model_loader: - kv 16: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 17: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 18: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - kv 20: llama.vocab_size u32 = 32768 llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 23: tokenizer.ggml.model str = llama llama_model_loader: - kv 24: tokenizer.ggml.pre str = default llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32768] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32768] = [3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, ... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: general.quantization_version u32 = 2 llama_model_loader: - type f32: 177 tensors llama_model_loader: - type q4_0: 617 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 771 llm_load_vocab: token to piece cache size = 0.1732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 12288 llm_load_print_meta: n_layer = 88 llm_load_print_meta: n_head = 96 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 12 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 28672 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 122.61 B llm_load_print_meta: model size = 64.34 GiB (4.51 BPW) llm_load_print_meta: general.name = Mistral Large Instruct 2407 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes llm_load_tensors: ggml ctx size = 0.74 MiB time=2024-08-23T17:58:33.993Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" time=2024-08-23T17:58:37.143Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" llm_load_tensors: offloading 62 repeating layers to GPU llm_load_tensors: offloaded 62/89 layers to GPU llm_load_tensors: CPU buffer size = 65879.30 MiB llm_load_tensors: CUDA0 buffer size = 46040.81 MiB time=2024-08-23T17:58:37.394Z level=DEBUG source=server.go:636 msg="model load progress 0.33" time=2024-08-23T17:58:37.645Z level=DEBUG source=server.go:636 msg="model load progress 0.42" time=2024-08-23T17:58:37.896Z level=DEBUG source=server.go:636 msg="model load progress 0.51" time=2024-08-23T17:58:38.147Z level=DEBUG source=server.go:636 msg="model load progress 0.60" time=2024-08-23T17:58:38.398Z level=DEBUG source=server.go:636 msg="model load progress 0.70" time=2024-08-23T17:58:38.649Z level=DEBUG source=server.go:636 msg="model load progress 0.79" time=2024-08-23T17:58:38.900Z level=DEBUG source=server.go:636 msg="model load progress 0.88" time=2024-08-23T17:58:39.151Z level=DEBUG source=server.go:636 msg="model load progress 0.97" llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 208.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 496.00 MiB llama_new_context_with_model: KV self size = 704.00 MiB, K (f16): 352.00 MiB, V (f16): 352.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.17 MiB llama_new_context_with_model: CUDA0 compute buffer size = 513.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 28.01 MiB llama_new_context_with_model: graph nodes = 2822 llama_new_context_with_model: graph splits = 290 time=2024-08-23T17:58:39.401Z level=DEBUG source=server.go:636 msg="model load progress 1.00" time=2024-08-23T17:58:39.652Z level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server loading model" DEBUG [initialize] initializing slots | n_slots=1 tid="129151660158976" timestamp=1724435920 DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="129151660158976" timestamp=1724435920 INFO [main] model loaded | tid="129151660158976" timestamp=1724435920 DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="129151660158976" timestamp=1724435920 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="129151660158976" timestamp=1724435920 time=2024-08-23T17:58:40.154Z level=INFO source=server.go:630 msg="llama runner started in 6.61 seconds" time=2024-08-23T17:58:40.154Z level=DEBUG source=sched.go:458 msg="finished setting up runner" model=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 [GIN] 2024/08/23 - 17:58:40 | 200 | 6.777621249s | 127.0.0.1 | POST "/api/chat" time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:462 msg="context for request finished" time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:334 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 duration=5m0s time=2024-08-23T17:58:40.155Z level=DEBUG source=sched.go:352 msg="after processing request finished event" modelPath=/home/dfm/.ollama/models/blobs/sha256-e16120252a9b0e49ed8074d11838d8b0227957a09d749d18425e491243e13822 refCount=0 ```
GiteaMirror commented 2026-04-28 16:27:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 23, 2024):

Please post the start of the log.

<!-- gh-comment-id:2307567900 --> @rick-github commented on GitHub (Aug 23, 2024): Please post the start of the log.
GiteaMirror commented 2026-04-28 16:27:32 -05:00
Author
Owner
Copy Link

@Maltz42 commented on GitHub (Aug 23, 2024):

Third time's the charm I guess... lol It does see both cards, and uses both when I load two models. That's why I limited it to just the load part of the log, but here's the log from server launch to just before I ran the model that resulted in the log above.

2024/08/23 18:06:38 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/dfm/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-08-23T18:06:38.408Z level=INFO source=images.go:752 msg="total blobs: 35"
time=2024-08-23T18:06:38.408Z level=INFO source=images.go:759 msg="total unused blobs removed: 0"
time=2024-08-23T18:06:38.409Z level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.7-rc5)"
time=2024-08-23T18:06:38.409Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3793318785/runners
time=2024-08-23T18:06:38.409Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/ollama_llama_server.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libggml.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libllama.so.gz
time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/ollama_llama_server.gz
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx2/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v11/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v12/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/rocm_v60102/ollama_llama_server
time=2024-08-23T18:06:43.205Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]"
time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-08-23T18:06:43.205Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-08-23T18:06:43.205Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/home/dfm/ollama/lib/ollama/libcuda.so* /home/dfm/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-08-23T18:06:43.206Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01]
CUDA driver version: 12.2
time=2024-08-23T18:06:46.702Z level=DEBUG source=gpu.go:119 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA totalMem 48676 mb
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA freeMem 48410 mb
[GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] Compute Capability 8.6
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA totalMem 48676 mb
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA freeMem 48410 mb
[GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] Compute Capability 8.6
time=2024-08-23T18:06:47.026Z level=DEBUG source=amd_linux.go:371 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB"
time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB"
<!-- gh-comment-id:2307578093 --> @Maltz42 commented on GitHub (Aug 23, 2024): Third time's the charm I guess... lol It does see both cards, and uses both when I load two models. That's why I limited it to just the load part of the log, but here's the log from server launch to just before I ran the model that resulted in the log above. ``` 2024/08/23 18:06:38 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/dfm/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-08-23T18:06:38.408Z level=INFO source=images.go:752 msg="total blobs: 35" time=2024-08-23T18:06:38.408Z level=INFO source=images.go:759 msg="total unused blobs removed: 0" time=2024-08-23T18:06:38.409Z level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.7-rc5)" time=2024-08-23T18:06:38.409Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3793318785/runners time=2024-08-23T18:06:38.409Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v12 file=build/linux/x86_64/cuda_v12/bin/ollama_llama_server.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libggml.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/libllama.so.gz time=2024-08-23T18:06:38.410Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60102 file=build/linux/x86_64/rocm_v60102/bin/ollama_llama_server.gz time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu/ollama_llama_server time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx/ollama_llama_server time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cpu_avx2/ollama_llama_server time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v11/ollama_llama_server time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/cuda_v12/ollama_llama_server time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama3793318785/runners/rocm_v60102/ollama_llama_server time=2024-08-23T18:06:43.205Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]" time=2024-08-23T18:06:43.205Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-08-23T18:06:43.205Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-08-23T18:06:43.205Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs" time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA" time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so* time=2024-08-23T18:06:43.205Z level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[/home/dfm/ollama/lib/ollama/libcuda.so* /home/dfm/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2024-08-23T18:06:43.206Z level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01] CUDA driver version: 12.2 time=2024-08-23T18:06:46.702Z level=DEBUG source=gpu.go:119 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01 [GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA totalMem 48676 mb [GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] CUDA freeMem 48410 mb [GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55] Compute Capability 8.6 [GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA totalMem 48676 mb [GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] CUDA freeMem 48410 mb [GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877] Compute Capability 8.6 time=2024-08-23T18:06:47.026Z level=DEBUG source=amd_linux.go:371 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-e82bfaa5-39d7-87e2-c7f6-a94037447a55 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB" time=2024-08-23T18:06:47.026Z level=INFO source=types.go:107 msg="inference compute" id=GPU-ac1016b1-dfff-9e33-d9a3-753373cc1877 library=cuda variant=v11 compute=8.6 driver=12.2 name="NVIDIA RTX A6000" total="47.5 GiB" available="47.3 GiB" ```
GiteaMirror commented 2026-04-28 16:27:33 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 23, 2024):

I've repro'd. We'll get a fix for this before shipping 0.3.7. Thanks!

<!-- gh-comment-id:2307784768 --> @dhiltgen commented on GitHub (Aug 23, 2024): I've repro'd. We'll get a fix for this before shipping 0.3.7. Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#50589
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?