github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9858] Fast inference via huggingface/unsloth, but slow inference speed in Ollama (converted to Q8_0 GGUF) #6455

New Issue
Closed
opened 2026-04-12 18:00:52 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-04-12 18:00:52 -05:00
Owner
Copy Link

Originally created by @phiwi on GitHub (Mar 18, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9858

What is the issue?

I finetrained the unsloth/Llama-3.3-70B-Instruct with unsloth and running it directly as FastLanguageModel produces fast and reasonable results. When I save it via

    model.save_pretrained_gguf(
        f"{sft_model_path}.GGUF",
        tokenizer,
        quantization_method="q8_0",
    )

it produces a ~70B big unsloth.Q8_0.gguf savefile and a corresponding Modelfile.

When I convert it with Ollama

Ollama create -f Modelfile llama-custom # arbitrary model name

and load it in Ollama, it takes ages to laod and inference takes an hour on 4 x A40 or 1 x H100.

Can anyone deduct the root cause from the Ollama starting output below? Much appreciated.

Relevant log output

time=2025-03-18T16:11:04.607+01:00 level=INFO source=server.go:97 msg="system memory" total="377.4 GiB" free="371.5 GiB" free_swap="971.7 MiB"
time=2025-03-18T16:11:04.608+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=55 layers.split="" memory.available="[92.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="126.0 GiB" memory.required.partial
="91.6 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[91.6 GiB]" memory.weights.total="106.8 GiB" memory.weights.repeating="105.8 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="15.9 GiB" memory.graph.partial="16.4 GiB"
time=2025-03-18T16:11:04.610+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e --ctx-siz
e 128000 --batch-size 512 --n-gpu-layers 55 --threads 32 --parallel 1 --port 45467"
time=2025-03-18T16:11:04.634+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-18T16:11:04.636+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"                                                                                                                                                             
time=2025-03-18T16:11:04.645+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"                                
time=2025-03-18T16:11:04.694+01:00 level=INFO source=runner.go:931 msg="starting go runner"   
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
 Device 0: NVIDIA H100 NVL, compute capability 9.0, VMM: yes                                                                            
load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so                                   
load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so                                      
time=2025-03-18T16:11:04.890+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | 
FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=32                                          
time=2025-03-18T16:11:04.891+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:45467"                             
time=2025-03-18T16:11:04.898+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"                                                                                                                                 
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA H100 NVL) - 94805 MiB free                                                   
llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e (version GGUF V3 (latest))                                                       
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.                                        
llama_model_loader: - kv   0:                       general.architecture str              = llama                                        
llama_model_loader: - kv   1:                               general.type str              = model                                        
llama_model_loader: - kv   2:                               general.name str              = Llama 3.3 70b Instruct Bnb 4bit              
llama_model_loader: - kv   3:                       general.organization str              = Unsloth                                      
llama_model_loader: - kv   4:                           general.finetune str              = instruct-bnb-4bit                            
llama_model_loader: - kv   5:                           general.basename str              = llama-3.3                                    
llama_model_loader: - kv   6:                         general.size_label str              = 70B                                          
llama_model_loader: - kv   7:                          llama.block_count u32              = 80                                           
llama_model_loader: - kv   8:                       llama.context_length u32              = 131072                                       
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 8192                                         
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 28672                                        
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 64                                           
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 8                                            
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 500000.000000                                
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010                                     
llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 128                                          
llama_model_loader: - kv  16:               llama.attention.value_length u32              = 128                                          
llama_model_loader: - kv  17:                          general.file_type u32              = 7                                            
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256                                       
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128                                          
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2                                         
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe                                    
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...     
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...     
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...               
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000                                       
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009                                       
llama_model_loader: - kv  27:            tokenizer.ggml.padding_token_id u32              = 128004                                       
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {% for message in messages %}{% if me...     
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  162 tensors
llama_model_loader: - type q8_0:  562 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 69.82 GiB (8.50 BPW) 
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 8192
print_info: n_layer          = 80
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 28672
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 70B
print_info: model params     = 70.55 B
print_info: general.name     = Llama 3.3 70b Instruct Bnb 4bit
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|im_end|>'
print_info: EOT token        = 128009 '<|im_end|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: PAD token        = 128004 '<|finetune_right_pad_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @phiwi on GitHub (Mar 18, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9858 ### What is the issue? I finetrained the [unsloth/Llama-3.3-70B-Instruct](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct) with unsloth and running it directly as `FastLanguageModel` produces **fast** and **reasonable** results. When I save it via ``` model.save_pretrained_gguf( f"{sft_model_path}.GGUF", tokenizer, quantization_method="q8_0", ) ``` it produces a ~70B big `unsloth.Q8_0.gguf` savefile and a corresponding `Modelfile`. When I convert it with Ollama ``` Ollama create -f Modelfile llama-custom # arbitrary model name ``` and load it in Ollama, it takes ages to laod and inference takes an hour on 4 x A40 or 1 x H100. Can anyone deduct the root cause from the Ollama starting output below? Much appreciated. ### Relevant log output ```shell time=2025-03-18T16:11:04.607+01:00 level=INFO source=server.go:97 msg="system memory" total="377.4 GiB" free="371.5 GiB" free_swap="971.7 MiB" time=2025-03-18T16:11:04.608+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=55 layers.split="" memory.available="[92.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="126.0 GiB" memory.required.partial ="91.6 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[91.6 GiB]" memory.weights.total="106.8 GiB" memory.weights.repeating="105.8 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="15.9 GiB" memory.graph.partial="16.4 GiB" time=2025-03-18T16:11:04.610+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e --ctx-siz e 128000 --batch-size 512 --n-gpu-layers 55 --threads 32 --parallel 1 --port 45467" time=2025-03-18T16:11:04.634+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-18T16:11:04.636+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-03-18T16:11:04.645+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-03-18T16:11:04.694+01:00 level=INFO source=runner.go:931 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA H100 NVL, compute capability 9.0, VMM: yes load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so time=2025-03-18T16:11:04.890+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=32 time=2025-03-18T16:11:04.891+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:45467" time=2025-03-18T16:11:04.898+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA H100 NVL) - 94805 MiB free llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Llama 3.3 70b Instruct Bnb 4bit llama_model_loader: - kv 3: general.organization str = Unsloth llama_model_loader: - kv 4: general.finetune str = instruct-bnb-4bit llama_model_loader: - kv 5: general.basename str = llama-3.3 llama_model_loader: - kv 6: general.size_label str = 70B llama_model_loader: - kv 7: llama.block_count u32 = 80 llama_model_loader: - kv 8: llama.context_length u32 = 131072 llama_model_loader: - kv 9: llama.embedding_length u32 = 8192 llama_model_loader: - kv 10: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 11: llama.attention.head_count u32 = 64 llama_model_loader: - kv 12: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 13: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 14: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 15: llama.attention.key_length u32 = 128 llama_model_loader: - kv 16: llama.attention.value_length u32 = 128 llama_model_loader: - kv 17: general.file_type u32 = 7 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.ggml.padding_token_id u32 = 128004 llama_model_loader: - kv 28: tokenizer.chat_template str = {% for message in messages %}{% if me... llama_model_loader: - kv 29: general.quantization_version u32 = 2 llama_model_loader: - type f32: 162 tensors llama_model_loader: - type q8_0: 562 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q8_0 print_info: file size = 69.82 GiB (8.50 BPW) load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 8192 print_info: n_layer = 80 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: n_ff = 28672 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = linear print_info: freq_base_train = 500000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 131072 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 70B print_info: model params = 70.55 B print_info: general.name = Llama 3.3 70b Instruct Bnb 4bit print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|im_end|>' print_info: EOT token = 128009 '<|im_end|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: PAD token = 128004 '<|finetune_right_pad_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|im_end|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-12 18:00:52 -05:00
GiteaMirror commented 2026-04-12 18:00:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 18, 2025):

A full log would make it easier to root cause. The start includes configuration, device detection, memory calculations, the end shows model loading, layer assignment, prompt processing.

<!-- gh-comment-id:2734083480 --> @rick-github commented on GitHub (Mar 18, 2025): A full log would make it easier to root cause. The start includes configuration, device detection, memory calculations, the end shows model loading, layer assignment, prompt processing.
GiteaMirror commented 2026-04-12 18:00:53 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

Thank you for your message. Here is the full log. I let the model run on 3xA40 (which was usually no problem).

I can see multiple concerning notifications:

  • time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
  • load_tensors: tensor 'token_embd.weight' (q8_0) (and 232 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
  • time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB

Has something gone wrong with saving the model from usnloth with llama.cpp or while creating the model via ollama?

2025/03/19 09:35:19 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,2,3 GPU_DEVICE_ORDINAL:0,2,3 HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:128000 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://10.250.135.153:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:240h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/pwiesenbach/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0,2,3 http_proxy: https_proxy: no_proxy:]"
time=2025-03-19T09:35:19.499+01:00 level=INFO source=images.go:432 msg="total blobs: 40"
time=2025-03-19T09:35:19.519+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-03-19T09:35:19.531+01:00 level=INFO source=routes.go:1277 msg="Listening on 10.250.135.153:11434 (version 0.5.13)"
time=2025-03-19T09:35:19.531+01:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler"
time=2025-03-19T09:35:19.532+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/beegfs/biosw/ollama/0.5.13/lib/ollama/libcuda.so* /biosw/ollama/0.5.13/lib/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-03-19T09:35:19.538+01:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:35:19.894+01:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=3 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA totalMem 45499 mb
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA freeMem 45230 mb
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] Compute Capability 8.6
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA totalMem 45499 mb
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA freeMem 45230 mb
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] Compute Capability 8.6
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA totalMem 45499 mb
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA freeMem 45230 mb
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] Compute Capability 8.6
time=2025-03-19T09:35:20.411+01:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
[GIN] 2025/03/19 - 09:35:37 | 200 |     659.915µs |  10.250.135.104 | HEAD     "/"
[GIN] 2025/03/19 - 09:35:38 | 200 |  950.058751ms |  10.250.135.104 | POST     "/api/show"
time=2025-03-19T09:35:38.403+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.2 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:35:38.580+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:35:38.755+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:35:38.926+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:35:38.926+01:00 level=DEBUG source=sched.go:182 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3
time=2025-03-19T09:35:38.990+01:00 level=DEBUG source=sched.go:225 msg="loading first model" model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:35:38.990+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.994+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.995+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T09:35:38.995+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T09:35:38.997+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:35:39.161+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:35:39.322+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:35:39.484+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:35:39.484+01:00 level=INFO source=server.go:97 msg="system memory" total="503.6 GiB" free="481.1 GiB" free_swap="152.6 GiB"
time=2025-03-19T09:35:39.484+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T09:35:39.485+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=57 layers.split=19,19,19 memory.available="[44.2 GiB 44.2 GiB 44.2 GiB]" memory.gpu_overhead="0 B" memory.required.full="162.5 GiB" memory.required.partial="130.7 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[43.6 GiB 43.6 GiB 43.6 GiB]" memory.weights.total="106.8 GiB" memory.weights.repeating="105.8 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="16.4 GiB" memory.graph.partial="16.4 GiB"
time=2025-03-19T09:35:39.486+01:00 level=DEBUG source=server.go:259 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]"
time=2025-03-19T09:35:39.487+01:00 level=DEBUG source=server.go:302 msg="adding gpu library" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12
time=2025-03-19T09:35:39.487+01:00 level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12]
time=2025-03-19T09:35:39.488+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e --ctx-size 128000 --batch-size 512 --n-gpu-layers 57 --verbose --threads 16 --parallel 1 --tensor-split 19,19,19 --port 42097"
time=2025-03-19T09:35:39.488+01:00 level=DEBUG source=server.go:398 msg=subprocess environment="[ROCR_VISIBLE_DEVICES=0,2,3 CUDA_VISIBLE_DEVICES=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18,GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452,GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 GPU_DEVICE_ORDINAL=0,2,3 LD_LIBRARY_PATH=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/biosw/ollama/0.5.13/lib:/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/beegfs/biosw/ollama/0.5.13/lib/ollama PATH=.local/bin:/opt/singularity/bin:/biosw/ollama/0.5.13/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/home/pwiesenbach/bin:/home/pwiesenbach/.cargo/bin:.local/bin:/opt/singularity/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games]"
time=2025-03-19T09:35:39.496+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-19T09:35:39.498+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
time=2025-03-19T09:35:39.502+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
time=2025-03-19T09:35:39.558+01:00 level=INFO source=runner.go:931 msg="starting go runner"
time=2025-03-19T09:35:39.558+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 CUDA devices:
  Device 0: NVIDIA A40, compute capability 8.6, VMM: yes
  Device 1: NVIDIA A40, compute capability 8.6, VMM: yes
  Device 2: NVIDIA A40, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-03-19T09:35:40.012+01:00 level=DEBUG source=ggml.go:78 msg="skipping path which is not part of ollama" path=/biosw/ollama/0.5.13/lib
time=2025-03-19T09:35:40.012+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-sandybridge.so score: 20
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-skylakex.so score: 0
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so score: 55
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-alderlake.so score: 0
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so score: 0
load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so
time=2025-03-19T09:35:40.026+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16
time=2025-03-19T09:35:40.027+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:42097"
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45230 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45230 MiB free
llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45230 MiB free
llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.3 70b Instruct Bnb 4bit
llama_model_loader: - kv   3:                       general.organization str              = Unsloth
llama_model_loader: - kv   4:                           general.finetune str              = instruct-bnb-4bit
llama_model_loader: - kv   5:                           general.basename str              = llama-3.3
llama_model_loader: - kv   6:                         general.size_label str              = 70B
llama_model_loader: - kv   7:                          llama.block_count u32              = 80
llama_model_loader: - kv   8:                       llama.context_length u32              = 131072
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 8192
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 64
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  16:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  17:                          general.file_type u32              = 7
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
time=2025-03-19T09:35:40.257+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:            tokenizer.ggml.padding_token_id u32              = 128004
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {% for message in messages %}{% if me...
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  162 tensors
llama_model_loader: - type q8_0:  562 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 69.82 GiB (8.50 BPW) 
init_tokenizer: initializing tokenizer for type 2
load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG
load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG
load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG
load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
load: control token: 128007 '<|end_header_id|>' is not marked as EOG
load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG
load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
load: control token: 128001 '<|end_of_text|>' is not marked as EOG
load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
load: control token: 128006 '<|start_header_id|>' is not marked as EOG
load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
load: control token: 128000 '<|begin_of_text|>' is not marked as EOG
load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
load: control token: 128010 '<|python_tag|>' is not marked as EOG
load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG
load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 8192
print_info: n_layer          = 80
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 28672
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 70B
print_info: model params     = 70.55 B
print_info: general.name     = Llama 3.3 70b Instruct Bnb 4bit
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|im_end|>'
print_info: EOT token        = 128009 '<|im_end|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: PAD token        = 128004 '<|finetune_right_pad_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer   0 assigned to device CPU
load_tensors: layer   1 assigned to device CPU
load_tensors: layer   2 assigned to device CPU
load_tensors: layer   3 assigned to device CPU
load_tensors: layer   4 assigned to device CPU
load_tensors: layer   5 assigned to device CPU
load_tensors: layer   6 assigned to device CPU
load_tensors: layer   7 assigned to device CPU
load_tensors: layer   8 assigned to device CPU
load_tensors: layer   9 assigned to device CPU
load_tensors: layer  10 assigned to device CPU
load_tensors: layer  11 assigned to device CPU
load_tensors: layer  12 assigned to device CPU
load_tensors: layer  13 assigned to device CPU
load_tensors: layer  14 assigned to device CPU
load_tensors: layer  15 assigned to device CPU
load_tensors: layer  16 assigned to device CPU
load_tensors: layer  17 assigned to device CPU
load_tensors: layer  18 assigned to device CPU
load_tensors: layer  19 assigned to device CPU
load_tensors: layer  20 assigned to device CPU
load_tensors: layer  21 assigned to device CPU
load_tensors: layer  22 assigned to device CPU
load_tensors: layer  23 assigned to device CUDA0
load_tensors: layer  24 assigned to device CUDA0
load_tensors: layer  25 assigned to device CUDA0
load_tensors: layer  26 assigned to device CUDA0
load_tensors: layer  27 assigned to device CUDA0
load_tensors: layer  28 assigned to device CUDA0
load_tensors: layer  29 assigned to device CUDA0
load_tensors: layer  30 assigned to device CUDA0
load_tensors: layer  31 assigned to device CUDA0
load_tensors: layer  32 assigned to device CUDA0
load_tensors: layer  33 assigned to device CUDA0
load_tensors: layer  34 assigned to device CUDA0
load_tensors: layer  35 assigned to device CUDA0
load_tensors: layer  36 assigned to device CUDA0
load_tensors: layer  37 assigned to device CUDA0
load_tensors: layer  38 assigned to device CUDA0
load_tensors: layer  39 assigned to device CUDA0
load_tensors: layer  40 assigned to device CUDA0
load_tensors: layer  41 assigned to device CUDA0
load_tensors: layer  42 assigned to device CUDA1
load_tensors: layer  43 assigned to device CUDA1
load_tensors: layer  44 assigned to device CUDA1
load_tensors: layer  45 assigned to device CUDA1
load_tensors: layer  46 assigned to device CUDA1
load_tensors: layer  47 assigned to device CUDA1
load_tensors: layer  48 assigned to device CUDA1
load_tensors: layer  49 assigned to device CUDA1
load_tensors: layer  50 assigned to device CUDA1
load_tensors: layer  51 assigned to device CUDA1
load_tensors: layer  52 assigned to device CUDA1
load_tensors: layer  53 assigned to device CUDA1
load_tensors: layer  54 assigned to device CUDA1
load_tensors: layer  55 assigned to device CUDA1
load_tensors: layer  56 assigned to device CUDA1
load_tensors: layer  57 assigned to device CUDA1
load_tensors: layer  58 assigned to device CUDA1
load_tensors: layer  59 assigned to device CUDA1
load_tensors: layer  60 assigned to device CUDA1
load_tensors: layer  61 assigned to device CUDA2
load_tensors: layer  62 assigned to device CUDA2
load_tensors: layer  63 assigned to device CUDA2
load_tensors: layer  64 assigned to device CUDA2
load_tensors: layer  65 assigned to device CUDA2
load_tensors: layer  66 assigned to device CUDA2
load_tensors: layer  67 assigned to device CUDA2
load_tensors: layer  68 assigned to device CUDA2
load_tensors: layer  69 assigned to device CUDA2
load_tensors: layer  70 assigned to device CUDA2
load_tensors: layer  71 assigned to device CUDA2
load_tensors: layer  72 assigned to device CUDA2
load_tensors: layer  73 assigned to device CUDA2
load_tensors: layer  74 assigned to device CUDA2
load_tensors: layer  75 assigned to device CUDA2
load_tensors: layer  76 assigned to device CUDA2
load_tensors: layer  77 assigned to device CUDA2
load_tensors: layer  78 assigned to device CUDA2
load_tensors: layer  79 assigned to device CUDA2
load_tensors: layer  80 assigned to device CPU
load_tensors: tensor 'token_embd.weight' (q8_0) (and 232 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
time=2025-03-19T09:40:39.629+01:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - "
time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:361 msg="runner expired event received" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
[GIN] 2025/03/19 - 09:40:39 | 500 |          5m1s |  10.250.135.104 | POST     "/api/generate"
time=2025-03-19T09:40:39.654+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.6 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:39.801+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:39.949+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=server.go:1081 msg="stopping llama server"
time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=server.go:1087 msg="waiting for llama server to exit"
time=2025-03-19T09:40:40.345+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.6 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:40.489+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:40.637+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:40.779+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:40.779+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:40.922+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:41.065+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:41.207+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:41.208+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:41.351+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:41.497+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:41.639+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:41.639+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:41.784+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:41.928+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:42.070+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:42.071+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:42.219+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:42.362+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:42.510+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:42.510+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:42.660+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:42.803+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:42.947+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:42.947+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:43.091+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:43.236+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:43.378+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
releasing cuda driver library
time=2025-03-19T09:40:43.378+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:43.523+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB"
time=2025-03-19T09:40:43.701+01:00 level=DEBUG source=server.go:1091 msg="llama server stopped"
time=2025-03-19T09:40:43.701+01:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:43.783+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:43.945+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:40:43.945+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:44.101+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:44.260+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:44.424+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:40:44.424+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:44.585+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:44.746+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:40:44.907+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.277475022 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=sched.go:309 msg="ignoring unload event with no pending requests"
time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:45.067+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:45.232+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:45.392+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:40:45.392+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.763018958 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
time=2025-03-19T09:40:45.392+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fbf63d0de00
dlsym: cuDriverGetVersion - 0x7fbf63d0de20
dlsym: cuDeviceGetCount - 0x7fbf63d0de60
dlsym: cuDeviceGet - 0x7fbf63d0de40
dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40
dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0
dlsym: cuDeviceGetName - 0x7fbf63d0de80
dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120
dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0
dlsym: cuCtxDestroy - 0x7fbf63d6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T09:40:45.554+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:45.714+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T09:40:45.874+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T09:40:45.874+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=6.244668086 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e
<!-- gh-comment-id:2735758192 --> @phiwi commented on GitHub (Mar 19, 2025): Thank you for your message. Here is the full log. I let the model run on 3xA40 (which was usually no problem). I can see multiple concerning notifications: * `time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"` * `load_tensors: tensor 'token_embd.weight' (q8_0) (and 232 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead` * `time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB` Has something gone wrong with saving the model from usnloth with llama.cpp or while creating the model via ollama? ```bash 2025/03/19 09:35:19 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,2,3 GPU_DEVICE_ORDINAL:0,2,3 HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:128000 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://10.250.135.153:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:240h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/pwiesenbach/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0,2,3 http_proxy: https_proxy: no_proxy:]" time=2025-03-19T09:35:19.499+01:00 level=INFO source=images.go:432 msg="total blobs: 40" time=2025-03-19T09:35:19.519+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-03-19T09:35:19.531+01:00 level=INFO source=routes.go:1277 msg="Listening on 10.250.135.153:11434 (version 0.5.13)" time=2025-03-19T09:35:19.531+01:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler" time=2025-03-19T09:35:19.532+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-03-19T09:35:19.533+01:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/beegfs/biosw/ollama/0.5.13/lib/ollama/libcuda.so* /biosw/ollama/0.5.13/lib/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-03-19T09:35:19.538+01:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:35:19.894+01:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=3 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA totalMem 45499 mb [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA freeMem 45230 mb [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] Compute Capability 8.6 [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA totalMem 45499 mb [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA freeMem 45230 mb [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] Compute Capability 8.6 [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA totalMem 45499 mb [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA freeMem 45230 mb [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] Compute Capability 8.6 time=2025-03-19T09:35:20.411+01:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" time=2025-03-19T09:35:20.411+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" [GIN] 2025/03/19 - 09:35:37 | 200 | 659.915µs | 10.250.135.104 | HEAD "/" [GIN] 2025/03/19 - 09:35:38 | 200 | 950.058751ms | 10.250.135.104 | POST "/api/show" time=2025-03-19T09:35:38.403+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.2 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:35:38.580+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:35:38.755+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:35:38.926+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:35:38.926+01:00 level=DEBUG source=sched.go:182 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3 time=2025-03-19T09:35:38.990+01:00 level=DEBUG source=sched.go:225 msg="loading first model" model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:35:38.990+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T09:35:38.991+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T09:35:38.992+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T09:35:38.993+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.994+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.995+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T09:35:38.995+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.8 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T09:35:38.996+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T09:35:38.997+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:35:39.161+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:35:39.322+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:35:39.484+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:35:39.484+01:00 level=INFO source=server.go:97 msg="system memory" total="503.6 GiB" free="481.1 GiB" free_swap="152.6 GiB" time=2025-03-19T09:35:39.484+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T09:35:39.485+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=57 layers.split=19,19,19 memory.available="[44.2 GiB 44.2 GiB 44.2 GiB]" memory.gpu_overhead="0 B" memory.required.full="162.5 GiB" memory.required.partial="130.7 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[43.6 GiB 43.6 GiB 43.6 GiB]" memory.weights.total="106.8 GiB" memory.weights.repeating="105.8 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="16.4 GiB" memory.graph.partial="16.4 GiB" time=2025-03-19T09:35:39.486+01:00 level=DEBUG source=server.go:259 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" time=2025-03-19T09:35:39.487+01:00 level=DEBUG source=server.go:302 msg="adding gpu library" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12 time=2025-03-19T09:35:39.487+01:00 level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12] time=2025-03-19T09:35:39.488+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e --ctx-size 128000 --batch-size 512 --n-gpu-layers 57 --verbose --threads 16 --parallel 1 --tensor-split 19,19,19 --port 42097" time=2025-03-19T09:35:39.488+01:00 level=DEBUG source=server.go:398 msg=subprocess environment="[ROCR_VISIBLE_DEVICES=0,2,3 CUDA_VISIBLE_DEVICES=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18,GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452,GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 GPU_DEVICE_ORDINAL=0,2,3 LD_LIBRARY_PATH=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/biosw/ollama/0.5.13/lib:/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/beegfs/biosw/ollama/0.5.13/lib/ollama PATH=.local/bin:/opt/singularity/bin:/biosw/ollama/0.5.13/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/home/pwiesenbach/bin:/home/pwiesenbach/.cargo/bin:.local/bin:/opt/singularity/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games]" time=2025-03-19T09:35:39.496+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-19T09:35:39.498+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-03-19T09:35:39.502+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-03-19T09:35:39.558+01:00 level=INFO source=runner.go:931 msg="starting go runner" time=2025-03-19T09:35:39.558+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 3 CUDA devices: Device 0: NVIDIA A40, compute capability 8.6, VMM: yes Device 1: NVIDIA A40, compute capability 8.6, VMM: yes Device 2: NVIDIA A40, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so time=2025-03-19T09:35:40.012+01:00 level=DEBUG source=ggml.go:78 msg="skipping path which is not part of ollama" path=/biosw/ollama/0.5.13/lib time=2025-03-19T09:35:40.012+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-sandybridge.so score: 20 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-skylakex.so score: 0 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so score: 55 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-alderlake.so score: 0 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so score: 0 load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so time=2025-03-19T09:35:40.026+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16 time=2025-03-19T09:35:40.027+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:42097" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45230 MiB free llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45230 MiB free llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45230 MiB free llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Llama 3.3 70b Instruct Bnb 4bit llama_model_loader: - kv 3: general.organization str = Unsloth llama_model_loader: - kv 4: general.finetune str = instruct-bnb-4bit llama_model_loader: - kv 5: general.basename str = llama-3.3 llama_model_loader: - kv 6: general.size_label str = 70B llama_model_loader: - kv 7: llama.block_count u32 = 80 llama_model_loader: - kv 8: llama.context_length u32 = 131072 llama_model_loader: - kv 9: llama.embedding_length u32 = 8192 llama_model_loader: - kv 10: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 11: llama.attention.head_count u32 = 64 llama_model_loader: - kv 12: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 13: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 14: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 15: llama.attention.key_length u32 = 128 llama_model_loader: - kv 16: llama.attention.value_length u32 = 128 llama_model_loader: - kv 17: general.file_type u32 = 7 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... time=2025-03-19T09:35:40.257+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.ggml.padding_token_id u32 = 128004 llama_model_loader: - kv 28: tokenizer.chat_template str = {% for message in messages %}{% if me... llama_model_loader: - kv 29: general.quantization_version u32 = 2 llama_model_loader: - type f32: 162 tensors llama_model_loader: - type q8_0: 562 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q8_0 print_info: file size = 69.82 GiB (8.50 BPW) init_tokenizer: initializing tokenizer for type 2 load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG load: control token: 128007 '<|end_header_id|>' is not marked as EOG load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG load: control token: 128001 '<|end_of_text|>' is not marked as EOG load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG load: control token: 128006 '<|start_header_id|>' is not marked as EOG load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG load: control token: 128000 '<|begin_of_text|>' is not marked as EOG load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG load: control token: 128010 '<|python_tag|>' is not marked as EOG load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 8192 print_info: n_layer = 80 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: n_ff = 28672 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = linear print_info: freq_base_train = 500000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 131072 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 70B print_info: model params = 70.55 B print_info: general.name = Llama 3.3 70b Instruct Bnb 4bit print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|im_end|>' print_info: EOT token = 128009 '<|im_end|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: PAD token = 128004 '<|finetune_right_pad_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|im_end|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: layer 0 assigned to device CPU load_tensors: layer 1 assigned to device CPU load_tensors: layer 2 assigned to device CPU load_tensors: layer 3 assigned to device CPU load_tensors: layer 4 assigned to device CPU load_tensors: layer 5 assigned to device CPU load_tensors: layer 6 assigned to device CPU load_tensors: layer 7 assigned to device CPU load_tensors: layer 8 assigned to device CPU load_tensors: layer 9 assigned to device CPU load_tensors: layer 10 assigned to device CPU load_tensors: layer 11 assigned to device CPU load_tensors: layer 12 assigned to device CPU load_tensors: layer 13 assigned to device CPU load_tensors: layer 14 assigned to device CPU load_tensors: layer 15 assigned to device CPU load_tensors: layer 16 assigned to device CPU load_tensors: layer 17 assigned to device CPU load_tensors: layer 18 assigned to device CPU load_tensors: layer 19 assigned to device CPU load_tensors: layer 20 assigned to device CPU load_tensors: layer 21 assigned to device CPU load_tensors: layer 22 assigned to device CPU load_tensors: layer 23 assigned to device CUDA0 load_tensors: layer 24 assigned to device CUDA0 load_tensors: layer 25 assigned to device CUDA0 load_tensors: layer 26 assigned to device CUDA0 load_tensors: layer 27 assigned to device CUDA0 load_tensors: layer 28 assigned to device CUDA0 load_tensors: layer 29 assigned to device CUDA0 load_tensors: layer 30 assigned to device CUDA0 load_tensors: layer 31 assigned to device CUDA0 load_tensors: layer 32 assigned to device CUDA0 load_tensors: layer 33 assigned to device CUDA0 load_tensors: layer 34 assigned to device CUDA0 load_tensors: layer 35 assigned to device CUDA0 load_tensors: layer 36 assigned to device CUDA0 load_tensors: layer 37 assigned to device CUDA0 load_tensors: layer 38 assigned to device CUDA0 load_tensors: layer 39 assigned to device CUDA0 load_tensors: layer 40 assigned to device CUDA0 load_tensors: layer 41 assigned to device CUDA0 load_tensors: layer 42 assigned to device CUDA1 load_tensors: layer 43 assigned to device CUDA1 load_tensors: layer 44 assigned to device CUDA1 load_tensors: layer 45 assigned to device CUDA1 load_tensors: layer 46 assigned to device CUDA1 load_tensors: layer 47 assigned to device CUDA1 load_tensors: layer 48 assigned to device CUDA1 load_tensors: layer 49 assigned to device CUDA1 load_tensors: layer 50 assigned to device CUDA1 load_tensors: layer 51 assigned to device CUDA1 load_tensors: layer 52 assigned to device CUDA1 load_tensors: layer 53 assigned to device CUDA1 load_tensors: layer 54 assigned to device CUDA1 load_tensors: layer 55 assigned to device CUDA1 load_tensors: layer 56 assigned to device CUDA1 load_tensors: layer 57 assigned to device CUDA1 load_tensors: layer 58 assigned to device CUDA1 load_tensors: layer 59 assigned to device CUDA1 load_tensors: layer 60 assigned to device CUDA1 load_tensors: layer 61 assigned to device CUDA2 load_tensors: layer 62 assigned to device CUDA2 load_tensors: layer 63 assigned to device CUDA2 load_tensors: layer 64 assigned to device CUDA2 load_tensors: layer 65 assigned to device CUDA2 load_tensors: layer 66 assigned to device CUDA2 load_tensors: layer 67 assigned to device CUDA2 load_tensors: layer 68 assigned to device CUDA2 load_tensors: layer 69 assigned to device CUDA2 load_tensors: layer 70 assigned to device CUDA2 load_tensors: layer 71 assigned to device CUDA2 load_tensors: layer 72 assigned to device CUDA2 load_tensors: layer 73 assigned to device CUDA2 load_tensors: layer 74 assigned to device CUDA2 load_tensors: layer 75 assigned to device CUDA2 load_tensors: layer 76 assigned to device CUDA2 load_tensors: layer 77 assigned to device CUDA2 load_tensors: layer 78 assigned to device CUDA2 load_tensors: layer 79 assigned to device CUDA2 load_tensors: layer 80 assigned to device CPU load_tensors: tensor 'token_embd.weight' (q8_0) (and 232 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead time=2025-03-19T09:40:39.629+01:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - " time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:361 msg="runner expired event received" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:39.629+01:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e [GIN] 2025/03/19 - 09:40:39 | 500 | 5m1s | 10.250.135.104 | POST "/api/generate" time=2025-03-19T09:40:39.654+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.6 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:39.801+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:39.949+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=server.go:1081 msg="stopping llama server" time=2025-03-19T09:40:40.095+01:00 level=DEBUG source=server.go:1087 msg="waiting for llama server to exit" time=2025-03-19T09:40:40.345+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.6 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:40.489+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:40.637+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:40.779+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:40.779+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:40.922+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:41.065+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:41.207+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:41.208+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:41.351+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:41.497+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:41.639+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:41.639+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:41.784+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:41.928+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:42.070+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:42.071+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:42.219+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:42.362+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:42.510+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:42.510+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:42.660+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:42.803+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:42.947+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:42.947+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:43.091+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:43.236+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:43.378+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" releasing cuda driver library time=2025-03-19T09:40:43.378+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="480.7 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:43.523+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="43.9 GiB" now.used="533.7 MiB" time=2025-03-19T09:40:43.701+01:00 level=DEBUG source=server.go:1091 msg="llama server stopped" time=2025-03-19T09:40:43.701+01:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:43.783+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:43.945+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:40:43.945+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="480.7 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:44.101+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="43.9 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:44.260+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:44.424+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:40:44.424+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:44.585+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:44.746+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:40:44.907+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.277475022 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=sched.go:309 msg="ignoring unload event with no pending requests" time=2025-03-19T09:40:44.907+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:45.067+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:45.232+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:45.392+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:40:45.392+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.763018958 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e time=2025-03-19T09:40:45.392+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.1 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.1 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fbf63d0de00 dlsym: cuDriverGetVersion - 0x7fbf63d0de20 dlsym: cuDeviceGetCount - 0x7fbf63d0de60 dlsym: cuDeviceGet - 0x7fbf63d0de40 dlsym: cuDeviceGetAttribute - 0x7fbf63d0df40 dlsym: cuDeviceGetUuid - 0x7fbf63d0dea0 dlsym: cuDeviceGetName - 0x7fbf63d0de80 dlsym: cuCtxCreate_v3 - 0x7fbf63d0e120 dlsym: cuMemGetInfo_v2 - 0x7fbf63d0e8a0 dlsym: cuCtxDestroy - 0x7fbf63d6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T09:40:45.554+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:45.714+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T09:40:45.874+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T09:40:45.874+01:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=6.244668086 model=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e ```
GiteaMirror commented 2026-04-12 18:00:54 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 19, 2025):

time=2025-03-19T09:40:39.629+01:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - "

Set OLLAMA_LOAD_TIMEOUT=30m.

<!-- gh-comment-id:2735875986 --> @rick-github commented on GitHub (Mar 19, 2025): ``` time=2025-03-19T09:40:39.629+01:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - " ``` Set `OLLAMA_LOAD_TIMEOUT=30m`.
GiteaMirror commented 2026-04-12 18:00:55 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

Apologies, I was mixing up A40's with A100's. To be clear: I used 3 x A40's for this experiment.

<!-- gh-comment-id:2736090182 --> @phiwi commented on GitHub (Mar 19, 2025): Apologies, I was mixing up A40's with A100's. To be clear: I used 3 x A40's for this experiment.
GiteaMirror commented 2026-04-12 18:00:55 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 19, 2025):

No orphans were created. This log shows ollama starting a runner but the model load took too long and failed. Because the model load failed, there's no data that can be used to diagnose the original fault (slow inference). Extending the load timeout with OLLAMA_LOAD_TIMEOUT will allow the model to load and then the reasons for the slow inference might be uncovered.

The slow load is another issue. The model is being loaded from /home/pwiesenbac. Is the filesystem local disk, cloud storage like AWS, or some other network fs? What's the output of

dd if=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e of=/dev/null
<!-- gh-comment-id:2736149592 --> @rick-github commented on GitHub (Mar 19, 2025): No orphans were created. This log shows ollama starting a runner but the model load took too long and failed. Because the model load failed, there's no data that can be used to diagnose the original fault (slow inference). Extending the load timeout with `OLLAMA_LOAD_TIMEOUT` will allow the model to load and then the reasons for the slow inference might be uncovered. The slow load is another issue. The model is being loaded from `/home/pwiesenbac`. Is the filesystem local disk, cloud storage like AWS, or some other network fs? What's the output of ``` dd if=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e of=/dev/null ```
GiteaMirror commented 2026-04-12 18:00:56 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

Without having done the test (I will do it soon) the hdd is a distributed files system with an expected throuput of 100MB - 1,5GB/sec given a high enough blocksize.

Thank you for all your hints. I am now testing your proposals and will also load the original llama3.3:70B and the same settings, because for that very model inference was normal.

<!-- gh-comment-id:2736199309 --> @phiwi commented on GitHub (Mar 19, 2025): Without having done the test (I will do it soon) the hdd is a distributed files system with an expected throuput of 100MB - 1,5GB/sec given a high enough blocksize. Thank you for all your hints. I am now testing your proposals and will also load the original llama3.3:70B and the same settings, because for that very model inference was normal.
GiteaMirror commented 2026-04-12 18:00:58 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

The llama.3.3:70b (not finetuned, pulled directly from Ollama) loads fine:

2025/03/19 11:59:34 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,2,3 GPU_DEVICE_ORDINAL:0,2,3 HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:128000 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://10.250.135.153:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:240h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/pwiesenbach/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0,2,3 http_proxy: https_proxy: no_proxy:]"
time=2025-03-19T11:59:34.667+01:00 level=INFO source=images.go:432 msg="total blobs: 40"
time=2025-03-19T11:59:34.689+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-03-19T11:59:34.712+01:00 level=INFO source=routes.go:1277 msg="Listening on 10.250.135.153:11434 (version 0.5.13)"
time=2025-03-19T11:59:34.712+01:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler"
time=2025-03-19T11:59:34.712+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/beegfs/biosw/ollama/0.5.13/lib/ollama/libcuda.so* /biosw/ollama/0.5.13/lib/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-03-19T11:59:34.718+01:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7f147fd0de00
dlsym: cuDriverGetVersion - 0x7f147fd0de20
dlsym: cuDeviceGetCount - 0x7f147fd0de60
dlsym: cuDeviceGet - 0x7f147fd0de40
dlsym: cuDeviceGetAttribute - 0x7f147fd0df40
dlsym: cuDeviceGetUuid - 0x7f147fd0dea0
dlsym: cuDeviceGetName - 0x7f147fd0de80
dlsym: cuCtxCreate_v3 - 0x7f147fd0e120
dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0
dlsym: cuCtxDestroy - 0x7f147fd6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T11:59:35.076+01:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=3 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA totalMem 45499 mb
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA freeMem 45230 mb
[GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] Compute Capability 8.6
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA totalMem 45499 mb
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA freeMem 45230 mb
[GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] Compute Capability 8.6
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA totalMem 45499 mb
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA freeMem 45230 mb
[GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] Compute Capability 8.6
time=2025-03-19T11:59:35.605+01:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB"
[GIN] 2025/03/19 - 11:59:54 | 200 |     802.496µs |  10.250.135.104 | HEAD     "/"
[GIN] 2025/03/19 - 11:59:54 | 200 |  315.753326ms |  10.250.135.104 | POST     "/api/show"
time=2025-03-19T11:59:54.498+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.3 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.2 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7f147fd0de00
dlsym: cuDriverGetVersion - 0x7f147fd0de20
dlsym: cuDeviceGetCount - 0x7f147fd0de60
dlsym: cuDeviceGet - 0x7f147fd0de40
dlsym: cuDeviceGetAttribute - 0x7f147fd0df40
dlsym: cuDeviceGetUuid - 0x7f147fd0dea0
dlsym: cuDeviceGetName - 0x7f147fd0de80
dlsym: cuCtxCreate_v3 - 0x7f147fd0e120
dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0
dlsym: cuCtxDestroy - 0x7f147fd6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T11:59:54.682+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T11:59:54.858+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T11:59:55.029+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T11:59:55.029+01:00 level=DEBUG source=sched.go:182 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3
time=2025-03-19T11:59:55.088+01:00 level=DEBUG source=sched.go:225 msg="loading first model" model=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d
time=2025-03-19T11:59:55.088+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB"
time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.091+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.092+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]"
time=2025-03-19T11:59:55.093+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB"
time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers"
time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T11:59:55.095+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.2 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.2 GiB" now.free_swap="152.6 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7f147fd0de00
dlsym: cuDriverGetVersion - 0x7f147fd0de20
dlsym: cuDeviceGetCount - 0x7f147fd0de60
dlsym: cuDeviceGet - 0x7f147fd0de40
dlsym: cuDeviceGetAttribute - 0x7f147fd0df40
dlsym: cuDeviceGetUuid - 0x7f147fd0dea0
dlsym: cuDeviceGetName - 0x7f147fd0de80
dlsym: cuCtxCreate_v3 - 0x7f147fd0e120
dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0
dlsym: cuCtxDestroy - 0x7f147fd6c9f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 3
time=2025-03-19T11:59:55.257+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T11:59:55.417+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
time=2025-03-19T11:59:55.576+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB"
releasing cuda driver library
time=2025-03-19T11:59:55.577+01:00 level=INFO source=server.go:97 msg="system memory" total="503.6 GiB" free="481.2 GiB" free_swap="152.6 GiB"
time=2025-03-19T11:59:55.577+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]"
time=2025-03-19T11:59:55.577+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=80 layers.split=26,27,27 memory.available="[44.2 GiB 44.2 GiB 44.2 GiB]" memory.gpu_overhead="0 B" memory.required.full="131.7 GiB" memory.required.partial="130.9 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[43.7 GiB 43.6 GiB 43.6 GiB]" memory.weights.total="77.3 GiB" memory.weights.repeating="76.5 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="16.4 GiB" memory.graph.partial="16.4 GiB"
time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:259 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]"
time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:302 msg="adding gpu library" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12
time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12]
time=2025-03-19T11:59:55.581+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d --ctx-size 128000 --batch-size 512 --n-gpu-layers 80 --verbose --threads 16 --parallel 1 --tensor-split 26,27,27 --port 39243"
time=2025-03-19T11:59:55.581+01:00 level=DEBUG source=server.go:398 msg=subprocess environment="[ROCR_VISIBLE_DEVICES=0,2,3 CUDA_VISIBLE_DEVICES=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18,GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452,GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 GPU_DEVICE_ORDINAL=0,2,3 LD_LIBRARY_PATH=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/biosw/ollama/0.5.13/lib:/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/beegfs/biosw/ollama/0.5.13/lib/ollama PATH=.local/bin:/opt/singularity/bin:/biosw/ollama/0.5.13/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/home/pwiesenbach/bin:/home/pwiesenbach/.cargo/bin:.local/bin:/opt/singularity/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games]"
time=2025-03-19T11:59:55.589+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-19T11:59:55.591+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
time=2025-03-19T11:59:55.594+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
time=2025-03-19T11:59:55.670+01:00 level=INFO source=runner.go:931 msg="starting go runner"
time=2025-03-19T11:59:55.670+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 CUDA devices:
  Device 0: NVIDIA A40, compute capability 8.6, VMM: yes
  Device 1: NVIDIA A40, compute capability 8.6, VMM: yes
  Device 2: NVIDIA A40, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-03-19T11:59:56.821+01:00 level=DEBUG source=ggml.go:78 msg="skipping path which is not part of ollama" path=/biosw/ollama/0.5.13/lib
time=2025-03-19T11:59:56.821+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-sandybridge.so score: 20
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-skylakex.so score: 0
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so score: 55
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-alderlake.so score: 0
ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so score: 0
load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so
time=2025-03-19T11:59:56.861+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16
time=2025-03-19T11:59:56.862+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:39243"
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45230 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45230 MiB free
llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45230 MiB free
llama_model_loader: loaded meta data with 36 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.1 70B Instruct 2024 12
llama_model_loader: - kv   3:                            general.version str              = 2024-12
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Llama-3.1
llama_model_loader: - kv   6:                         general.size_label str              = 70B
llama_model_loader: - kv   7:                            general.license str              = llama3.1
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Llama 3.1 70B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Meta Llama
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/meta-llama/Lla...
llama_model_loader: - kv  12:                               general.tags arr[str,5]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv  13:                          general.languages arr[str,7]       = ["fr", "it", "pt", "hi", "es", "th", ...
llama_model_loader: - kv  14:                          llama.block_count u32              = 80
llama_model_loader: - kv  15:                       llama.context_length u32              = 131072
llama_model_loader: - kv  16:                     llama.embedding_length u32              = 8192
llama_model_loader: - kv  17:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv  18:                 llama.attention.head_count u32              = 64
llama_model_loader: - kv  19:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  20:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  21:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  22:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  23:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  24:                          general.file_type u32              = 15
llama_model_loader: - kv  25:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  26:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  27:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  29:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  30:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
time=2025-03-19T11:59:57.100+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: - kv  31:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  32:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  33:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  162 tensors
llama_model_loader: - type q4_K:  441 tensors
llama_model_loader: - type q5_K:   40 tensors
llama_model_loader: - type q6_K:   81 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 39.59 GiB (4.82 BPW) 
init_tokenizer: initializing tokenizer for type 2
load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG
load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG
load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG
load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
load: control token: 128007 '<|end_header_id|>' is not marked as EOG
load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG
load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
load: control token: 128001 '<|end_of_text|>' is not marked as EOG
load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
load: control token: 128006 '<|start_header_id|>' is not marked as EOG
load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
load: control token: 128000 '<|begin_of_text|>' is not marked as EOG
load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
load: control token: 128010 '<|python_tag|>' is not marked as EOG
load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG
load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 8192
print_info: n_layer          = 80
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 28672
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 70B
print_info: model params     = 70.55 B
print_info: general.name     = Llama 3.1 70B Instruct 2024 12
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer   0 assigned to device CUDA0
load_tensors: layer   1 assigned to device CUDA0
load_tensors: layer   2 assigned to device CUDA0
load_tensors: layer   3 assigned to device CUDA0
load_tensors: layer   4 assigned to device CUDA0
load_tensors: layer   5 assigned to device CUDA0
load_tensors: layer   6 assigned to device CUDA0
load_tensors: layer   7 assigned to device CUDA0
load_tensors: layer   8 assigned to device CUDA0
load_tensors: layer   9 assigned to device CUDA0
load_tensors: layer  10 assigned to device CUDA0
load_tensors: layer  11 assigned to device CUDA0
load_tensors: layer  12 assigned to device CUDA0
load_tensors: layer  13 assigned to device CUDA0
load_tensors: layer  14 assigned to device CUDA0
load_tensors: layer  15 assigned to device CUDA0
load_tensors: layer  16 assigned to device CUDA0
load_tensors: layer  17 assigned to device CUDA0
load_tensors: layer  18 assigned to device CUDA0
load_tensors: layer  19 assigned to device CUDA0
load_tensors: layer  20 assigned to device CUDA0
load_tensors: layer  21 assigned to device CUDA0
load_tensors: layer  22 assigned to device CUDA0
load_tensors: layer  23 assigned to device CUDA0
load_tensors: layer  24 assigned to device CUDA0
load_tensors: layer  25 assigned to device CUDA0
load_tensors: layer  26 assigned to device CUDA1
load_tensors: layer  27 assigned to device CUDA1
load_tensors: layer  28 assigned to device CUDA1
load_tensors: layer  29 assigned to device CUDA1
load_tensors: layer  30 assigned to device CUDA1
load_tensors: layer  31 assigned to device CUDA1
load_tensors: layer  32 assigned to device CUDA1
load_tensors: layer  33 assigned to device CUDA1
load_tensors: layer  34 assigned to device CUDA1
load_tensors: layer  35 assigned to device CUDA1
load_tensors: layer  36 assigned to device CUDA1
load_tensors: layer  37 assigned to device CUDA1
load_tensors: layer  38 assigned to device CUDA1
load_tensors: layer  39 assigned to device CUDA1
load_tensors: layer  40 assigned to device CUDA1
load_tensors: layer  41 assigned to device CUDA1
load_tensors: layer  42 assigned to device CUDA1
load_tensors: layer  43 assigned to device CUDA1
load_tensors: layer  44 assigned to device CUDA1
load_tensors: layer  45 assigned to device CUDA1
load_tensors: layer  46 assigned to device CUDA1
load_tensors: layer  47 assigned to device CUDA1
load_tensors: layer  48 assigned to device CUDA1
load_tensors: layer  49 assigned to device CUDA1
load_tensors: layer  50 assigned to device CUDA1
load_tensors: layer  51 assigned to device CUDA1
load_tensors: layer  52 assigned to device CUDA1
load_tensors: layer  53 assigned to device CUDA2
load_tensors: layer  54 assigned to device CUDA2
load_tensors: layer  55 assigned to device CUDA2
load_tensors: layer  56 assigned to device CUDA2
load_tensors: layer  57 assigned to device CUDA2
load_tensors: layer  58 assigned to device CUDA2
load_tensors: layer  59 assigned to device CUDA2
load_tensors: layer  60 assigned to device CUDA2
load_tensors: layer  61 assigned to device CUDA2
load_tensors: layer  62 assigned to device CUDA2
load_tensors: layer  63 assigned to device CUDA2
load_tensors: layer  64 assigned to device CUDA2
load_tensors: layer  65 assigned to device CUDA2
load_tensors: layer  66 assigned to device CUDA2
load_tensors: layer  67 assigned to device CUDA2
load_tensors: layer  68 assigned to device CUDA2
load_tensors: layer  69 assigned to device CUDA2
load_tensors: layer  70 assigned to device CUDA2
load_tensors: layer  71 assigned to device CUDA2
load_tensors: layer  72 assigned to device CUDA2
load_tensors: layer  73 assigned to device CUDA2
load_tensors: layer  74 assigned to device CUDA2
load_tensors: layer  75 assigned to device CUDA2
load_tensors: layer  76 assigned to device CUDA2
load_tensors: layer  77 assigned to device CUDA2
load_tensors: layer  78 assigned to device CUDA2
load_tensors: layer  79 assigned to device CUDA2
load_tensors: layer  80 assigned to device CPU
load_tensors: tensor 'token_embd.weight' (q4_K) (and 2 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
time=2025-03-19T12:02:52.693+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
time=2025-03-19T12:02:52.944+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
load_tensors: offloading 80 repeating layers to GPU
load_tensors: offloaded 80/81 layers to GPU
load_tensors:        CUDA0 model buffer size = 12843.81 MiB
load_tensors:        CUDA1 model buffer size = 12951.00 MiB
load_tensors:        CUDA2 model buffer size = 13362.69 MiB
load_tensors:   CPU_Mapped model buffer size = 40543.11 MiB
time=2025-03-19T12:03:22.291+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.04"
time=2025-03-19T12:03:22.542+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.09"
time=2025-03-19T12:03:22.793+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.13"
time=2025-03-19T12:03:23.044+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.18"
time=2025-03-19T12:03:23.295+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.23"
time=2025-03-19T12:03:23.546+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.28"
time=2025-03-19T12:03:23.797+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.33"
time=2025-03-19T12:03:24.048+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.37"
time=2025-03-19T12:03:24.299+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.42"
time=2025-03-19T12:03:24.550+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.47"
time=2025-03-19T12:03:24.801+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.52"
time=2025-03-19T12:03:25.051+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.57"
time=2025-03-19T12:03:25.302+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.61"
time=2025-03-19T12:03:25.553+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.66"
time=2025-03-19T12:03:25.804+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.71"
time=2025-03-19T12:03:26.055+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.76"
time=2025-03-19T12:03:26.306+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.80"
time=2025-03-19T12:03:26.557+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.85"
time=2025-03-19T12:03:26.808+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.89"
time=2025-03-19T12:03:27.059+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.95"
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 128000
llama_init_from_model: n_ctx_per_seq = 128000
llama_init_from_model: n_batch       = 512
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 500000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_per_seq (128000) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 128000, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80, can_shift = 1
llama_kv_cache_init: layer 0: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 1: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 2: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 3: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 4: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 5: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 6: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 7: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 8: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 9: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 10: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 11: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 12: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 13: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 14: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 15: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 16: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 17: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 18: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 19: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 20: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 21: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 22: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 23: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 24: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 25: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 26: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 27: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 28: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 29: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 30: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 31: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 32: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 33: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 34: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 35: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 36: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 37: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 38: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 39: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 40: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 41: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 42: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 43: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 44: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 45: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 46: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 47: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 48: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 49: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 50: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 51: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 52: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 53: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 54: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 55: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 56: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 57: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 58: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 59: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 60: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 61: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 62: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 63: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 64: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 65: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 66: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 67: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 68: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 69: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 70: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 71: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 72: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 73: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 74: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 75: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 76: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 77: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 78: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 79: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init:      CUDA0 KV buffer size = 13000.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size = 13500.00 MiB
llama_kv_cache_init:      CUDA2 KV buffer size = 13500.00 MiB
llama_init_from_model: KV self size  = 40000.00 MiB, K (f16): 20000.00 MiB, V (f16): 20000.00 MiB
llama_init_from_model:        CPU  output buffer size =     0.52 MiB
time=2025-03-19T12:03:27.310+01:00 level=DEBUG source=server.go:602 msg="model load progress 1.00"
llama_init_from_model:      CUDA0 compute buffer size = 16314.00 MiB
llama_init_from_model:      CUDA1 compute buffer size = 16314.00 MiB
llama_init_from_model:      CUDA2 compute buffer size = 16314.00 MiB
llama_init_from_model:  CUDA_Host compute buffer size =   266.01 MiB
llama_init_from_model: graph nodes  = 2566
llama_init_from_model: graph splits = 6 (with bs=512), 5 (with bs=1)
time=2025-03-19T12:03:27.561+01:00 level=INFO source=server.go:596 msg="llama runner started in 211.97 seconds"
time=2025-03-19T12:03:27.562+01:00 level=DEBUG source=sched.go:463 msg="finished setting up runner" model=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d
[GIN] 2025/03/19 - 12:03:27 | 200 |         3m33s |  10.250.135.104 | POST     "/api/generate"
time=2025-03-19T12:03:27.564+01:00 level=DEBUG source=sched.go:467 msg="context for request finished"
time=2025-03-19T12:03:27.564+01:00 level=DEBUG source=sched.go:340 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d duration=240h0m0s
time=2025-03-19T12:03:27.565+01:00 level=DEBUG source=sched.go:358 msg="after processing request finished event" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d refCount=0
<!-- gh-comment-id:2736237798 --> @phiwi commented on GitHub (Mar 19, 2025): The llama.3.3:70b (not finetuned, pulled directly from Ollama) loads fine: ```bash 2025/03/19 11:59:34 routes.go:1215: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,2,3 GPU_DEVICE_ORDINAL:0,2,3 HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:128000 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://10.250.135.153:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:240h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/pwiesenbach/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0,2,3 http_proxy: https_proxy: no_proxy:]" time=2025-03-19T11:59:34.667+01:00 level=INFO source=images.go:432 msg="total blobs: 40" time=2025-03-19T11:59:34.689+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-03-19T11:59:34.712+01:00 level=INFO source=routes.go:1277 msg="Listening on 10.250.135.153:11434 (version 0.5.13)" time=2025-03-19T11:59:34.712+01:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler" time=2025-03-19T11:59:34.712+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-03-19T11:59:34.713+01:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/beegfs/biosw/ollama/0.5.13/lib/ollama/libcuda.so* /biosw/ollama/0.5.13/lib/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-03-19T11:59:34.718+01:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7f147fd0de00 dlsym: cuDriverGetVersion - 0x7f147fd0de20 dlsym: cuDeviceGetCount - 0x7f147fd0de60 dlsym: cuDeviceGet - 0x7f147fd0de40 dlsym: cuDeviceGetAttribute - 0x7f147fd0df40 dlsym: cuDeviceGetUuid - 0x7f147fd0dea0 dlsym: cuDeviceGetName - 0x7f147fd0de80 dlsym: cuCtxCreate_v3 - 0x7f147fd0e120 dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0 dlsym: cuCtxDestroy - 0x7f147fd6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T11:59:35.076+01:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=3 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA totalMem 45499 mb [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] CUDA freeMem 45230 mb [GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18] Compute Capability 8.6 [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA totalMem 45499 mb [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] CUDA freeMem 45230 mb [GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452] Compute Capability 8.6 [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA totalMem 45499 mb [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] CUDA freeMem 45230 mb [GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590] Compute Capability 8.6 time=2025-03-19T11:59:35.605+01:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" time=2025-03-19T11:59:35.605+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" [GIN] 2025/03/19 - 11:59:54 | 200 | 802.496µs | 10.250.135.104 | HEAD "/" [GIN] 2025/03/19 - 11:59:54 | 200 | 315.753326ms | 10.250.135.104 | POST "/api/show" time=2025-03-19T11:59:54.498+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.3 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.2 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7f147fd0de00 dlsym: cuDriverGetVersion - 0x7f147fd0de20 dlsym: cuDeviceGetCount - 0x7f147fd0de60 dlsym: cuDeviceGet - 0x7f147fd0de40 dlsym: cuDeviceGetAttribute - 0x7f147fd0df40 dlsym: cuDeviceGetUuid - 0x7f147fd0dea0 dlsym: cuDeviceGetName - 0x7f147fd0de80 dlsym: cuCtxCreate_v3 - 0x7f147fd0e120 dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0 dlsym: cuCtxDestroy - 0x7f147fd6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T11:59:54.682+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T11:59:54.858+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T11:59:55.029+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T11:59:55.029+01:00 level=DEBUG source=sched.go:182 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3 time=2025-03-19T11:59:55.088+01:00 level=DEBUG source=sched.go:225 msg="loading first model" model=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d time=2025-03-19T11:59:55.088+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.089+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="63.5 GiB" time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T11:59:55.090+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.091+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.092+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[44.2 GiB]" time=2025-03-19T11:59:55.093+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:185 msg="gpu has too little memory to allocate any layers" id=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA A40" total="44.4 GiB" available="44.2 GiB" minimum_memory=479199232 layer_size="2.5 GiB" gpu_zer_overhead="0 B" partial_offload="65.5 GiB" full_offload="65.5 GiB" time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:329 msg="insufficient VRAM to load any model layers" time=2025-03-19T11:59:55.094+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T11:59:55.095+01:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="503.6 GiB" before.free="481.2 GiB" before.free_swap="152.6 GiB" now.total="503.6 GiB" now.free="481.2 GiB" now.free_swap="152.6 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7f147fd0de00 dlsym: cuDriverGetVersion - 0x7f147fd0de20 dlsym: cuDeviceGetCount - 0x7f147fd0de60 dlsym: cuDeviceGet - 0x7f147fd0de40 dlsym: cuDeviceGetAttribute - 0x7f147fd0df40 dlsym: cuDeviceGetUuid - 0x7f147fd0dea0 dlsym: cuDeviceGetName - 0x7f147fd0de80 dlsym: cuCtxCreate_v3 - 0x7f147fd0e120 dlsym: cuMemGetInfo_v2 - 0x7f147fd0e8a0 dlsym: cuCtxDestroy - 0x7f147fd6c9f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 3 time=2025-03-19T11:59:55.257+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T11:59:55.417+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" time=2025-03-19T11:59:55.576+01:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 name="NVIDIA A40" overhead="0 B" before.total="44.4 GiB" before.free="44.2 GiB" now.total="44.4 GiB" now.free="44.2 GiB" now.used="268.4 MiB" releasing cuda driver library time=2025-03-19T11:59:55.577+01:00 level=INFO source=server.go:97 msg="system memory" total="503.6 GiB" free="481.2 GiB" free_swap="152.6 GiB" time=2025-03-19T11:59:55.577+01:00 level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=3 available="[44.2 GiB 44.2 GiB 44.2 GiB]" time=2025-03-19T11:59:55.577+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=80 layers.split=26,27,27 memory.available="[44.2 GiB 44.2 GiB 44.2 GiB]" memory.gpu_overhead="0 B" memory.required.full="131.7 GiB" memory.required.partial="130.9 GiB" memory.required.kv="39.1 GiB" memory.required.allocations="[43.7 GiB 43.6 GiB 43.6 GiB]" memory.weights.total="77.3 GiB" memory.weights.repeating="76.5 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="16.4 GiB" memory.graph.partial="16.4 GiB" time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:259 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:302 msg="adding gpu library" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12 time=2025-03-19T11:59:55.580+01:00 level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12] time=2025-03-19T11:59:55.581+01:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/beegfs/biosw/ollama/0.5.13/bin/ollama runner --model /home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d --ctx-size 128000 --batch-size 512 --n-gpu-layers 80 --verbose --threads 16 --parallel 1 --tensor-split 26,27,27 --port 39243" time=2025-03-19T11:59:55.581+01:00 level=DEBUG source=server.go:398 msg=subprocess environment="[ROCR_VISIBLE_DEVICES=0,2,3 CUDA_VISIBLE_DEVICES=GPU-2f2b88b7-dca2-5058-5b25-e203ebd8ed18,GPU-96396d7c-d1dc-bbeb-0c2e-3501e00b0452,GPU-950c5c3f-58dd-edd8-7da1-3c5778a1d590 GPU_DEVICE_ORDINAL=0,2,3 LD_LIBRARY_PATH=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/biosw/ollama/0.5.13/lib:/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12:/beegfs/biosw/ollama/0.5.13/lib/ollama PATH=.local/bin:/opt/singularity/bin:/biosw/ollama/0.5.13/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/home/pwiesenbach/bin:/home/pwiesenbach/.cargo/bin:.local/bin:/opt/singularity/bin:/biosw/golang/latest/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games]" time=2025-03-19T11:59:55.589+01:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-19T11:59:55.591+01:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-03-19T11:59:55.594+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-03-19T11:59:55.670+01:00 level=INFO source=runner.go:931 msg="starting go runner" time=2025-03-19T11:59:55.670+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 3 CUDA devices: Device 0: NVIDIA A40, compute capability 8.6, VMM: yes Device 1: NVIDIA A40, compute capability 8.6, VMM: yes Device 2: NVIDIA A40, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/cuda_v12/libggml-cuda.so time=2025-03-19T11:59:56.821+01:00 level=DEBUG source=ggml.go:78 msg="skipping path which is not part of ollama" path=/biosw/ollama/0.5.13/lib time=2025-03-19T11:59:56.821+01:00 level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/beegfs/biosw/ollama/0.5.13/lib/ollama ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-sandybridge.so score: 20 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-skylakex.so score: 0 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so score: 55 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-alderlake.so score: 0 ggml_backend_load_best: /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-icelake.so score: 0 load_backend: loaded CPU backend from /beegfs/biosw/ollama/0.5.13/lib/ollama/libggml-cpu-haswell.so time=2025-03-19T11:59:56.861+01:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16 time=2025-03-19T11:59:56.862+01:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:39243" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A40) - 45230 MiB free llama_model_load_from_file_impl: using device CUDA1 (NVIDIA A40) - 45230 MiB free llama_model_load_from_file_impl: using device CUDA2 (NVIDIA A40) - 45230 MiB free llama_model_loader: loaded meta data with 36 key-value pairs and 724 tensors from /home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Llama 3.1 70B Instruct 2024 12 llama_model_loader: - kv 3: general.version str = 2024-12 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Llama-3.1 llama_model_loader: - kv 6: general.size_label str = 70B llama_model_loader: - kv 7: general.license str = llama3.1 llama_model_loader: - kv 8: general.base_model.count u32 = 1 llama_model_loader: - kv 9: general.base_model.0.name str = Llama 3.1 70B llama_model_loader: - kv 10: general.base_model.0.organization str = Meta Llama llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla... llama_model_loader: - kv 12: general.tags arr[str,5] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 13: general.languages arr[str,7] = ["fr", "it", "pt", "hi", "es", "th", ... llama_model_loader: - kv 14: llama.block_count u32 = 80 llama_model_loader: - kv 15: llama.context_length u32 = 131072 llama_model_loader: - kv 16: llama.embedding_length u32 = 8192 llama_model_loader: - kv 17: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 18: llama.attention.head_count u32 = 64 llama_model_loader: - kv 19: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 20: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 21: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 22: llama.attention.key_length u32 = 128 llama_model_loader: - kv 23: llama.attention.value_length u32 = 128 llama_model_loader: - kv 24: general.file_type u32 = 15 llama_model_loader: - kv 25: llama.vocab_size u32 = 128256 llama_model_loader: - kv 26: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 28: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... time=2025-03-19T11:59:57.100+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 32: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 33: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 34: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 35: general.quantization_version u32 = 2 llama_model_loader: - type f32: 162 tensors llama_model_loader: - type q4_K: 441 tensors llama_model_loader: - type q5_K: 40 tensors llama_model_loader: - type q6_K: 81 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 39.59 GiB (4.82 BPW) init_tokenizer: initializing tokenizer for type 2 load: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG load: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG load: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG load: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG load: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG load: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG load: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG load: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG load: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG load: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG load: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG load: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG load: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG load: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG load: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG load: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG load: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG load: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG load: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG load: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG load: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG load: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG load: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG load: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG load: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG load: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG load: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG load: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG load: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG load: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG load: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG load: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG load: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG load: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG load: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG load: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG load: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG load: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG load: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG load: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG load: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG load: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG load: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG load: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG load: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG load: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG load: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG load: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG load: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG load: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG load: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG load: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG load: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG load: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG load: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG load: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG load: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG load: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG load: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG load: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG load: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG load: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG load: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG load: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG load: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG load: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG load: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG load: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG load: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG load: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG load: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG load: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG load: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG load: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG load: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG load: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG load: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG load: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG load: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG load: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG load: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG load: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG load: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG load: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG load: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG load: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG load: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG load: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG load: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG load: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG load: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG load: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG load: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG load: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG load: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG load: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG load: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG load: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG load: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG load: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG load: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG load: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG load: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG load: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG load: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG load: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG load: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG load: control token: 128015 '<|reserved_special_token_7|>' is not marked as EOG load: control token: 128013 '<|reserved_special_token_5|>' is not marked as EOG load: control token: 128011 '<|reserved_special_token_3|>' is not marked as EOG load: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG load: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG load: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG load: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG load: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG load: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG load: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG load: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG load: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG load: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG load: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG load: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG load: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG load: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG load: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG load: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG load: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG load: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG load: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG load: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG load: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG load: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG load: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG load: control token: 128007 '<|end_header_id|>' is not marked as EOG load: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG load: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG load: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG load: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG load: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG load: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG load: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG load: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG load: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG load: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG load: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG load: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG load: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG load: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG load: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG load: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG load: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG load: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG load: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG load: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG load: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG load: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG load: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG load: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG load: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG load: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG load: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG load: control token: 128012 '<|reserved_special_token_4|>' is not marked as EOG load: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG load: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG load: control token: 128001 '<|end_of_text|>' is not marked as EOG load: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG load: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG load: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG load: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG load: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG load: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG load: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG load: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG load: control token: 128006 '<|start_header_id|>' is not marked as EOG load: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG load: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG load: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG load: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG load: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG load: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG load: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG load: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG load: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG load: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG load: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG load: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG load: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG load: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG load: control token: 128000 '<|begin_of_text|>' is not marked as EOG load: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG load: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG load: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG load: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG load: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG load: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG load: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG load: control token: 128010 '<|python_tag|>' is not marked as EOG load: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG load: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG load: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG load: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG load: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG load: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG load: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG load: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG load: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG load: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG load: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG load: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG load: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG load: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG load: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG load: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG load: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG load: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG load: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG load: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG load: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG load: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG load: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG load: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG load: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG load: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG load: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG load: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG load: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG load: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG load: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG load: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG load: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG load: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG load: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG load: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG load: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG load: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG load: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG load: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG load: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG load: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG load: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG load: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG load: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG load: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG load: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG load: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG load: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG load: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG load: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG load: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG load: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG load: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG load: control token: 128014 '<|reserved_special_token_6|>' is not marked as EOG load: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG load: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 8192 print_info: n_layer = 80 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: n_ff = 28672 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = linear print_info: freq_base_train = 500000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 131072 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 70B print_info: model params = 70.55 B print_info: general.name = Llama 3.1 70B Instruct 2024 12 print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: layer 0 assigned to device CUDA0 load_tensors: layer 1 assigned to device CUDA0 load_tensors: layer 2 assigned to device CUDA0 load_tensors: layer 3 assigned to device CUDA0 load_tensors: layer 4 assigned to device CUDA0 load_tensors: layer 5 assigned to device CUDA0 load_tensors: layer 6 assigned to device CUDA0 load_tensors: layer 7 assigned to device CUDA0 load_tensors: layer 8 assigned to device CUDA0 load_tensors: layer 9 assigned to device CUDA0 load_tensors: layer 10 assigned to device CUDA0 load_tensors: layer 11 assigned to device CUDA0 load_tensors: layer 12 assigned to device CUDA0 load_tensors: layer 13 assigned to device CUDA0 load_tensors: layer 14 assigned to device CUDA0 load_tensors: layer 15 assigned to device CUDA0 load_tensors: layer 16 assigned to device CUDA0 load_tensors: layer 17 assigned to device CUDA0 load_tensors: layer 18 assigned to device CUDA0 load_tensors: layer 19 assigned to device CUDA0 load_tensors: layer 20 assigned to device CUDA0 load_tensors: layer 21 assigned to device CUDA0 load_tensors: layer 22 assigned to device CUDA0 load_tensors: layer 23 assigned to device CUDA0 load_tensors: layer 24 assigned to device CUDA0 load_tensors: layer 25 assigned to device CUDA0 load_tensors: layer 26 assigned to device CUDA1 load_tensors: layer 27 assigned to device CUDA1 load_tensors: layer 28 assigned to device CUDA1 load_tensors: layer 29 assigned to device CUDA1 load_tensors: layer 30 assigned to device CUDA1 load_tensors: layer 31 assigned to device CUDA1 load_tensors: layer 32 assigned to device CUDA1 load_tensors: layer 33 assigned to device CUDA1 load_tensors: layer 34 assigned to device CUDA1 load_tensors: layer 35 assigned to device CUDA1 load_tensors: layer 36 assigned to device CUDA1 load_tensors: layer 37 assigned to device CUDA1 load_tensors: layer 38 assigned to device CUDA1 load_tensors: layer 39 assigned to device CUDA1 load_tensors: layer 40 assigned to device CUDA1 load_tensors: layer 41 assigned to device CUDA1 load_tensors: layer 42 assigned to device CUDA1 load_tensors: layer 43 assigned to device CUDA1 load_tensors: layer 44 assigned to device CUDA1 load_tensors: layer 45 assigned to device CUDA1 load_tensors: layer 46 assigned to device CUDA1 load_tensors: layer 47 assigned to device CUDA1 load_tensors: layer 48 assigned to device CUDA1 load_tensors: layer 49 assigned to device CUDA1 load_tensors: layer 50 assigned to device CUDA1 load_tensors: layer 51 assigned to device CUDA1 load_tensors: layer 52 assigned to device CUDA1 load_tensors: layer 53 assigned to device CUDA2 load_tensors: layer 54 assigned to device CUDA2 load_tensors: layer 55 assigned to device CUDA2 load_tensors: layer 56 assigned to device CUDA2 load_tensors: layer 57 assigned to device CUDA2 load_tensors: layer 58 assigned to device CUDA2 load_tensors: layer 59 assigned to device CUDA2 load_tensors: layer 60 assigned to device CUDA2 load_tensors: layer 61 assigned to device CUDA2 load_tensors: layer 62 assigned to device CUDA2 load_tensors: layer 63 assigned to device CUDA2 load_tensors: layer 64 assigned to device CUDA2 load_tensors: layer 65 assigned to device CUDA2 load_tensors: layer 66 assigned to device CUDA2 load_tensors: layer 67 assigned to device CUDA2 load_tensors: layer 68 assigned to device CUDA2 load_tensors: layer 69 assigned to device CUDA2 load_tensors: layer 70 assigned to device CUDA2 load_tensors: layer 71 assigned to device CUDA2 load_tensors: layer 72 assigned to device CUDA2 load_tensors: layer 73 assigned to device CUDA2 load_tensors: layer 74 assigned to device CUDA2 load_tensors: layer 75 assigned to device CUDA2 load_tensors: layer 76 assigned to device CUDA2 load_tensors: layer 77 assigned to device CUDA2 load_tensors: layer 78 assigned to device CUDA2 load_tensors: layer 79 assigned to device CUDA2 load_tensors: layer 80 assigned to device CPU load_tensors: tensor 'token_embd.weight' (q4_K) (and 2 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead time=2025-03-19T12:02:52.693+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" time=2025-03-19T12:02:52.944+01:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" load_tensors: offloading 80 repeating layers to GPU load_tensors: offloaded 80/81 layers to GPU load_tensors: CUDA0 model buffer size = 12843.81 MiB load_tensors: CUDA1 model buffer size = 12951.00 MiB load_tensors: CUDA2 model buffer size = 13362.69 MiB load_tensors: CPU_Mapped model buffer size = 40543.11 MiB time=2025-03-19T12:03:22.291+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.04" time=2025-03-19T12:03:22.542+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.09" time=2025-03-19T12:03:22.793+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.13" time=2025-03-19T12:03:23.044+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.18" time=2025-03-19T12:03:23.295+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.23" time=2025-03-19T12:03:23.546+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.28" time=2025-03-19T12:03:23.797+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.33" time=2025-03-19T12:03:24.048+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.37" time=2025-03-19T12:03:24.299+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.42" time=2025-03-19T12:03:24.550+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.47" time=2025-03-19T12:03:24.801+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.52" time=2025-03-19T12:03:25.051+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.57" time=2025-03-19T12:03:25.302+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.61" time=2025-03-19T12:03:25.553+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.66" time=2025-03-19T12:03:25.804+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.71" time=2025-03-19T12:03:26.055+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.76" time=2025-03-19T12:03:26.306+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.80" time=2025-03-19T12:03:26.557+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.85" time=2025-03-19T12:03:26.808+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.89" time=2025-03-19T12:03:27.059+01:00 level=DEBUG source=server.go:602 msg="model load progress 0.95" llama_init_from_model: n_seq_max = 1 llama_init_from_model: n_ctx = 128000 llama_init_from_model: n_ctx_per_seq = 128000 llama_init_from_model: n_batch = 512 llama_init_from_model: n_ubatch = 512 llama_init_from_model: flash_attn = 0 llama_init_from_model: freq_base = 500000.0 llama_init_from_model: freq_scale = 1 llama_init_from_model: n_ctx_per_seq (128000) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 128000, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80, can_shift = 1 llama_kv_cache_init: layer 0: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 1: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 2: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 3: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 4: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 5: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 6: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 7: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 8: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 9: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 10: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 11: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 12: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 13: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 14: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 15: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 16: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 17: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 18: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 19: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 20: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 21: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 22: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 23: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 24: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 25: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 26: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 27: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 28: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 29: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 30: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 31: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 32: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 33: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 34: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 35: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 36: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 37: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 38: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 39: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 40: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 41: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 42: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 43: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 44: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 45: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 46: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 47: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 48: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 49: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 50: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 51: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 52: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 53: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 54: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 55: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 56: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 57: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 58: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 59: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 60: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 61: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 62: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 63: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 64: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 65: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 66: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 67: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 68: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 69: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 70: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 71: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 72: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 73: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 74: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 75: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 76: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 77: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 78: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 79: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: CUDA0 KV buffer size = 13000.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 13500.00 MiB llama_kv_cache_init: CUDA2 KV buffer size = 13500.00 MiB llama_init_from_model: KV self size = 40000.00 MiB, K (f16): 20000.00 MiB, V (f16): 20000.00 MiB llama_init_from_model: CPU output buffer size = 0.52 MiB time=2025-03-19T12:03:27.310+01:00 level=DEBUG source=server.go:602 msg="model load progress 1.00" llama_init_from_model: CUDA0 compute buffer size = 16314.00 MiB llama_init_from_model: CUDA1 compute buffer size = 16314.00 MiB llama_init_from_model: CUDA2 compute buffer size = 16314.00 MiB llama_init_from_model: CUDA_Host compute buffer size = 266.01 MiB llama_init_from_model: graph nodes = 2566 llama_init_from_model: graph splits = 6 (with bs=512), 5 (with bs=1) time=2025-03-19T12:03:27.561+01:00 level=INFO source=server.go:596 msg="llama runner started in 211.97 seconds" time=2025-03-19T12:03:27.562+01:00 level=DEBUG source=sched.go:463 msg="finished setting up runner" model=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d [GIN] 2025/03/19 - 12:03:27 | 200 | 3m33s | 10.250.135.104 | POST "/api/generate" time=2025-03-19T12:03:27.564+01:00 level=DEBUG source=sched.go:467 msg="context for request finished" time=2025-03-19T12:03:27.564+01:00 level=DEBUG source=sched.go:340 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d duration=240h0m0s time=2025-03-19T12:03:27.565+01:00 level=DEBUG source=sched.go:358 msg="after processing request finished event" modelPath=/home/pwiesenbach/.ollama/models/blobs/sha256-4824460d29f2058aaf6e1118a63a7a197a09bed509f0e7d4e2efb1ee273b447d refCount=0 ```
GiteaMirror commented 2026-04-12 18:00:59 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

result for the dd test:

dd if=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e of=/dev/null bs=1024k
71501+1 records in
71501+1 records out
74975049856 bytes (75 GB, 70 GiB) copied, 307.002 s, 244 MB/s
<!-- gh-comment-id:2736241583 --> @phiwi commented on GitHub (Mar 19, 2025): result for the `dd` test: ```bash dd if=/home/pwiesenbach/.ollama/models/blobs/sha256-72e17486f230316a5f9270ad4e2a9e1bf5b25816583a7f36be1b2a7ddd4dfb7e of=/dev/null bs=1024k 71501+1 records in 71501+1 records out 74975049856 bytes (75 GB, 70 GiB) copied, 307.002 s, 244 MB/s ```
GiteaMirror commented 2026-04-12 18:01:01 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 19, 2025):

74975049856 bytes (75 GB, 70 GiB) copied, 307.002 s, 244 MB/s

5 minutes and 7 seconds to read the raw model off disk. The default load timeout is 5 minutes.

<!-- gh-comment-id:2736256441 --> @rick-github commented on GitHub (Mar 19, 2025): > 74975049856 bytes (75 GB, 70 GiB) copied, 307.002 s, 244 MB/s 5 minutes and 7 seconds to read the raw model off disk. The default load timeout is 5 minutes.
GiteaMirror commented 2026-04-12 18:01:02 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

I set OLLAMA_TIMEOUT=30 and seemingly it worked. Let me repeat the process one more time.

<!-- gh-comment-id:2736288982 --> @phiwi commented on GitHub (Mar 19, 2025): I set `OLLAMA_TIMEOUT=30` and seemingly it worked. Let me repeat the process one more time.
GiteaMirror commented 2026-04-12 18:01:04 -05:00
Author
Owner
Copy Link

@phiwi commented on GitHub (Mar 19, 2025):

I will close it for now as I at least found the problem for not even loading the model (timout). Now, I need to wait for resources being freed up again to check for inference speed and test all your hints.

Thanks a bunch guys - this was great help!

<!-- gh-comment-id:2736552720 --> @phiwi commented on GitHub (Mar 19, 2025): I will close it for now as I at least found the problem for not even loading the model (timout). Now, I need to wait for resources being freed up again to check for inference speed and test all your hints. Thanks a bunch guys - this was great help!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6455
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?