[GH-ISSUE #12247] using embeddinggemma unloads other loaded llm models, even if there is VRAM available #33907

New Issue

Closed

opened 2026-04-22 17:05:26 -05:00 by GiteaMirror · 4 comments

GiteaMirror commented 2026-04-22 17:05:26 -05:00

Owner

Copy Link

Originally created by @tomaszbk on GitHub (Sep 11, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/12247

What is the issue?

tried embeddinggemma with qwen 2.5vl and llama 3.1, on 8GB VRAM GPU, and the llm models are unloaded upon an embedding request.
using granite embedding 300m doesnt unload the other models

Relevant log output

time=2025-09-11T01:28:15.420Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="2.3 GiB"
time=2025-09-11T01:28:15.553Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 --port 33085"
time=2025-09-11T01:28:15.564Z level=INFO source=runner.go:1251 msg="starting ollama engine"
time=2025-09-11T01:28:15.566Z level=INFO source=runner.go:1286 msg="Server listening on 127.0.0.1:33085"
time=2025-09-11T01:28:15.645Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.5 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:15.645Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B"
time=2025-09-11T01:28:19.869Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.098100875 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:20.119Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.348006231 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:20.185Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.9 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:20.185Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 library=cuda parallel=1 required="3.1 GiB" gpus=1
time=2025-09-11T01:28:20.185Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=25 layers.offload=25 layers.split=[25] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.1 GiB" memory.required.partial="3.1 GiB" memory.required.kv="58.0 MiB" memory.required.allocations="[3.1 GiB]" memory.weights.total="577.8 MiB" memory.weights.repeating="193.8 MiB" memory.weights.nonrepeating="384.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.2 GiB"
time=2025-09-11T01:28:20.187Z level=INFO source=runner.go:1170 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:2048 FlashAttention:false KvSize:2048 KvCacheType: NumThreads:8 GPULayers:25[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-09-11T01:28:20.207Z level=INFO source=ggml.go:131 msg="" architecture=gemma3 file_type=BF16 name="Embeddinggemma 300M" description="" num_tensors=316 num_key_values=37
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
time=2025-09-11T01:28:20.266Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-09-11T01:28:20.368Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.597301181 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:487 msg="offloading 24 repeating layers to GPU"
time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:493 msg="offloading output layer to GPU"
time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:498 msg="offloaded 25/25 layers to GPU"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="586.8 MiB"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="384.0 MiB"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="58.0 MiB"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="108.0 MiB"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="6.0 MiB"
time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:342 msg="total memory" size="1.1 GiB"
time=2025-09-11T01:28:20.391Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-09-11T01:28:20.391Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding"
time=2025-09-11T01:28:20.391Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model"
time=2025-09-11T01:28:20.895Z level=INFO source=server.go:1288 msg="llama runner started in 6.22 seconds"
[GIN] 2025/09/11 - 01:28:21 | 200 |  6.694510783s |      172.18.0.6 | POST     "/v1/embeddings"
time=2025-09-11T01:28:21.232Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="4.9 GiB"
llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = llama3.1
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 32
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.58 GiB (4.89 BPW) 
load: printing all EOG tokens:
load:   - 128001 ('<|end_of_text|>')
load:   - 128008 ('<|eom_id|>')
load:   - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 1
print_info: model type       = ?B
print_info: model params     = 8.03 B
print_info: general.name     = Meta Llama 3.1 8B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-09-11T01:28:21.538Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --model /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 --port 45399"
time=2025-09-11T01:28:21.550Z level=INFO source=runner.go:864 msg="starting go runner"
time=2025-09-11T01:28:21.631Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.2 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:21.632Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
time=2025-09-11T01:28:21.650Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-09-11T01:28:21.653Z level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:45399"
time=2025-09-11T01:28:26.720Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.087912251 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
time=2025-09-11T01:28:26.970Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.337829115 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
time=2025-09-11T01:28:27.030Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.8 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:27.030Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 library=cuda parallel=1 required="5.7 GiB" gpus=1
time=2025-09-11T01:28:27.030Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split=[33] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.7 GiB" memory.required.partial="5.7 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.7 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="677.5 MiB"
time=2025-09-11T01:28:27.031Z level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:8 GPULayers:33[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}"
time=2025-09-11T01:28:27.031Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding"
time=2025-09-11T01:28:27.032Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model"
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3070) - 7098 MiB free
llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = llama3.1
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 32
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.58 GiB (4.89 BPW) 
time=2025-09-11T01:28:27.220Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.5878821720000005 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
load: printing all EOG tokens:
load:   - 128001 ('<|end_of_text|>')
load:   - 128008 ('<|eom_id|>')
load:   - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 4096
print_info: n_layer          = 32
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 14336
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.03 B
print_info: general.name     = Meta Llama 3.1 8B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 32 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 33/33 layers to GPU
load_tensors:        CUDA0 model buffer size =  4403.49 MiB
load_tensors:   CPU_Mapped model buffer size =   281.81 MiB
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = false
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache_unified:      CUDA0 KV buffer size =   512.00 MiB
llama_kv_cache_unified: size =  512.00 MiB (  4096 cells,  32 layers,  1/1 seqs), K (f16):  256.00 MiB, V (f16):  256.00 MiB
llama_context:      CUDA0 compute buffer size =   300.01 MiB
llama_context:  CUDA_Host compute buffer size =    20.01 MiB
llama_context: graph nodes  = 1126
llama_context: graph splits = 2
time=2025-09-11T01:28:28.036Z level=INFO source=server.go:1288 msg="llama runner started in 6.50 seconds"
time=2025-09-11T01:28:28.036Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-09-11T01:28:28.036Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding"
time=2025-09-11T01:28:28.037Z level=INFO source=server.go:1288 msg="llama runner started in 6.50 seconds"
[GIN] 2025/09/11 - 01:28:28 | 200 |  7.589501463s |      172.18.0.6 | POST     "/v1/chat/completions"
[GIN] 2025/09/11 - 01:28:28 | 200 |  343.092569ms |      172.18.0.6 | POST     "/v1/chat/completions"
time=2025-09-11T01:28:29.170Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="2.3 GiB"
time=2025-09-11T01:28:29.309Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 --port 35901"
time=2025-09-11T01:28:29.321Z level=INFO source=runner.go:1251 msg="starting ollama engine"
time=2025-09-11T01:28:29.324Z level=INFO source=runner.go:1286 msg="Server listening on 127.0.0.1:35901"
time=2025-09-11T01:28:29.407Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.4 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:29.407Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B"
time=2025-09-11T01:28:34.504Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.096350349 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:34.747Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.340107962 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:34.842Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.9 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:34.842Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 library=cuda parallel=1 required="3.1 GiB" gpus=1
time=2025-09-11T01:28:34.842Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=25 layers.offload=25 layers.split=[25] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.1 GiB" memory.required.partial="3.1 GiB" memory.required.kv="58.0 MiB" memory.required.allocations="[3.1 GiB]" memory.weights.total="577.8 MiB" memory.weights.repeating="193.8 MiB" memory.weights.nonrepeating="384.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.2 GiB"
time=2025-09-11T01:28:34.844Z level=INFO source=runner.go:1170 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:2048 FlashAttention:false KvSize:2048 KvCacheType: NumThreads:8 GPULayers:25[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-09-11T01:28:34.867Z level=INFO source=ggml.go:131 msg="" architecture=gemma3 file_type=BF16 name="Embeddinggemma 300M" description="" num_tensors=316 num_key_values=37
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
time=2025-09-11T01:28:34.947Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-09-11T01:28:34.997Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58967704 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:487 msg="offloading 24 repeating layers to GPU"
time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:493 msg="offloading output layer to GPU"
time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:498 msg="offloaded 25/25 layers to GPU"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="586.8 MiB"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="384.0 MiB"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="58.0 MiB"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="108.0 MiB"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="6.0 MiB"
time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:342 msg="total memory" size="1.1 GiB"
time=2025-09-11T01:28:35.063Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-09-11T01:28:35.063Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding"
time=2025-09-11T01:28:35.064Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model"
time=2025-09-11T01:28:35.566Z level=INFO source=server.go:1288 msg="llama runner started in 6.26 seconds"
[GIN] 2025/09/11 - 01:28:35 | 200 |  6.940837124s |      172.18.0.6 | POST     "/v1/embeddings"
time=2025-09-11T01:28:36.133Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="4.9 GiB"
llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = llama3.1
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 32
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.58 GiB (4.89 BPW) 
load: printing all EOG tokens:
load:   - 128001 ('<|end_of_text|>')
load:   - 128008 ('<|eom_id|>')
load:   - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 1
print_info: model type       = ?B
print_info: model params     = 8.03 B
print_info: general.name     = Meta Llama 3.1 8B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-09-11T01:28:36.398Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --model /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 --port 38393"
time=2025-09-11T01:28:36.414Z level=INFO source=runner.go:864 msg="starting go runner"
time=2025-09-11T01:28:36.491Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="11.9 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:36.491Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
time=2025-09-11T01:28:36.519Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-09-11T01:28:36.522Z level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:38393"
time=2025-09-11T01:28:41.587Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.095617305 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
time=2025-09-11T01:28:41.837Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.346106826 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
time=2025-09-11T01:28:41.916Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.8 GiB" free_swap="3.5 GiB"
time=2025-09-11T01:28:41.916Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 library=cuda parallel=1 required="5.7 GiB" gpus=1
time=2025-09-11T01:28:41.916Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split=[33] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.7 GiB" memory.required.partial="5.7 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.7 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="677.5 MiB"
time=2025-09-11T01:28:41.917Z level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:8 GPULayers:33[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}"
time=2025-09-11T01:28:41.918Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding"
time=2025-09-11T01:28:41.918Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model"
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3070) - 7098 MiB free
llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 8B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1
llama_model_loader: - kv   5:                         general.size_label str              = 8B
llama_model_loader: - kv   6:                            general.license str              = llama3.1
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 32
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.58 GiB (4.89 BPW) 
time=2025-09-11T01:28:42.087Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.595772987 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515
load: printing all EOG tokens:
load:   - 128001 ('<|end_of_text|>')
load:   - 128008 ('<|eom_id|>')
load:   - 128009 ('<|eot_id|>')

OS

Windows, Docker

GPU

Nvidia

CPU

AMD

Ollama version

0.11.10

Originally created by @tomaszbk on GitHub (Sep 11, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/12247 ### What is the issue? tried embeddinggemma with qwen 2.5vl and llama 3.1, on 8GB VRAM GPU, and the llm models are unloaded upon an embedding request. using granite embedding 300m doesnt unload the other models ### Relevant log output ```shell time=2025-09-11T01:28:15.420Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="2.3 GiB" time=2025-09-11T01:28:15.553Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 --port 33085" time=2025-09-11T01:28:15.564Z level=INFO source=runner.go:1251 msg="starting ollama engine" time=2025-09-11T01:28:15.566Z level=INFO source=runner.go:1286 msg="Server listening on 127.0.0.1:33085" time=2025-09-11T01:28:15.645Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.5 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:15.645Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B" time=2025-09-11T01:28:19.869Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.098100875 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:20.119Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.348006231 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:20.185Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.9 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:20.185Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 library=cuda parallel=1 required="3.1 GiB" gpus=1 time=2025-09-11T01:28:20.185Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=25 layers.offload=25 layers.split=[25] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.1 GiB" memory.required.partial="3.1 GiB" memory.required.kv="58.0 MiB" memory.required.allocations="[3.1 GiB]" memory.weights.total="577.8 MiB" memory.weights.repeating="193.8 MiB" memory.weights.nonrepeating="384.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.2 GiB" time=2025-09-11T01:28:20.187Z level=INFO source=runner.go:1170 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:2048 FlashAttention:false KvSize:2048 KvCacheType: NumThreads:8 GPULayers:25[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-09-11T01:28:20.207Z level=INFO source=ggml.go:131 msg="" architecture=gemma3 file_type=BF16 name="Embeddinggemma 300M" description="" num_tensors=316 num_key_values=37 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so time=2025-09-11T01:28:20.266Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-09-11T01:28:20.368Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.597301181 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1647 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:487 msg="offloading 24 repeating layers to GPU" time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:493 msg="offloading output layer to GPU" time=2025-09-11T01:28:20.390Z level=INFO source=ggml.go:498 msg="offloaded 25/25 layers to GPU" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="586.8 MiB" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="384.0 MiB" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="58.0 MiB" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="108.0 MiB" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="6.0 MiB" time=2025-09-11T01:28:20.391Z level=INFO source=backend.go:342 msg="total memory" size="1.1 GiB" time=2025-09-11T01:28:20.391Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-09-11T01:28:20.391Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding" time=2025-09-11T01:28:20.391Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model" time=2025-09-11T01:28:20.895Z level=INFO source=server.go:1288 msg="llama runner started in 6.22 seconds" [GIN] 2025/09/11 - 01:28:21 | 200 | 6.694510783s | 172.18.0.6 | POST "/v1/embeddings" time=2025-09-11T01:28:21.232Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="4.9 GiB" llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 8B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 32 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 4096 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 15 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 4.58 GiB (4.89 BPW) load: printing all EOG tokens: load: - 128001 ('<|end_of_text|>') load: - 128008 ('<|eom_id|>') load: - 128009 ('<|eot_id|>') load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 8.03 B print_info: general.name = Meta Llama 3.1 8B Instruct print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128001 '<|end_of_text|>' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-09-11T01:28:21.538Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --model /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 --port 45399" time=2025-09-11T01:28:21.550Z level=INFO source=runner.go:864 msg="starting go runner" time=2025-09-11T01:28:21.631Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.2 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:21.632Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so time=2025-09-11T01:28:21.650Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-09-11T01:28:21.653Z level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:45399" time=2025-09-11T01:28:26.720Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.087912251 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 time=2025-09-11T01:28:26.970Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.337829115 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 time=2025-09-11T01:28:27.030Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.8 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:27.030Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 library=cuda parallel=1 required="5.7 GiB" gpus=1 time=2025-09-11T01:28:27.030Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split=[33] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.7 GiB" memory.required.partial="5.7 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.7 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="677.5 MiB" time=2025-09-11T01:28:27.031Z level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:8 GPULayers:33[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}" time=2025-09-11T01:28:27.031Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding" time=2025-09-11T01:28:27.032Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3070) - 7098 MiB free llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 8B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 32 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 4096 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 15 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 4.58 GiB (4.89 BPW) time=2025-09-11T01:28:27.220Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.5878821720000005 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1686 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 load: printing all EOG tokens: load: - 128001 ('<|end_of_text|>') load: - 128008 ('<|eom_id|>') load: - 128009 ('<|eot_id|>') load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 4096 print_info: n_layer = 32 print_info: n_head = 32 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 4 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 14336 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = linear print_info: freq_base_train = 500000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 131072 print_info: rope_finetuned = unknown print_info: model type = 8B print_info: model params = 8.03 B print_info: general.name = Meta Llama 3.1 8B Instruct print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128001 '<|end_of_text|>' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: offloading 32 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 33/33 layers to GPU load_tensors: CUDA0 model buffer size = 4403.49 MiB load_tensors: CPU_Mapped model buffer size = 281.81 MiB llama_context: constructing llama_context llama_context: n_seq_max = 1 llama_context: n_ctx = 4096 llama_context: n_ctx_per_seq = 4096 llama_context: n_batch = 512 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = 0 llama_context: kv_unified = false llama_context: freq_base = 500000.0 llama_context: freq_scale = 1 llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_context: CUDA_Host output buffer size = 0.50 MiB llama_kv_cache_unified: CUDA0 KV buffer size = 512.00 MiB llama_kv_cache_unified: size = 512.00 MiB ( 4096 cells, 32 layers, 1/1 seqs), K (f16): 256.00 MiB, V (f16): 256.00 MiB llama_context: CUDA0 compute buffer size = 300.01 MiB llama_context: CUDA_Host compute buffer size = 20.01 MiB llama_context: graph nodes = 1126 llama_context: graph splits = 2 time=2025-09-11T01:28:28.036Z level=INFO source=server.go:1288 msg="llama runner started in 6.50 seconds" time=2025-09-11T01:28:28.036Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-09-11T01:28:28.036Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding" time=2025-09-11T01:28:28.037Z level=INFO source=server.go:1288 msg="llama runner started in 6.50 seconds" [GIN] 2025/09/11 - 01:28:28 | 200 | 7.589501463s | 172.18.0.6 | POST "/v1/chat/completions" [GIN] 2025/09/11 - 01:28:28 | 200 | 343.092569ms | 172.18.0.6 | POST "/v1/chat/completions" time=2025-09-11T01:28:29.170Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="2.3 GiB" time=2025-09-11T01:28:29.309Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 --port 35901" time=2025-09-11T01:28:29.321Z level=INFO source=runner.go:1251 msg="starting ollama engine" time=2025-09-11T01:28:29.324Z level=INFO source=runner.go:1286 msg="Server listening on 127.0.0.1:35901" time=2025-09-11T01:28:29.407Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.4 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:29.407Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B" time=2025-09-11T01:28:34.504Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.096350349 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:34.747Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.340107962 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:34.842Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.9 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:34.842Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 library=cuda parallel=1 required="3.1 GiB" gpus=1 time=2025-09-11T01:28:34.842Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=25 layers.offload=25 layers.split=[25] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.1 GiB" memory.required.partial="3.1 GiB" memory.required.kv="58.0 MiB" memory.required.allocations="[3.1 GiB]" memory.weights.total="577.8 MiB" memory.weights.repeating="193.8 MiB" memory.weights.nonrepeating="384.0 MiB" memory.graph.full="2.0 GiB" memory.graph.partial="2.2 GiB" time=2025-09-11T01:28:34.844Z level=INFO source=runner.go:1170 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:2048 FlashAttention:false KvSize:2048 KvCacheType: NumThreads:8 GPULayers:25[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-09-11T01:28:34.867Z level=INFO source=ggml.go:131 msg="" architecture=gemma3 file_type=BF16 name="Embeddinggemma 300M" description="" num_tensors=316 num_key_values=37 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so time=2025-09-11T01:28:34.947Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-09-11T01:28:34.997Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.58967704 runner.size="5.7 GiB" runner.vram="5.7 GiB" runner.parallel=1 runner.pid=1742 runner.model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:487 msg="offloading 24 repeating layers to GPU" time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:493 msg="offloading output layer to GPU" time=2025-09-11T01:28:35.063Z level=INFO source=ggml.go:498 msg="offloaded 25/25 layers to GPU" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="586.8 MiB" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="384.0 MiB" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="58.0 MiB" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="108.0 MiB" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="6.0 MiB" time=2025-09-11T01:28:35.063Z level=INFO source=backend.go:342 msg="total memory" size="1.1 GiB" time=2025-09-11T01:28:35.063Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-09-11T01:28:35.063Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding" time=2025-09-11T01:28:35.064Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model" time=2025-09-11T01:28:35.566Z level=INFO source=server.go:1288 msg="llama runner started in 6.26 seconds" [GIN] 2025/09/11 - 01:28:35 | 200 | 6.940837124s | 172.18.0.6 | POST "/v1/embeddings" time=2025-09-11T01:28:36.133Z level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 library=cuda total="8.0 GiB" available="4.9 GiB" llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 8B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 32 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 4096 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 15 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 4.58 GiB (4.89 BPW) load: printing all EOG tokens: load: - 128001 ('<|end_of_text|>') load: - 128008 ('<|eom_id|>') load: - 128009 ('<|eot_id|>') load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 8.03 B print_info: general.name = Meta Llama 3.1 8B Instruct print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128001 '<|end_of_text|>' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-09-11T01:28:36.398Z level=INFO source=server.go:398 msg="starting runner" cmd="/usr/bin/ollama runner --model /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 --port 38393" time=2025-09-11T01:28:36.414Z level=INFO source=runner.go:864 msg="starting go runner" time=2025-09-11T01:28:36.491Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="11.9 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:36.491Z level=INFO source=server.go:510 msg="model requires more memory than is currently available, evicting a model to make space" estimate.library="" estimate.layers.requested=0 estimate.layers.model=0 estimate.layers.offload=0 estimate.layers.split=[] estimate.memory.available=[] estimate.memory.gpu_overhead="0 B" estimate.memory.required.full="0 B" estimate.memory.required.partial="0 B" estimate.memory.required.kv="0 B" estimate.memory.required.allocations=[] estimate.memory.weights.total="0 B" estimate.memory.weights.repeating="0 B" estimate.memory.weights.nonrepeating="0 B" estimate.memory.graph.full="0 B" estimate.memory.graph.partial="0 B" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes, ID: GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so time=2025-09-11T01:28:36.519Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-09-11T01:28:36.522Z level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:38393" time=2025-09-11T01:28:41.587Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.095617305 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 time=2025-09-11T01:28:41.837Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.346106826 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 time=2025-09-11T01:28:41.916Z level=INFO source=server.go:503 msg="system memory" total="15.2 GiB" free="12.8 GiB" free_swap="3.5 GiB" time=2025-09-11T01:28:41.916Z level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 library=cuda parallel=1 required="5.7 GiB" gpus=1 time=2025-09-11T01:28:41.916Z level=INFO source=server.go:543 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split=[33] memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.7 GiB" memory.required.partial="5.7 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.7 GiB]" memory.weights.total="4.3 GiB" memory.weights.repeating="3.9 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="677.5 MiB" time=2025-09-11T01:28:41.917Z level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:8 GPULayers:33[ID:GPU-98df2674-c0ca-b15b-1a24-bf2947db3290 Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}" time=2025-09-11T01:28:41.918Z level=INFO source=server.go:1250 msg="waiting for llama runner to start responding" time=2025-09-11T01:28:41.918Z level=INFO source=server.go:1284 msg="waiting for server to become available" status="llm server loading model" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3070) - 7098 MiB free llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 8B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 8B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 32 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 4096 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 15 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 4.58 GiB (4.89 BPW) time=2025-09-11T01:28:42.087Z level=WARN source=sched.go:652 msg="gpu VRAM usage didn't recover within timeout" seconds=5.595772987 runner.size="3.1 GiB" runner.vram="3.1 GiB" runner.parallel=1 runner.pid=1782 runner.model=/root/.ollama/models/blobs/sha256-0800cbac9c2064dde519420e75e512a83cb360de3ad5df176185dc69652fc515 load: printing all EOG tokens: load: - 128001 ('<|end_of_text|>') load: - 128008 ('<|eom_id|>') load: - 128009 ('<|eot_id|>') ``` ### OS Windows, Docker ### GPU Nvidia ### CPU AMD ### Ollama version 0.11.10

GiteaMirror added the bug label 2026-04-22 17:05:26 -05:00

GiteaMirror closed this issue 2026-04-22 17:05:27 -05:00

GiteaMirror commented 2026-04-22 17:05:28 -05:00

Author

Owner

Copy Link

@tomaszbk commented on GitHub (Sep 11, 2025):

my logs were taken after alternating llama 3.1 and gemmaembedding requests

<!-- gh-comment-id:3277071143 --> @tomaszbk commented on GitHub (Sep 11, 2025): my logs were taken after alternating llama 3.1 and gemmaembedding requests

GiteaMirror commented 2026-04-22 17:05:29 -05:00

Author

Owner

Copy Link

@tomaszbk commented on GitHub (Sep 11, 2025):

seems gemmaembedding takes like 3gb, but ollama reports its only 600mb

<!-- gh-comment-id:3277072986 --> @tomaszbk commented on GitHub (Sep 11, 2025): seems gemmaembedding takes like 3gb, but ollama reports its only 600mb

GiteaMirror commented 2026-04-22 17:05:29 -05:00

Author

Owner

Copy Link

@rick-github commented on GitHub (Sep 11, 2025):

A model requires VRAM for context and computation, so the loaded size of a model will always be greater than the size of just the weights. In this case, embeddinggemma needs 3.3GB of VRAM when loaded, so there is not enough room in your 3070 to host both that model and another LLM at the same time.

Size of weights:

NAME                        ID              SIZE      MODIFIED   
embeddinggemma:latest       693ca723e5e7    621 MB    6 days ago    
granite-embedding:latest    eb4c533ba6f7     62 MB    4 minutes ago    
llama3.1:8b                 46e0c10c039e    4.9 GB    6 months ago     
qwen2.5vl:latest            5ced39dfa4ba    6.0 GB    3 months ago  

VRAM footprint:

NAME                        ID              SIZE      PROCESSOR    CONTEXT    UNTIL   
embeddinggemma:latest       693ca723e5e7    3.3 GB    100% GPU     2048       Forever    
granite-embedding:latest    eb4c533ba6f7    543 MB    100% GPU     4096       Forever    
llama3.1:8b                 46e0c10c039e    6.1 GB    100% GPU     4096       Forever    
qwen2.5vl:latest            5ced39dfa4ba    8.5 GB    100% GPU     4096       Forever    

<!-- gh-comment-id:3279851358 --> @rick-github commented on GitHub (Sep 11, 2025): A model requires VRAM for context and computation, so the loaded size of a model will always be greater than the size of just the weights. In this case, embeddinggemma needs 3.3GB of VRAM when loaded, so there is not enough room in your 3070 to host both that model and another LLM at the same time. Size of weights: ``` NAME ID SIZE MODIFIED embeddinggemma:latest 693ca723e5e7 621 MB 6 days ago granite-embedding:latest eb4c533ba6f7 62 MB 4 minutes ago llama3.1:8b 46e0c10c039e 4.9 GB 6 months ago qwen2.5vl:latest 5ced39dfa4ba 6.0 GB 3 months ago ``` VRAM footprint: ``` NAME ID SIZE PROCESSOR CONTEXT UNTIL embeddinggemma:latest 693ca723e5e7 3.3 GB 100% GPU 2048 Forever granite-embedding:latest eb4c533ba6f7 543 MB 100% GPU 4096 Forever llama3.1:8b 46e0c10c039e 6.1 GB 100% GPU 4096 Forever qwen2.5vl:latest 5ced39dfa4ba 8.5 GB 100% GPU 4096 Forever ```

GiteaMirror commented 2026-04-22 17:05:29 -05:00

Author

Owner

Copy Link

@tomaszbk commented on GitHub (Sep 11, 2025):

Thanks! It's a shame gemma embeddiing was announced as a small model but turns out to be quite the opposite

<!-- gh-comment-id:3280854219 --> @tomaszbk commented on GitHub (Sep 11, 2025): Thanks! It's a shame gemma embeddiing was announced as a small model but turns out to be quite the opposite

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

v0.23.1

v0.23.1-rc0

v0.23.0

v0.23.0-rc0

v0.22.1

v0.22.1-rc1

v0.22.1-rc0

v0.22.0

v0.22.0-rc1

v0.21.3-rc0

v0.21.2-rc1

v0.21.2

v0.21.2-rc0

v0.21.1

v0.21.1-rc1

v0.21.1-rc0

v0.21.0

v0.21.0-rc1

v0.21.0-rc0

v0.20.8-rc0

v0.20.7

v0.20.7-rc1

v0.20.7-rc0

v0.20.6

v0.20.6-rc1

v0.20.6-rc0

v0.20.5

v0.20.5-rc2

v0.20.5-rc1

v0.20.5-rc0

v0.20.4

v0.20.4-rc2

v0.20.4-rc1

v0.20.4-rc0

v0.20.3

v0.20.3-rc0

v0.20.2

v0.20.1

v0.20.1-rc2

v0.20.1-rc1

v0.20.1-rc0

v0.20.0

v0.20.0-rc1

v0.20.0-rc0

v0.19.0

v0.19.0-rc2

v0.19.0-rc1

v0.19.0-rc0

v0.18.4-rc1

v0.18.4-rc0

v0.18.3

v0.18.3-rc2

v0.18.3-rc1

v0.18.3-rc0

v0.18.2

v0.18.2-rc1

v0.18.2-rc0

v0.18.1

v0.18.1-rc1

v0.18.1-rc0

v0.18.0

v0.18.0-rc2

v0.18.0-rc1

v0.18.0-rc0

v0.17.8-rc4

v0.17.8-rc3

v0.17.8-rc2

v0.17.8-rc1

v0.17.8-rc0

v0.17.7

v0.17.7-rc2

v0.17.7-rc1

v0.17.7-rc0

v0.17.6

v0.17.5

v0.17.4

v0.17.3

v0.17.2

v0.17.1

v0.17.1-rc2

v0.17.1-rc1

v0.17.1-rc0

v0.17.0

v0.17.0-rc2

v0.17.0-rc1

v0.17.0-rc0

v0.16.3

v0.16.3-rc2

v0.16.3-rc1

v0.16.3-rc0

v0.16.2

v0.16.2-rc0

v0.16.1

v0.16.0

v0.16.0-rc2

v0.16.0-rc0

v0.16.0-rc1

v0.15.6

v0.15.5

v0.15.5-rc5

v0.15.5-rc4

v0.15.5-rc3

v0.15.5-rc2

v0.15.5-rc1

v0.15.5-rc0

v0.15.4

v0.15.3

v0.15.2

v0.15.1

v0.15.1-rc1

v0.15.1-rc0

v0.15.0-rc6

v0.15.0

v0.15.0-rc5

v0.15.0-rc4

v0.15.0-rc3

v0.15.0-rc2

v0.15.0-rc1

v0.15.0-rc0

v0.14.3

v0.14.3-rc3

v0.14.3-rc2

v0.14.3-rc1

v0.14.3-rc0

v0.14.2

v0.14.2-rc1

v0.14.2-rc0

v0.14.1

v0.14.0-rc11

v0.14.0

v0.14.0-rc10

v0.14.0-rc9

v0.14.0-rc8

v0.14.0-rc7

v0.14.0-rc6

v0.14.0-rc5

v0.14.0-rc4

v0.14.0-rc3

v0.14.0-rc2

v0.14.0-rc1

v0.14.0-rc0

v0.13.5

v0.13.5-rc1

v0.13.5-rc0

v0.13.4-rc2

v0.13.4

v0.13.4-rc1

v0.13.4-rc0

v0.13.3

v0.13.3-rc1

v0.13.3-rc0

v0.13.2

v0.13.2-rc2

v0.13.2-rc1

v0.13.2-rc0

v0.13.1

v0.13.1-rc2

v0.13.1-rc1

v0.13.1-rc0

v0.13.0

v0.13.0-rc0

v0.12.11

v0.12.11-rc1

v0.12.11-rc0

v0.12.10

v0.12.10-rc1

v0.12.10-rc0

v0.12.9-rc0

v0.12.9

v0.12.8

v0.12.8-rc0

v0.12.7

v0.12.7-rc1

v0.12.7-rc0

v0.12.7-citest0

v0.12.6

v0.12.6-rc1

v0.12.6-rc0

v0.12.5

v0.12.5-rc0

v0.12.4

v0.12.4-rc7

v0.12.4-rc6

v0.12.4-rc5

v0.12.4-rc4

v0.12.4-rc3

v0.12.4-rc2

v0.12.4-rc1

v0.12.4-rc0

v0.12.3

v0.12.2

v0.12.2-rc0

v0.12.1

v0.12.1-rc1

v0.12.1-rc2

v0.12.1-rc0

v0.12.0

v0.12.0-rc1

v0.12.0-rc0

v0.11.11

v0.11.11-rc3

v0.11.11-rc2

v0.11.11-rc1

v0.11.11-rc0

v0.11.10

v0.11.9

v0.11.9-rc0

v0.11.8

v0.11.8-rc0

v0.11.7-rc1

v0.11.7-rc0

v0.11.7

v0.11.6

v0.11.6-rc0

v0.11.5-rc4

v0.11.5-rc3

v0.11.5

v0.11.5-rc5

v0.11.5-rc2

v0.11.5-rc1

v0.11.5-rc0

v0.11.4

v0.11.4-rc0

v0.11.3

v0.11.3-rc0

v0.11.2

v0.11.1

v0.11.0-rc0

v0.11.0-rc1

v0.11.0-rc2

v0.11.0

v0.10.2-int1

v0.10.1

v0.10.0

v0.10.0-rc4

v0.10.0-rc3

v0.10.0-rc2

v0.10.0-rc1

v0.10.0-rc0

v0.9.7-rc1

v0.9.7-rc0

v0.9.6

v0.9.6-rc0

v0.9.6-ci0

v0.9.5

v0.9.4-rc5

v0.9.4-rc6

v0.9.4

v0.9.4-rc3

v0.9.4-rc4

v0.9.4-rc1

v0.9.4-rc2

v0.9.4-rc0

v0.9.3

v0.9.3-rc5

v0.9.4-citest0

v0.9.3-rc4

v0.9.3-rc3

v0.9.3-rc2

v0.9.3-rc1

v0.9.3-rc0

v0.9.2

v0.9.1

v0.9.1-rc1

v0.9.1-rc0

v0.9.1-ci1

v0.9.1-ci0

v0.9.0

v0.9.0-rc0

v0.8.0

v0.8.0-rc0

v0.7.1-rc2

v0.7.1

v0.7.1-rc1

v0.7.1-rc0

v0.7.0

v0.7.0-rc1

v0.7.0-rc0

v0.6.9-rc0

v0.6.8

v0.6.8-rc0

v0.6.7

v0.6.7-rc2

v0.6.7-rc1

v0.6.7-rc0

v0.6.6

v0.6.6-rc2

v0.6.6-rc1

v0.6.6-rc0

v0.6.5-rc1

v0.6.5

v0.6.5-rc0

v0.6.4-rc0

v0.6.4

v0.6.3-rc1

v0.6.3

v0.6.3-rc0

v0.6.2

v0.6.2-rc0

v0.6.1

v0.6.1-rc0

v0.6.0-rc0

v0.6.0

v0.5.14-rc0

v0.5.13

v0.5.13-rc6

v0.5.13-rc5

v0.5.13-rc4

v0.5.13-rc3

v0.5.13-rc2

v0.5.13-rc1

v0.5.13-rc0

v0.5.12

v0.5.12-rc1

v0.5.12-rc0

v0.5.11

v0.5.10

v0.5.9

v0.5.9-rc0

v0.5.8-rc13

v0.5.8

v0.5.8-rc12

v0.5.8-rc11

v0.5.8-rc10

v0.5.8-rc9

v0.5.8-rc8

v0.5.8-rc7

v0.5.8-rc6

v0.5.8-rc5

v0.5.8-rc4

v0.5.8-rc3

v0.5.8-rc2

v0.5.8-rc1

v0.5.8-rc0

v0.5.7

v0.5.6

v0.5.5

v0.5.5-rc0

v0.5.4

v0.5.3

v0.5.3-rc0

v0.5.2

v0.5.2-rc3

v0.5.2-rc2

v0.5.2-rc1

v0.5.2-rc0

v0.5.1

v0.5.0

v0.5.0-rc1

v0.4.8-rc0

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.3-rc0

v0.4.2

v0.4.2-rc1

v0.4.2-rc0

v0.4.1

v0.4.1-rc0

v0.4.0

v0.4.0-rc8

v0.4.0-rc7

v0.4.0-rc6

v0.4.0-rc5

v0.4.0-rc4

v0.4.0-rc3

v0.4.0-rc2

v0.4.0-rc1

v0.4.0-rc0

v0.4.0-ci3

v0.3.14

v0.3.14-rc0

v0.3.13

v0.3.12

v0.3.12-rc5

v0.3.12-rc4

v0.3.12-rc3

v0.3.12-rc2

v0.3.12-rc1

v0.3.11

v0.3.11-rc4

v0.3.11-rc3

v0.3.11-rc2

v0.3.11-rc1

v0.3.10

v0.3.10-rc1

v0.3.9

v0.3.8

v0.3.7

v0.3.7-rc6

v0.3.7-rc5

v0.3.7-rc4

v0.3.7-rc3

v0.3.7-rc2

v0.3.7-rc1

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.8

v0.2.8-rc2

v0.2.8-rc1

v0.2.7

v0.2.6

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.2-rc2

v0.2.2-rc1

v0.2.1

v0.2.0

v0.1.49-rc14

v0.1.49-rc13

v0.1.49-rc12

v0.1.49-rc11

v0.1.49-rc10

v0.1.49-rc9

v0.1.49-rc8

v0.1.49-rc7

v0.1.49-rc6

v0.1.49-rc4

v0.1.49-rc5

v0.1.49-rc3

v0.1.49-rc2

v0.1.49-rc1

v0.1.48

v0.1.47

v0.1.46

v0.1.45-rc5

v0.1.45

v0.1.45-rc4

v0.1.45-rc3

v0.1.45-rc2

v0.1.45-rc1

v0.1.44

v0.1.43

v0.1.42

v0.1.41

v0.1.40

v0.1.40-rc1

v0.1.39

v0.1.39-rc2

v0.1.39-rc1

v0.1.38

v0.1.37

v0.1.36

v0.1.35

v0.1.35-rc1

v0.1.34

v0.1.34-rc1

v0.1.33

v0.1.33-rc7

v0.1.33-rc6

v0.1.33-rc5

v0.1.33-rc4

v0.1.33-rc3

v0.1.33-rc2

v0.1.33-rc1

v0.1.32

v0.1.32-rc2

v0.1.32-rc1

v0.1.31

v0.1.30

v0.1.29

v0.1.28

v0.1.27

v0.1.26

v0.1.25

v0.1.24

v0.1.23

v0.1.22

v0.1.21

v0.1.20

v0.1.19

v0.1.18

v0.1.17

v0.1.16

v0.1.15

v0.1.14

v0.1.13

v0.1.12

v0.1.11

v0.1.10

v0.1.9

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

amd

api

app

bug

build

cli

cloud

compatibility

context-length

create

docker

documentation

embeddings

feature request

feedback wanted

good first issue

gpt-oss

gpu

harmony

help wanted

image

install

intel

js

launch

linux

macos

memory

mlx

model

needs more info

networking

nvidia

ollama.com

performance

pull-request

Mirrored from GitHub Pull Request

python

question

registry

rendering

thinking

tools

top

vulkan

windows

wsl

No Label bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#33907