github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #12305] Too Much RAM Eaten in Ollama 0.11.11 #70235

New Issue
Closed
opened 2026-05-04 20:44:39 -05:00 by GiteaMirror · 30 comments
GiteaMirror commented 2026-05-04 20:44:39 -05:00
Owner
Copy Link

Originally created by @Panican-Whyasker on GitHub (Sep 16, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/12305

What is the issue?

After the 0.11.11 update (Windows 11 Pro here), models loaded in ollama eat too much RAM.

Example: phi4-mini-reasoning (3.2 GBytes on disk; should have fitted in the 6 GB of VRAM of my Quadro RTX3000) now loads in main computer RAM and eats about 22 GBytes).

Example 2: mistral-nemo (7.1 GBytes) eats about 32 GB of system RAM.

Example 3: devstral (14 GBytes) eats about 39 GBytes of system RAM.

Example 4: smollm:135m (91 MBytes only) eats only ~250 MB of system RAM.

Example 5: deepseek-r1:70b-llama-distill-q4_K_M (42 GBytes) eats about 96 GB of system RAM.

Note 1: My system has 128 GBytes of system RAM, 16 logical CPU cores (a 9th Gen Core i9), and 1+2 TB of NVMe4 SSDs.

Note 2: It looks like the ollama process eats the whole lot of RAM in two bites (steps), as seen in Task Manager (see attached screenshot below).

Image

Relevant log output

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.11.11

Originally created by @Panican-Whyasker on GitHub (Sep 16, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/12305 ### What is the issue? After the 0.11.11 update (Windows 11 Pro here), models loaded in ollama eat too much RAM. Example: phi4-mini-reasoning (3.2 GBytes on disk; should have fitted in the 6 GB of VRAM of my Quadro RTX3000) now loads in main computer RAM and eats about 22 GBytes). Example 2: mistral-nemo (7.1 GBytes) eats about 32 GB of system RAM. Example 3: devstral (14 GBytes) eats about 39 GBytes of system RAM. Example 4: smollm:135m (91 MBytes only) eats only ~250 MB of system RAM. Example 5: deepseek-r1:70b-llama-distill-q4_K_M (42 GBytes) eats about 96 GB of system RAM. Note 1: My system has 128 GBytes of system RAM, 16 logical CPU cores (a 9th Gen Core i9), and 1+2 TB of NVMe4 SSDs. Note 2: It looks like the ollama process eats the whole lot of RAM in two bites (steps), as seen in Task Manager (see attached screenshot below). ![Image](https://github.com/user-attachments/assets/0ce25f22-28b2-485f-9ba6-41bbe4585f19) ### Relevant log output ```shell ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.11.11
GiteaMirror added the bug label 2026-05-04 20:44:39 -05:00
GiteaMirror commented 2026-05-04 20:44:42 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 16, 2025):

To add: that happens in either PowerShell and the Ollama app (same amount of RAM is eaten by the same model when run in each interface).

<!-- gh-comment-id:3299068106 --> @Panican-Whyasker commented on GitHub (Sep 16, 2025): To add: that happens in either PowerShell and the Ollama app (same amount of RAM is eaten by the same model when run in each interface).
GiteaMirror commented 2026-05-04 20:44:42 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2025):

Server logs will help in debugging.

<!-- gh-comment-id:3299243323 --> @rick-github commented on GitHub (Sep 16, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will help in debugging.
GiteaMirror commented 2026-05-04 20:44:43 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 16, 2025):

Server logs will help in debugging.

Here you go (server.log):

time=2025-09-15T21:08:57.191+02:00 level=INFO source=routes.go:1332 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:131072 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\Users\gyordano\.ollama\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-09-15T21:08:57.336+02:00 level=INFO source=images.go:477 msg="total blobs: 110"
time=2025-09-15T21:08:57.342+02:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0"
time=2025-09-15T21:08:57.345+02:00 level=INFO source=routes.go:1385 msg="Listening on 127.0.0.1:11434 (version 0.11.11)"
time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=8 efficiency=0 threads=16
time=2025-09-15T21:08:57.498+02:00 level=WARN source=cuda_common.go:52 msg="old CUDA driver detected - please upgrade to a newer driver for best performance" version=12.7
time=2025-09-15T21:08:57.514+02:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-6da04c1e-b5be-4fd5-6109-a2d7c38483ca library=cuda variant=v12 compute=7.5 driver=12.7 name="Quadro RTX 3000" total="6.0 GiB" available="5.0 GiB"
time=2025-09-15T21:08:57.514+02:00 level=INFO source=routes.go:1426 msg="entering low vram mode" "total vram"="6.0 GiB" threshold="20.0 GiB"
[GIN] 2025/09/16 - 09:14:31 | 200 | 540.9µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/09/16 - 09:14:31 | 200 | 19.0499ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/09/16 - 09:17:02 | 200 | 0s | 127.0.0.1 | HEAD "/"
[GIN] 2025/09/16 - 09:17:02 | 404 | 6.4995ms | 127.0.0.1 | POST "/api/show"
time=2025-09-16T09:17:02.967+02:00 level=INFO source=download.go:177 msg="downloading dde5aa3fc5ff in 16 126 MB part(s)"
time=2025-09-16T09:20:34.318+02:00 level=INFO source=download.go:177 msg="downloading 966de95ca8a6 in 1 1.4 KB part(s)"
time=2025-09-16T09:20:35.613+02:00 level=INFO source=download.go:177 msg="downloading fcc5a6bec9da in 1 7.7 KB part(s)"
time=2025-09-16T09:20:36.909+02:00 level=INFO source=download.go:177 msg="downloading a70ff7e570d9 in 1 6.0 KB part(s)"
time=2025-09-16T09:20:38.200+02:00 level=INFO source=download.go:177 msg="downloading 34bb5ab01051 in 1 561 B part(s)"
[GIN] 2025/09/16 - 09:20:44 | 200 | 3m42s | 127.0.0.1 | POST "/api/pull"
[GIN] 2025/09/16 - 09:20:44 | 200 | 63.6119ms | 127.0.0.1 | POST "/api/show"
llama_model_loader: loaded meta data with 30 key-value pairs and 255 tensors from C:\Users\gyordano.ollama\models\blobs\sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Llama 3.2 3B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = Llama-3.2
llama_model_loader: - kv 5: general.size_label str = 3B
llama_model_loader: - kv 6: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 7: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv 8: llama.block_count u32 = 28
llama_model_loader: - kv 9: llama.context_length u32 = 131072
llama_model_loader: - kv 10: llama.embedding_length u32 = 3072
llama_model_loader: - kv 11: llama.feed_forward_length u32 = 8192
llama_model_loader: - kv 12: llama.attention.head_count u32 = 24
llama_model_loader: - kv 13: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 14: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 15: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 16: llama.attention.key_length u32 = 128
llama_model_loader: - kv 17: llama.attention.value_length u32 = 128
llama_model_loader: - kv 18: general.file_type u32 = 15
llama_model_loader: - kv 19: llama.vocab_size u32 = 128256
llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 22: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 128009
llama_model_loader: - kv 28: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv 29: general.quantization_version u32 = 2
llama_model_loader: - type f32: 58 tensors
llama_model_loader: - type q4_K: 168 tensors
llama_model_loader: - type q6_K: 29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 1.87 GiB (5.01 BPW)
load: printing all EOG tokens:
load: - 128001 ('<|end_of_text|>')
load: - 128008 ('<|eom_id|>')
load: - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch = llama
print_info: vocab_only = 1
print_info: model type = ?B
print_info: model params = 3.21 B
print_info: general.name = Llama 3.2 3B Instruct
print_info: vocab type = BPE
print_info: n_vocab = 128256
print_info: n_merges = 280147
print_info: BOS token = 128000 '<|begin_of_text|>'
print_info: EOS token = 128009 '<|eot_id|>'
print_info: EOT token = 128009 '<|eot_id|>'
print_info: EOM token = 128008 '<|eom_id|>'
print_info: LF token = 198 'Ċ'
print_info: EOG token = 128001 '<|end_of_text|>'
print_info: EOG token = 128008 '<|eom_id|>'
print_info: EOG token = 128009 '<|eot_id|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-09-16T09:20:44.878+02:00 level=INFO source=server.go:399 msg="starting runner" cmd="C:\Users\gyordano\AppData\Local\Programs\Ollama\ollama.exe runner --model C:\Users\gyordano\.ollama\models\blobs\sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff --port 19700"
time=2025-09-16T09:20:56.859+02:00 level=INFO source=server.go:504 msg="system memory" total="127.7 GiB" free="106.3 GiB" free_swap="119.5 GiB"
time=2025-09-16T09:20:56.861+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=0 layers.split=[] memory.available="[4.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="15.9 GiB" memory.required.partial="0 B" memory.required.kv="14.0 GiB" memory.required.allocations="[0 B]" memory.weights.total="1.9 GiB" memory.weights.repeating="1.6 GiB" memory.weights.nonrepeating="308.2 MiB" memory.graph.full="6.3 GiB" memory.graph.partial="6.8 GiB"
time=2025-09-16T09:20:56.968+02:00 level=INFO source=runner.go:864 msg="starting go runner"
load_backend: loaded CPU backend from C:\Users\gyordano\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: Quadro RTX 3000, compute capability 7.5, VMM: yes, ID: GPU-6da04c1e-b5be-4fd5-6109-a2d7c38483ca
load_backend: loaded CUDA backend from C:\Users\gyordano\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll

(THE REST OF THE 2000+ LINE SERVER LOG WAS TRUNCATED FOR BREVITY.)

<!-- gh-comment-id:3299306681 --> @Panican-Whyasker commented on GitHub (Sep 16, 2025): > [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will help in debugging. Here you go (server.log): time=2025-09-15T21:08:57.191+02:00 level=INFO source=routes.go:1332 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:131072 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\gyordano\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-09-15T21:08:57.336+02:00 level=INFO source=images.go:477 msg="total blobs: 110" time=2025-09-15T21:08:57.342+02:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0" time=2025-09-15T21:08:57.345+02:00 level=INFO source=routes.go:1385 msg="Listening on 127.0.0.1:11434 (version 0.11.11)" time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-09-15T21:08:57.345+02:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=8 efficiency=0 threads=16 time=2025-09-15T21:08:57.498+02:00 level=WARN source=cuda_common.go:52 msg="old CUDA driver detected - please upgrade to a newer driver for best performance" version=12.7 time=2025-09-15T21:08:57.514+02:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-6da04c1e-b5be-4fd5-6109-a2d7c38483ca library=cuda variant=v12 compute=7.5 driver=12.7 name="Quadro RTX 3000" total="6.0 GiB" available="5.0 GiB" time=2025-09-15T21:08:57.514+02:00 level=INFO source=routes.go:1426 msg="entering low vram mode" "total vram"="6.0 GiB" threshold="20.0 GiB" [GIN] 2025/09/16 - 09:14:31 | 200 | 540.9µs | 127.0.0.1 | HEAD "/" [GIN] 2025/09/16 - 09:14:31 | 200 | 19.0499ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/09/16 - 09:17:02 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/09/16 - 09:17:02 | 404 | 6.4995ms | 127.0.0.1 | POST "/api/show" time=2025-09-16T09:17:02.967+02:00 level=INFO source=download.go:177 msg="downloading dde5aa3fc5ff in 16 126 MB part(s)" time=2025-09-16T09:20:34.318+02:00 level=INFO source=download.go:177 msg="downloading 966de95ca8a6 in 1 1.4 KB part(s)" time=2025-09-16T09:20:35.613+02:00 level=INFO source=download.go:177 msg="downloading fcc5a6bec9da in 1 7.7 KB part(s)" time=2025-09-16T09:20:36.909+02:00 level=INFO source=download.go:177 msg="downloading a70ff7e570d9 in 1 6.0 KB part(s)" time=2025-09-16T09:20:38.200+02:00 level=INFO source=download.go:177 msg="downloading 34bb5ab01051 in 1 561 B part(s)" [GIN] 2025/09/16 - 09:20:44 | 200 | 3m42s | 127.0.0.1 | POST "/api/pull" [GIN] 2025/09/16 - 09:20:44 | 200 | 63.6119ms | 127.0.0.1 | POST "/api/show" llama_model_loader: loaded meta data with 30 key-value pairs and 255 tensors from C:\Users\gyordano\.ollama\models\blobs\sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Llama 3.2 3B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Llama-3.2 llama_model_loader: - kv 5: general.size_label str = 3B llama_model_loader: - kv 6: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 7: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 8: llama.block_count u32 = 28 llama_model_loader: - kv 9: llama.context_length u32 = 131072 llama_model_loader: - kv 10: llama.embedding_length u32 = 3072 llama_model_loader: - kv 11: llama.feed_forward_length u32 = 8192 llama_model_loader: - kv 12: llama.attention.head_count u32 = 24 llama_model_loader: - kv 13: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 14: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 15: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 16: llama.attention.key_length u32 = 128 llama_model_loader: - kv 17: llama.attention.value_length u32 = 128 llama_model_loader: - kv 18: general.file_type u32 = 15 llama_model_loader: - kv 19: llama.vocab_size u32 = 128256 llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 22: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 28: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 29: general.quantization_version u32 = 2 llama_model_loader: - type f32: 58 tensors llama_model_loader: - type q4_K: 168 tensors llama_model_loader: - type q6_K: 29 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 1.87 GiB (5.01 BPW) load: printing all EOG tokens: load: - 128001 ('<|end_of_text|>') load: - 128008 ('<|eom_id|>') load: - 128009 ('<|eot_id|>') load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 3.21 B print_info: general.name = Llama 3.2 3B Instruct print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128001 '<|end_of_text|>' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-09-16T09:20:44.878+02:00 level=INFO source=server.go:399 msg="starting runner" cmd="C:\\Users\\gyordano\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --model C:\\Users\\gyordano\\.ollama\\models\\blobs\\sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff --port 19700" time=2025-09-16T09:20:56.859+02:00 level=INFO source=server.go:504 msg="system memory" total="127.7 GiB" free="106.3 GiB" free_swap="119.5 GiB" time=2025-09-16T09:20:56.861+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=0 layers.split=[] memory.available="[4.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="15.9 GiB" memory.required.partial="0 B" memory.required.kv="14.0 GiB" memory.required.allocations="[0 B]" memory.weights.total="1.9 GiB" memory.weights.repeating="1.6 GiB" memory.weights.nonrepeating="308.2 MiB" memory.graph.full="6.3 GiB" memory.graph.partial="6.8 GiB" time=2025-09-16T09:20:56.968+02:00 level=INFO source=runner.go:864 msg="starting go runner" load_backend: loaded CPU backend from C:\Users\gyordano\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: Quadro RTX 3000, compute capability 7.5, VMM: yes, ID: GPU-6da04c1e-b5be-4fd5-6109-a2d7c38483ca load_backend: loaded CUDA backend from C:\Users\gyordano\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll (THE REST OF THE 2000+ LINE SERVER LOG WAS TRUNCATED FOR BREVITY.)
GiteaMirror commented 2026-05-04 20:44:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2025):

First model loaded was llama3.2:3b-instruct-q4_K_M with a context size of 128k (KvSize:131072).

time=2025-09-16T09:20:56.861+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1
 layers.model=29 layers.offload=0 layers.split=[] memory.available="[4.5 GiB]" memory.gpu_overhead="0 B"
 memory.required.full="15.9 GiB" memory.required.partial="0 B" memory.required.kv="14.0 GiB" memory.required.allocations="[0 B]"
 memory.weights.total="1.9 GiB" memory.weights.repeating="1.6 GiB" memory.weights.nonrepeating="308.2 MiB"
 memory.graph.full="6.3 GiB" memory.graph.partial="6.8 GiB"

As a result of the large context size, the amount of memory required for the KV cache was too large (14.0GiB) to allow any layers to be loaded on the GPU. As a consequence, the model was loaded into system RAM.

Second model loaded was devstral:24b-small-2505-q4_K_M, again with a context of 128k.

time=2025-09-16T15:19:39.230+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1
 layers.model=41 layers.offload=0 layers.split=[] memory.available="[4.7 GiB]" memory.gpu_overhead="0 B"
 memory.required.full="33.0 GiB" memory.required.partial="0 B" memory.required.kv="20.0 GiB" memory.required.allocations="[0 B]"
 memory.weights.total="13.0 GiB" memory.weights.repeating="12.5 GiB" memory.weights.nonrepeating="525.0 MiB"
 memory.graph.full="8.3 GiB" memory.graph.partial="8.9 GiB"

As before, the size of the KV cache (20GiB) precluded loading any layers in VRAM, so the model was loaded in system RAM.

I suspect this is where the step in the memory usage occurs - both models are resident in system RAM. ollama ps will show loaded models.

If you reduce OLLAMA_CONTEXT_LENGTH (or unset it) it will increase the chances of being able to load a model into VRAM.

0.11.10:

NAME                              ID              SIZE     PROCESSOR    CONTEXT    UNTIL   
phi4-mini-reasoning:latest        3ca8c2865ce9    30 GB    100% GPU     131072     Forever    
devstral:24b-small-2505-q4_K_M    9bd74193e939    45 GB    100% GPU     131072     Forever    
llama3.2:3b-instruct-q4_K_M       a80c4f17acd5    24 GB    100% GPU     131072     Forever

0.11.11:

NAME                              ID              SIZE     PROCESSOR    CONTEXT    UNTIL   
phi4-mini-reasoning:latest        3ca8c2865ce9    30 GB    100% GPU     131072     Forever    
devstral:24b-small-2505-q4_K_M    9bd74193e939    45 GB    100% GPU     131072     Forever    
llama3.2:3b-instruct-q4_K_M       a80c4f17acd5    24 GB    100% GPU     131072     Forever
<!-- gh-comment-id:3299422629 --> @rick-github commented on GitHub (Sep 16, 2025): First model loaded was llama3.2:3b-instruct-q4_K_M with a context size of 128k (`KvSize:131072`). ``` time=2025-09-16T09:20:56.861+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=0 layers.split=[] memory.available="[4.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="15.9 GiB" memory.required.partial="0 B" memory.required.kv="14.0 GiB" memory.required.allocations="[0 B]" memory.weights.total="1.9 GiB" memory.weights.repeating="1.6 GiB" memory.weights.nonrepeating="308.2 MiB" memory.graph.full="6.3 GiB" memory.graph.partial="6.8 GiB" ``` As a result of the large context size, the amount of memory required for the KV cache was too large (14.0GiB) to allow any layers to be loaded on the GPU. As a consequence, the model was loaded into system RAM. Second model loaded was devstral:24b-small-2505-q4_K_M, again with a context of 128k. ``` time=2025-09-16T15:19:39.230+02:00 level=INFO source=server.go:544 msg=offload library=cuda layers.requested=-1 layers.model=41 layers.offload=0 layers.split=[] memory.available="[4.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="33.0 GiB" memory.required.partial="0 B" memory.required.kv="20.0 GiB" memory.required.allocations="[0 B]" memory.weights.total="13.0 GiB" memory.weights.repeating="12.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="8.3 GiB" memory.graph.partial="8.9 GiB" ``` As before, the size of the KV cache (20GiB) precluded loading any layers in VRAM, so the model was loaded in system RAM. I suspect this is where the step in the memory usage occurs - both models are resident in system RAM. `ollama ps` will show loaded models. If you reduce `OLLAMA_CONTEXT_LENGTH` (or unset it) it will increase the chances of being able to load a model into VRAM. 0.11.10: ``` NAME ID SIZE PROCESSOR CONTEXT UNTIL phi4-mini-reasoning:latest 3ca8c2865ce9 30 GB 100% GPU 131072 Forever devstral:24b-small-2505-q4_K_M 9bd74193e939 45 GB 100% GPU 131072 Forever llama3.2:3b-instruct-q4_K_M a80c4f17acd5 24 GB 100% GPU 131072 Forever ``` 0.11.11: ``` NAME ID SIZE PROCESSOR CONTEXT UNTIL phi4-mini-reasoning:latest 3ca8c2865ce9 30 GB 100% GPU 131072 Forever devstral:24b-small-2505-q4_K_M 9bd74193e939 45 GB 100% GPU 131072 Forever llama3.2:3b-instruct-q4_K_M a80c4f17acd5 24 GB 100% GPU 131072 Forever ```
GiteaMirror commented 2026-05-04 20:44:44 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 16, 2025):

First model loaded was llama3.2:3b-instruct-q4_K_M with a context size of 128k (KvSize:131072).

As a result of the large context size, the amount of memory required for the KV cache was too large (14.0GiB) to allow any layers to be loaded on the GPU. As a consequence, the model was loaded into system RAM.

Hi Rick,

That was not the case in earlier versions for models with a context size of 128k.

In most older versions of Ollama, a model that took e.g. 32 GBytes on the SSD would take slightly more than that in the system RAM.

With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM.

I can check older Server Logs to see when that drastic change happened; the earliest one that I see is from Aug 26.

The whole idea or democratizing LLMs by allowing one to run them locally in Ollama on personal desktop/laptop machines with as little as 4-8-16 GB of system RAM in some cases, has been completely ruined. I upgraded my work laptop to the max. of 128 GB of RAM in order to be able to run some large (100B+ parameters) LLMs.

Suddenly, that huge advantage of Ollama is (nearly) gone!... (Or, are Ollama developers sick & tired of the soooo many laymen out there, just like myself??!!....)

<!-- gh-comment-id:3299816225 --> @Panican-Whyasker commented on GitHub (Sep 16, 2025): > First model loaded was llama3.2:3b-instruct-q4_K_M with a context size of 128k (`KvSize:131072`). > > ``` > ``` > As a result of the large context size, the amount of memory required for the KV cache was too large (14.0GiB) to allow any layers to be loaded on the GPU. As a consequence, the model was loaded into system RAM. > ``` Hi Rick, That was not the case in earlier versions for models with a context size of 128k. In most older versions of Ollama, a model that took e.g. 32 GBytes on the SSD would take slightly more than that in the system RAM. With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM. I can check older Server Logs to see when that drastic change happened; the earliest one that I see is from Aug 26. The whole idea or democratizing LLMs by allowing one to run them locally in Ollama on personal desktop/laptop machines with as little as 4-8-16 GB of system RAM in some cases, has been completely ruined. I upgraded my work laptop to the max. of 128 GB of RAM in order to be able to run some large (100B+ parameters) LLMs. Suddenly, that huge advantage of Ollama is (nearly) gone!... (Or, are Ollama developers sick & tired of the soooo many laymen out there, just like myself??!!....)
GiteaMirror commented 2026-05-04 20:44:44 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2025):

With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM.

Not with a context of 128k. What version of ollama were you running?

<!-- gh-comment-id:3299827155 --> @rick-github commented on GitHub (Sep 16, 2025): > With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM. Not with a context of 128k. What version of ollama were you running?
GiteaMirror commented 2026-05-04 20:44:44 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 16, 2025):

With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM.

Not with a context of 128k. What version of ollama were you running?

Ollama v. 0.11.11 (updated earlier today). I usually update Ollama on the same day as I see the prompt.

I see in a server log from Aug 29 that I ran a 4 GB model with 7B params and it required memory.required.full="10.9 GiB". Context length was 32k. Was not like that before. Mistral 7B used to fit into the available VRAM (5 GB out of a 6 GB total).

One thing: I find the present continuous format of the Server Logs too tedious to read and understand.
Any idea, if opened in Notepad++, which language coloring scheme should I optimally choose for those files? How to best distinguish between different model sessions (more clearly identify when one session starts and ends)??...

<!-- gh-comment-id:3299866686 --> @Panican-Whyasker commented on GitHub (Sep 16, 2025): > > With 6 GB of dedicated VRAM, all models within ~5 GB on the SSD would load in the VRAM. > > Not with a context of 128k. What version of ollama were you running? Ollama v. 0.11.11 (updated earlier today). I usually update Ollama on the same day as I see the prompt. I see in a server log from Aug 29 that I ran a 4 GB model with 7B params and it required memory.required.full="10.9 GiB". Context length was 32k. Was not like that before. Mistral 7B used to fit into the available VRAM (5 GB out of a 6 GB total). One thing: I find the present continuous format of the Server Logs too tedious to read and understand. Any idea, if opened in Notepad++, which language coloring scheme should I optimally choose for those files? How to best distinguish between different model sessions (more clearly identify when one session starts and ends)??...
GiteaMirror commented 2026-05-04 20:44:45 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2025):

VERSION  NAME                               ID              SIZE     PROCESSOR    CONTEXT    UNTIL   
0.3.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% CPU     0          Forever    
0.4.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% CPU     0          Forever    
0.5.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% CPU     0          Forever    
0.6.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     0          Forever    
0.7.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     0          Forever    
0.8.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     0          Forever    
0.9.0    mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     0          Forever    
0.10.0   mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     32768      Forever    
0.11.0   mistral:7b-instruct-v0.3-q4_K_M    6577803aa9a0    11 GB    100% GPU     32768      Forever
<!-- gh-comment-id:3299975820 --> @rick-github commented on GitHub (Sep 16, 2025): ``` VERSION NAME ID SIZE PROCESSOR CONTEXT UNTIL 0.3.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% CPU 0 Forever 0.4.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% CPU 0 Forever 0.5.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% CPU 0 Forever 0.6.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 0 Forever 0.7.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 0 Forever 0.8.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 0 Forever 0.9.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 0 Forever 0.10.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 32768 Forever 0.11.0 mistral:7b-instruct-v0.3-q4_K_M 6577803aa9a0 11 GB 100% GPU 32768 Forever ```
GiteaMirror commented 2026-05-04 20:44:46 -05:00
Author
Owner
Copy Link

@etoulas commented on GitHub (Sep 17, 2025):

@rick-github Not sure what your last comment means. Did some default values change since 0.10?

I have the same issue as reported. My setup is Debian 13, 16 GB VRAM Nvidia card. Had no issue running a variety of models before. Since upgrading to 0.11 (not sure of my previous version) the time to first output has increased and gpt-oss-20b fails to load which was not an issue before.

I suspect that some defaults have changed. Probably regarding the context size if I understand the previous discussion.
Unfortunately, I cannot find any server logs.

EDIT: found the logs:

Before upgrading to 0.11.11 I was on version 0.11.8.

In my configuration I had num_ctx=16384 and num_gpu=99 defined. I had to change the num_gpu to "default" (in Open WebUI) and the problem disappeared. I didn't change the context.

Something must have changed between the versions...

Logs of the failing attempt:

Sep 17 08:10:32 freebear ollama[1373]: time=2025-09-17T08:10:32.965+02:00 level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 library=cuda total="15.7 GiB" available="10.5 GiB"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:200 msg="model wants flash attention"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:217 msg="enabling flash attention"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:399 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /home/et/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --port 33985"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:672 msg="loading model" "model layers"=25 requested=99
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.132+02:00 level=INFO source=runner.go:1254 msg="starting ollama engine"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.133+02:00 level=INFO source=runner.go:1289 msg="Server listening on 127.0.0.1:33985"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.169+02:00 level=INFO source=server.go:678 msg="system memory" total="15.6 GiB" free="13.9 GiB" free_swap="113.1 MiB"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.169+02:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 available="10.1 GiB" free="10.5 GiB" minimum="457.0 MiB" overhead="0 B"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.170+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:4 GPULayers:25[ID:GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.217+02:00 level=INFO source=ggml.go:131 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
Sep 17 08:10:33 freebear ollama[1373]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: found 1 CUDA devices:
Sep 17 08:10:33 freebear ollama[1373]:   Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes, ID: GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953
Sep 17 08:10:33 freebear ollama[1373]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.273+02:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.315+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:4 GPULayers:25[ID:GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Sep 17 08:10:33 freebear ollama[1373]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12036.68 MiB on device 0: cudaMalloc failed: out of memory
Sep 17 08:10:33 freebear ollama[1373]: alloc_tensor_range: failed to allocate CUDA0 buffer of size 12621369600
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:close LoraPath:[] Parallel:0 BatchSize:0 FlashAttention:false KvSize:0 KvCacheType: NumThreads:0 GPULayers:[] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="11.8 GiB"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=backend.go:315 msg="model weights" device=CPU size="1.1 GiB"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.364+02:00 level=INFO source=backend.go:342 msg="total memory" size="12.8 GiB"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.364+02:00 level=INFO source=sched.go:441 msg="Load failed" model=/home/et/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 error="memory layout cannot be allocated with num_gpu = 99"
Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.382+02:00 level=ERROR source=server.go:425 msg="llama runner terminated" error="signal: killed"
<!-- gh-comment-id:3301473133 --> @etoulas commented on GitHub (Sep 17, 2025): @rick-github Not sure what your last comment means. Did some default values change since 0.10? I have the same issue as reported. My setup is Debian 13, 16 GB VRAM Nvidia card. Had no issue running a variety of models before. Since upgrading to 0.11 (not sure of my previous version) the time to first output has increased and gpt-oss-20b fails to load which was not an issue before. I suspect that some defaults have changed. Probably regarding the context size if I understand the previous discussion. Unfortunately, I cannot find any server logs. EDIT: found the logs: Before upgrading to 0.11.11 I was on version 0.11.8. In my configuration I had num_ctx=16384 and num_gpu=99 defined. I had to change the num_gpu to "default" (in Open WebUI) and the problem disappeared. I didn't change the context. Something must have changed between the versions... Logs of the failing attempt: ``` Sep 17 08:10:32 freebear ollama[1373]: time=2025-09-17T08:10:32.965+02:00 level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 library=cuda total="15.7 GiB" available="10.5 GiB" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:200 msg="model wants flash attention" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:217 msg="enabling flash attention" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:399 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /home/et/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --port 33985" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.122+02:00 level=INFO source=server.go:672 msg="loading model" "model layers"=25 requested=99 Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.132+02:00 level=INFO source=runner.go:1254 msg="starting ollama engine" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.133+02:00 level=INFO source=runner.go:1289 msg="Server listening on 127.0.0.1:33985" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.169+02:00 level=INFO source=server.go:678 msg="system memory" total="15.6 GiB" free="13.9 GiB" free_swap="113.1 MiB" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.169+02:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 available="10.1 GiB" free="10.5 GiB" minimum="457.0 MiB" overhead="0 B" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.170+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:4 GPULayers:25[ID:GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.217+02:00 level=INFO source=ggml.go:131 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30 Sep 17 08:10:33 freebear ollama[1373]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Sep 17 08:10:33 freebear ollama[1373]: ggml_cuda_init: found 1 CUDA devices: Sep 17 08:10:33 freebear ollama[1373]: Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes, ID: GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 Sep 17 08:10:33 freebear ollama[1373]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.273+02:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.315+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:4 GPULayers:25[ID:GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 Layers:25(0..24)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Sep 17 08:10:33 freebear ollama[1373]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12036.68 MiB on device 0: cudaMalloc failed: out of memory Sep 17 08:10:33 freebear ollama[1373]: alloc_tensor_range: failed to allocate CUDA0 buffer of size 12621369600 Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=runner.go:1173 msg=load request="{Operation:close LoraPath:[] Parallel:0 BatchSize:0 FlashAttention:false KvSize:0 KvCacheType: NumThreads:0 GPULayers:[] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="11.8 GiB" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.363+02:00 level=INFO source=backend.go:315 msg="model weights" device=CPU size="1.1 GiB" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.364+02:00 level=INFO source=backend.go:342 msg="total memory" size="12.8 GiB" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.364+02:00 level=INFO source=sched.go:441 msg="Load failed" model=/home/et/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 error="memory layout cannot be allocated with num_gpu = 99" Sep 17 08:10:33 freebear ollama[1373]: time=2025-09-17T08:10:33.382+02:00 level=ERROR source=server.go:425 msg="llama runner terminated" error="signal: killed" ```
GiteaMirror commented 2026-05-04 20:44:46 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

Not sure what your last comment means.

The size of the mistral model with 32k context has been constant and more than 5GB since 0.3.0.

Sep 17 08:10:32 freebear ollama[1373]: time=2025-09-17T08:10:32.965+02:00 level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 library=cuda total="15.7 GiB" available="10.5 GiB"

Some other processes are using 5.2GiB of VRAM, leaving 10.5GiB available for ollama.

Sep 17 08:10:33 freebear ollama[1373]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12036.68 MiB on device 0: cudaMalloc failed: out of memory

ollama tried to allocate 11.75GiB to load the model and failed because of insufficient free VRAM.

nvidia-smi will show what other processes are using VRAM. Terminate some and ollama will be able to load the whole model into VRAM.

<!-- gh-comment-id:3301664943 --> @rick-github commented on GitHub (Sep 17, 2025): > Not sure what your last comment means. The size of the mistral model with 32k context has been constant and more than 5GB since 0.3.0. ``` Sep 17 08:10:32 freebear ollama[1373]: time=2025-09-17T08:10:32.965+02:00 level=INFO source=sched.go:540 msg="updated VRAM based on existing loaded models" gpu=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 library=cuda total="15.7 GiB" available="10.5 GiB" ``` Some other processes are using 5.2GiB of VRAM, leaving 10.5GiB available for ollama. ``` Sep 17 08:10:33 freebear ollama[1373]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12036.68 MiB on device 0: cudaMalloc failed: out of memory ``` ollama tried to allocate 11.75GiB to load the model and failed because of insufficient free VRAM. `nvidia-smi` will show what other processes are using VRAM. Terminate some and ollama will be able to load the whole model into VRAM.
GiteaMirror commented 2026-05-04 20:44:47 -05:00
Author
Owner
Copy Link

@etoulas commented on GitHub (Sep 17, 2025):

@rick-github Thanks for your help.

I just tried it again with the num_gpu=99 - that's the value that worked before the update.

The logs again show that not all VRAM is available but:

Sep 17 10:59:12 freebear ollama[1373]: time=2025-09-17T10:59:12.406+02:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 available="10.1 GiB" free="10.5 GiB" minimum="457.0 MiB" overhead="0 B"

nvidia-smi shows that ollama is occupying ca. 5 GB. But that appeared only when I started the chat (via Open WebUI). Right before only the two other processes where visible.

Wed Sep 17 11:02:34 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 Ti     On  |   00000000:01:00.0 Off |                  N/A |
|  0%   45C    P8             15W /  165W |    5110MiB /  16380MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1261      G   /usr/lib/xorg/Xorg                             85MiB |
|    0   N/A  N/A      1318      G   /usr/bin/sddm-greeter-qt6                      38MiB |
|    0   N/A  N/A     17864      C   /usr/local/bin/ollama                        4972MiB |
+-----------------------------------------------------------------------------------------+

Visually with nvtop it looks like this:

Image
<!-- gh-comment-id:3302042989 --> @etoulas commented on GitHub (Sep 17, 2025): @rick-github Thanks for your help. I just tried it again with the num_gpu=99 - that's the value that worked before the update. The logs again show that not all VRAM is available but: ``` Sep 17 10:59:12 freebear ollama[1373]: time=2025-09-17T10:59:12.406+02:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-c780ce2e-fd07-cc30-a3df-c3cd23cf8953 available="10.1 GiB" free="10.5 GiB" minimum="457.0 MiB" overhead="0 B" ``` nvidia-smi shows that ollama is occupying ca. 5 GB. But that appeared only when I started the chat (via Open WebUI). Right before only the two other processes where visible. ``` Wed Sep 17 11:02:34 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.163.01 Driver Version: 550.163.01 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4060 Ti On | 00000000:01:00.0 Off | N/A | | 0% 45C P8 15W / 165W | 5110MiB / 16380MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 1261 G /usr/lib/xorg/Xorg 85MiB | | 0 N/A N/A 1318 G /usr/bin/sddm-greeter-qt6 38MiB | | 0 N/A N/A 17864 C /usr/local/bin/ollama 4972MiB | +-----------------------------------------------------------------------------------------+ ``` Visually with nvtop it looks like this: <img width="1216" height="548" alt="Image" src="https://github.com/user-attachments/assets/7eca0b92-b8ac-4640-a2d7-44d52c3ce20c" />
GiteaMirror commented 2026-05-04 20:44:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

ollama ps
<!-- gh-comment-id:3302056306 --> @rick-github commented on GitHub (Sep 17, 2025): ``` ollama ps ```
GiteaMirror commented 2026-05-04 20:44:48 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

@rick-github we don't doubt your explanations and we clearly see that in the present behavior of Ollama v.0.11.11.

But we are certain that, until recently, all models run with Ollama took about the same about of (V)RAM as was the model's file size on the hard drive. Perhaps the older versions of Ollama were buggy (thanks to which we were able to run larger models on more modest HW). Perhaps the models were run with zero (or near-zero) context lengths before? (That could explain why I found DeepseekR1:70B to be extremely dumb.)

But there is a clear difference in Ollama's (V)RAM use before and after, perhaps some key Config setting was changed fairly recently.

<!-- gh-comment-id:3302071566 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): @rick-github we don't doubt your explanations and we clearly see that in the _present_ behavior of Ollama v.0.11.11. But we are certain that, until recently, all models run with Ollama took about the same about of (V)RAM as was the model's file size on the hard drive. Perhaps the older versions of Ollama were buggy (thanks to which we were able to run larger models on more modest HW). Perhaps the models were run with zero (or near-zero) context lengths before? (That could explain why I found DeepseekR1:70B to be extremely dumb.) But there is a clear difference in Ollama's (V)RAM use before and after, perhaps some key Config setting was changed fairly recently.
GiteaMirror commented 2026-05-04 20:44:48 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

ollama ps

FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous!

Image

Image

<!-- gh-comment-id:3302088115 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): > ``` > ollama ps > ``` FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous! ![Image](https://github.com/user-attachments/assets/97d87d30-488a-4d49-8b24-4797d699d110) ![Image](https://github.com/user-attachments/assets/d11cb380-3c93-4771-8502-3c7307e51e3f)
GiteaMirror commented 2026-05-04 20:44:49 -05:00
Author
Owner
Copy Link

@etoulas commented on GitHub (Sep 17, 2025):

@rick-github Thanks for the hint checking ollama ps. Well, I noticed I had an embedding model loaded (5.6 GB)... Will try again.

<!-- gh-comment-id:3302156467 --> @etoulas commented on GitHub (Sep 17, 2025): @rick-github Thanks for the hint checking ollama ps. Well, I noticed I had an embedding model loaded (5.6 GB)... Will try again.
GiteaMirror commented 2026-05-04 20:44:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

But we are certain that, until recently, all models run with Ollama took about the same about of (V)RAM as was the model's file size on the hard drive.

This has never been the case when using a context more then the default, as shown in https://github.com/ollama/ollama/issues/12305#issuecomment-3299975820.

The default context was 2048 until around 0.7.0 when it was increased to 4096. At the same time the default value for OLLAMA_NUM_PARALLEL was reduced, so while the size of an individual context buffer was increased, the overall memory allocation stayed the same.

But there is a clear difference in Ollama's (V)RAM use before and after, perhaps some key Config setting was changed fairly recently.

The size of the your model loads is explained by the size of the context buffer, and the failure of etoulas's model load is explained by the lack of free VRAM.

There has been a recent change in ollama's memory estimation logic when using the new engine. It should now be more accurate and reduces the need to override num_gpu when loading the model, although there are corner cases where this remains a valid tactic.

FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous!

The context buffer is the space where tokens are stored. If you want to store more tokens by increasing the context, you need more space. Different models require a different amount of space per token, so some models will be able to sustain a larger context than others. There are additional data structures that are proportional to the size of the context as well, so as context grows, so do they (primarily referencing the memory graph). gpt-oss, for example, has a quite efficient tokenization and can support 128k in 17GB (13G weights + 4G context/graph). Compare to mistral-nemo using 28G (7G weights + 21G context/graph).

NAME                   ID              SIZE     PROCESSOR    CONTEXT    UNTIL   
gpt-oss:20b            aa4295ac10c3    17 GB    100% GPU     131072     Forever    
mistral-nemo:latest    e7e06d107c6c    28 GB    100% GPU     131072     Forever
<!-- gh-comment-id:3302194339 --> @rick-github commented on GitHub (Sep 17, 2025): > But we are certain that, until recently, all models run with Ollama took about the same about of (V)RAM as was the model's file size on the hard drive. This has never been the case when using a context more then the default, as shown in https://github.com/ollama/ollama/issues/12305#issuecomment-3299975820. The default context was 2048 until around 0.7.0 when it was increased to 4096. At the same time the default value for OLLAMA_NUM_PARALLEL was reduced, so while the size of an individual context buffer was increased, the overall memory allocation stayed the same. > But there is a clear difference in Ollama's (V)RAM use before and after, perhaps some key Config setting was changed fairly recently. The size of the your model loads is explained by the size of the context buffer, and the failure of etoulas's model load is explained by the lack of free VRAM. There has been a recent change in ollama's memory estimation logic when using the new engine. It should now be more accurate and reduces the need to override `num_gpu` when loading the model, although there are corner cases where this remains a valid tactic. > FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous! The context buffer is the space where tokens are stored. If you want to store more tokens by increasing the context, you need more space. Different models require a different amount of space per token, so some models will be able to sustain a larger context than others. There are additional data structures that are proportional to the size of the context as well, so as context grows, so do they (primarily referencing the memory graph). gpt-oss, for example, has a quite efficient tokenization and can support 128k in 17GB (13G weights + 4G context/graph). Compare to mistral-nemo using 28G (7G weights + 21G context/graph). ```console NAME ID SIZE PROCESSOR CONTEXT UNTIL gpt-oss:20b aa4295ac10c3 17 GB 100% GPU 131072 Forever mistral-nemo:latest e7e06d107c6c 28 GB 100% GPU 131072 Forever ```
GiteaMirror commented 2026-05-04 20:44:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

Re: #12315

VRAM usage has not increased drastically with recent developments. You can test this by rolling back to earlier versions of ollama by running the installer, eg 0.11.10. It will preserve your models and you can experiment with the various versions, context sizes and configuration variables to examine the impact on memory requirements.

<!-- gh-comment-id:3302275816 --> @rick-github commented on GitHub (Sep 17, 2025): Re: #12315 VRAM usage has not increased drastically with recent developments. You can test this by rolling back to earlier versions of ollama by running the installer, eg [0.11.10](https://github.com/ollama/ollama/releases/download/v0.11.10/OllamaSetup.exe). It will preserve your models and you can experiment with the various versions, context sizes and configuration variables to examine the impact on memory requirements.
GiteaMirror commented 2026-05-04 20:44:52 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

There has been a recent change in ollama's memory estimation logic when using the new engine. It should now be more accurate and reduces the need to override num_gpu when loading the model, although there are corner cases where this remains a valid tactic.

FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous!

The context buffer is the space where tokens are stored. If you want to store more tokens by increasing the context, you need more space. Different models require a different amount of space per token, so some models will be able to sustain a larger context than others. There are additional data structures that are proportional to the size of the context as well, so as context grows, so do they (primarily referencing the memory graph). gpt-oss, for example, has a quite efficient tokenization and can support 128k in 17GB (13G weights + 4G context/graph). Compare to mistral-nemo using 28G (7G weights + 21G context/graph).

NAME ID SIZE PROCESSOR CONTEXT UNTIL
gpt-oss:20b aa4295ac10c3 17 GB 100% GPU 131072 Forever
mistral-nemo:latest e7e06d107c6c 28 GB 100% GPU 131072 Forever

Excellent explanation, thank you!

That clearly ruined the idea of Ollama as "LLM for the Masses"... :(

<!-- gh-comment-id:3302285861 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): > There has been a recent change in ollama's memory estimation logic when using the new engine. It should now be more accurate and reduces the need to override `num_gpu` when loading the model, although there are corner cases where this remains a valid tactic. > > > FIle size is 7 GB, but Ollama runs it in 28 GB of RAM (larger by a factor of 4). That's ridiculous! > > The context buffer is the space where tokens are stored. If you want to store more tokens by increasing the context, you need more space. Different models require a different amount of space per token, so some models will be able to sustain a larger context than others. There are additional data structures that are proportional to the size of the context as well, so as context grows, so do they (primarily referencing the memory graph). gpt-oss, for example, has a quite efficient tokenization and can support 128k in 17GB (13G weights + 4G context/graph). Compare to mistral-nemo using 28G (7G weights + 21G context/graph). > > NAME ID SIZE PROCESSOR CONTEXT UNTIL > gpt-oss:20b aa4295ac10c3 17 GB 100% GPU 131072 Forever > mistral-nemo:latest e7e06d107c6c 28 GB 100% GPU 131072 Forever Excellent explanation, thank you! That clearly ruined the idea of Ollama as "LLM for the Masses"... :(
GiteaMirror commented 2026-05-04 20:44:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

That clearly ruined the idea of Ollama as "LLM for the Masses"... :(

Why? You can run models that your hardware supports. That's always been the case.

<!-- gh-comment-id:3302296709 --> @rick-github commented on GitHub (Sep 17, 2025): > That clearly ruined the idea of Ollama as "LLM for the Masses"... :( Why? You can run models that your hardware supports. That's always been the case.
GiteaMirror commented 2026-05-04 20:44:57 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

Re: #12315

VRAM usage has not increased drastically with recent developments. You can test this by rolling back to earlier versions of ollama by running the installer, eg 0.11.10. It will preserve your models and you can experiment with the various versions, context sizes and configuration variables to examine the impact on memory requirements.

Maybe I used Ollama more actively before version 0.7.0, then. There have been many newer vesions recently.

But I am certain that, back then, that 2nd step (bite) in the RAM use by Ollama was not there. RAM usage may have been slightly above the model's file size on disk, but not DRASTICALLY larger.

That naturally leads to the need of having a good Required Memory Calculator feature for each model variant on the Ollama website (or each model's vendor should provide such an estimate for each model variant), as I just requested in Issue #12315 .

<!-- gh-comment-id:3302315769 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): > Re: [#12315](https://github.com/ollama/ollama/issues/12315) > > VRAM usage has not increased drastically with recent developments. You can test this by rolling back to earlier versions of ollama by running the installer, eg [0.11.10](https://github.com/ollama/ollama/releases/download/v0.11.10/OllamaSetup.exe). It will preserve your models and you can experiment with the various versions, context sizes and configuration variables to examine the impact on memory requirements. Maybe I used Ollama more actively before version 0.7.0, then. There have been many newer vesions recently. But I am certain that, back then, that 2nd step (bite) in the RAM use by Ollama was not there. RAM usage may have been slightly above the model's file size on disk, but not DRASTICALLY larger. That naturally leads to the need of having a good Required Memory Calculator feature for each model variant on the Ollama website (or each model's vendor should provide such an estimate for each model variant), as I just requested in Issue #12315 .
GiteaMirror commented 2026-05-04 20:45:01 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

That clearly ruined the idea of Ollama as "LLM for the Masses"... :(

Why? You can run models that your hardware supports. That's always been the case.

True, with one big caveat:

Presently, we are unable to run (some of) the Same models on the Same hardware that we were able to run before.

<!-- gh-comment-id:3302323609 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): > > That clearly ruined the idea of Ollama as "LLM for the Masses"... :( > > Why? You can run models that your hardware supports. That's always been the case. True, with one big caveat: Presently, we are unable to run (some of) the Same models on the Same hardware that we were able to run before.
GiteaMirror commented 2026-05-04 20:45:04 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

But I am certain that, back then, that 2nd step (bite) in the RAM use by Ollama was not there. RAM usage may have been slightly above the model's file size on disk, but not DRASTICALLY larger.

If memory size was not drastically larger than the model's size on disk then you were not using a context size of 128k. The RAM bite I explained above.

Presently, we are unable to run (some of) the Same models on the Same hardware that we were able to do before.

This is incorrect. You can run the same models on the same hardware using the same context size.

<!-- gh-comment-id:3302340986 --> @rick-github commented on GitHub (Sep 17, 2025): > But I am certain that, back then, that 2nd step (bite) in the RAM use by Ollama was not there. RAM usage may have been slightly above the model's file size on disk, but not DRASTICALLY larger. If memory size was not drastically larger than the model's size on disk then you were not using a context size of 128k. The RAM bite I explained above. > Presently, we are unable to run (some of) the Same models on the Same hardware that we were able to do before. This is incorrect. You can run the same models on the same hardware using the same context size.
GiteaMirror commented 2026-05-04 20:45:04 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

It's possible you were using a version prior to 0.5.13 and had configured OLLAMA_CONTEXT_LENGTH to set a 128k context buffer. That variable is only supported from 0.5.13 onwards, so if you were running (for example) 0.5.12 and had set the context length to 128k with that variable, ollama would have continued to use the default value of 2048. In that case, the size of the model shown in ollama ps would have only been slightly larger than the disk size.

<!-- gh-comment-id:3302374114 --> @rick-github commented on GitHub (Sep 17, 2025): It's possible you were using a version prior to 0.5.13 and had configured `OLLAMA_CONTEXT_LENGTH` to set a 128k context buffer. That variable is only supported from 0.5.13 onwards, so if you were running (for example) 0.5.12 and had set the context length to 128k with that variable, ollama would have continued to use the default value of 2048. In that case, the size of the model shown in `ollama ps` would have only been slightly larger than the disk size.
GiteaMirror commented 2026-05-04 20:45:06 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

@rick-github I rolled back to Ollama 0.7.0, here is how mistral-nemo works now:

Image

Image

Image

Image

<!-- gh-comment-id:3302389987 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): @rick-github I rolled back to Ollama 0.7.0, here is how mistral-nemo works now: ![Image](https://github.com/user-attachments/assets/b8ed1f5d-9faf-43f0-bbef-c8e3e14ed4b3) ![Image](https://github.com/user-attachments/assets/41a9c844-81b4-4d38-84a9-da0ec07950a2) ![Image](https://github.com/user-attachments/assets/36be2a4b-af0a-41ef-94da-964171fd22ff) ![Image](https://github.com/user-attachments/assets/3b3d911b-bb75-440c-b679-5a1630ae7750)
GiteaMirror commented 2026-05-04 20:45:14 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

$ ollama -v
ollama version is 0.7.0
Warning: client version is 0.11.10
$ ollama run mistral-nemo
>>> 
$ ollama ps
NAME                   ID              SIZE      PROCESSOR    CONTEXT    UNTIL   
mistral-nemo:latest    e7e06d107c6c    8.7 GB    100% GPU     0          Forever    
$ ollama run mistral-nemo
>>> /set parameter num_ctx 131072
Set parameter 'num_ctx' to '131072'
>>> hello
Hello! How can I assist you today? If you have any questions or just want to chat, feel free! 😊

>>> 
$ ollama ps
NAME                   ID              SIZE     PROCESSOR    CONTEXT    UNTIL   
mistral-nemo:latest    e7e06d107c6c    38 GB    100% GPU     0          Forever

Loading the model uses the default context, so the allocated VRAM is small at 8.7G. (It's a bit larger in your case because there's overhead from distributing a model across multiple devices, GPU and CPU). When the context is set larger, the allocated VRAM increases as expected to 38G.

<!-- gh-comment-id:3302414885 --> @rick-github commented on GitHub (Sep 17, 2025): ```console $ ollama -v ollama version is 0.7.0 Warning: client version is 0.11.10 $ ollama run mistral-nemo >>> $ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL mistral-nemo:latest e7e06d107c6c 8.7 GB 100% GPU 0 Forever $ ollama run mistral-nemo >>> /set parameter num_ctx 131072 Set parameter 'num_ctx' to '131072' >>> hello Hello! How can I assist you today? If you have any questions or just want to chat, feel free! 😊 >>> $ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL mistral-nemo:latest e7e06d107c6c 38 GB 100% GPU 0 Forever ``` Loading the model uses the default context, so the allocated VRAM is small at 8.7G. (It's a bit larger in your case because there's overhead from distributing a model across multiple devices, GPU and CPU). When the context is set larger, the allocated VRAM increases as expected to 38G.
GiteaMirror commented 2026-05-04 20:45:16 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

@rick-github you seem to be running on Linux. Your versions of ollama and the client differ.

In my Windows case, ollama -v returns just 0.7.0 and ollama ps does not show the CONTEXT collumn. Is there a way to add/see that?

<!-- gh-comment-id:3302431817 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): @rick-github you seem to be running on Linux. Your versions of ollama and the client differ. In my Windows case, ollama -v returns just 0.7.0 and ollama ps does not show the CONTEXT collumn. Is there a way to add/see that?
GiteaMirror commented 2026-05-04 20:45:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

Your versions of ollama and the client differ.

The server component (at 0.7.0) is running in a docker container for easy management. The client component is in my local filesystem and has been upgraded to 0.11.10. The different versions are mostly compatible at the API level so I don't bother with matching versions for these sorts of simple tests.

In my Windows case, ollama -v returns just 0.7.0 and ollama ps does not show the CONTEXT collumn

The API only supports returning the context size of the loaded model from around version 0.10.0, which is why in my examples above you see that the context is 0 for older versions - the information is just not part of the response to the API call. So 0.7.0 is not capable of displaying that information, you need to look at the logs. If you look for the string "llama_context: n_ctx_per_seq" you will see the size of the context buffer per processing slot.

<!-- gh-comment-id:3302461108 --> @rick-github commented on GitHub (Sep 17, 2025): > Your versions of ollama and the client differ. The server component (at 0.7.0) is running in a docker container for easy management. The client component is in my local filesystem and has been upgraded to 0.11.10. The different versions are mostly compatible at the API level so I don't bother with matching versions for these sorts of simple tests. > In my Windows case, ollama -v returns just 0.7.0 and ollama ps does not show the CONTEXT collumn The API only supports returning the context size of the loaded model from around version 0.10.0, which is why in my examples above you see that the context is 0 for older versions - the information is just not part of the response to the API call. So 0.7.0 is not capable of displaying that information, you need to look at the logs. If you look for the string "llama_context: n_ctx_per_seq" you will see the size of the context buffer per processing slot.
GiteaMirror commented 2026-05-04 20:45:23 -05:00
Author
Owner
Copy Link

@Panican-Whyasker commented on GitHub (Sep 17, 2025):

The API only supports returning the context size of the loaded model from around version 0.10.0, which is why in my examples above you see that the context is 0 for older versions - the information is just not part of the response to the API call. So 0.7.0 is not capable of displaying that information, you need to look at the logs. If you look for the string "llama_context: n_ctx_per_seq" you will see the size of the context buffer per processing slot.

Thank you, indeed:
...
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 4096
llama_context: n_ctx_per_seq = 4096
...
llama_context: n_ctx_per_seq (4096) < n_ctx_train (1024000) -- the full capacity of the model will not be utilized.

For me, with that, the Issue is closed. Pls. feel free to close it, or I can do so.

<!-- gh-comment-id:3302488698 --> @Panican-Whyasker commented on GitHub (Sep 17, 2025): > The API only supports returning the context size of the loaded model from around version 0.10.0, which is why in my examples above you see that the context is 0 for older versions - the information is just not part of the response to the API call. So 0.7.0 is not capable of displaying that information, you need to look at the logs. If you look for the string "llama_context: n_ctx_per_seq" you will see the size of the context buffer per processing slot. Thank you, indeed: ... llama_context: constructing llama_context llama_context: n_seq_max = 1 llama_context: n_ctx = 4096 llama_context: n_ctx_per_seq = 4096 ... llama_context: n_ctx_per_seq (4096) < n_ctx_train (1024000) -- the full capacity of the model will not be utilized. For me, with that, the Issue is closed. Pls. feel free to close it, or I can do so.
GiteaMirror commented 2026-05-04 20:45:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 17, 2025):

@etoulas If you continue to have problems, please open another issue.

<!-- gh-comment-id:3302498571 --> @rick-github commented on GitHub (Sep 17, 2025): @etoulas If you continue to have problems, please open another issue.
GiteaMirror commented 2026-05-04 20:45:25 -05:00
Author
Owner
Copy Link

@etoulas commented on GitHub (Sep 17, 2025):

@rick-github Thanks for your support.

I narrowed it down btw. It seems that Open WebUI triggers the embedding model probably because of my very long prompt?! Not sure, but it's definitely another issue - probably because of me messing with the configuration...

Thanks for the excellent explanation of the context size and VRAM need. That was insightful for me! 🙏

<!-- gh-comment-id:3302555368 --> @etoulas commented on GitHub (Sep 17, 2025): @rick-github Thanks for your support. I narrowed it down btw. It seems that Open WebUI triggers the embedding model probably because of my very long prompt?! Not sure, but it's definitely another issue - probably because of me messing with the configuration... Thanks for the excellent explanation of the context size and VRAM need. That was insightful for me! 🙏
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#70235
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?