github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-23 06:01:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10040] System memory leak. Gemma3 #53097

New Issue
Closed
opened 2026-04-29 01:55:30 -05:00 by GiteaMirror · 15 comments
GiteaMirror commented 2026-04-29 01:55:30 -05:00
Owner
Copy Link

Originally created by @murzein on GitHub (Mar 29, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10040

Originally assigned to: @jessegross on GitHub.

What is the issue?

When using Gemma3, system memory is fully consumed over time.
When using the qwen2.5 model, approximately 1.5Gb is consumed.

Image

Image

Relevant log output

Mar 29 16:13:35 ai systemd[1]: Stopping ollama.service - Ollama Service...
Mar 29 16:13:35 ai systemd[1]: ollama.service: Deactivated successfully.
Mar 29 16:13:35 ai systemd[1]: Stopped ollama.service - Ollama Service.
Mar 29 16:13:35 ai systemd[1]: ollama.service: Consumed 1min 30.648s CPU time, 359.2M memory peak, 0B memory swap peak.
Mar 29 16:13:35 ai systemd[1]: Started ollama.service - Ollama Service.
Mar 29 16:13:35 ai ollama[143819]: 2025/03/29 16:13:35 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.648+05:00 level=INFO source=images.go:432 msg="total blobs: 19"
Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.648+05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.649+05:00 level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.3)"
Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.649+05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Mar 29 16:13:36 ai ollama[143819]: time=2025-03-29T16:13:36.035+05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.5 GiB" available="23.3 GiB"
Mar 29 16:13:36 ai ollama[143819]: time=2025-03-29T16:13:36.892+05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b gpu=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d parallel=1 available=24997330944 required="9.7 GiB"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.164+05:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="29.7 GiB" free_swap="8.0 GiB"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[23.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.7 GiB" memory.required.partial="9.7 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[9.7 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB" projector.weights="814.6 MiB" projector.graph="0 B"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=INFO source=server.go:185 msg="enabling flash attention"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=WARN source=server.go:193 msg="kv cache type not supported by model" type=""
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.489+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.501+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 35711"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.522+05:00 level=INFO source=runner.go:765 msg="starting ollama engine"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.522+05:00 level=INFO source=runner.go:828 msg="Server listening on 127.0.0.1:35711"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.575+05:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.575+05:00 level=INFO source=ggml.go:69 msg="" architecture=gemma3 file_type=Q4_K_M name="Gemma 3 12b It" description="" num_tensors=626 num_key_values=45
Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: found 1 CUDA devices:
Mar 29 16:13:37 ai ollama[143819]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Mar 29 16:13:37 ai ollama[143819]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Mar 29 16:13:37 ai ollama[143819]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.638+05:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.765+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.816+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CUDA0 size="6.8 GiB"
Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.816+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CPU size="787.5 MiB"
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CPU buffer_type=CUDA_Host
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.828+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 29 16:13:39 ai ollama[143819]: time=2025-03-29T16:13:39.020+05:00 level=INFO source=server.go:619 msg="llama runner started in 1.51 seconds"
Mar 29 16:13:39 ai ollama[143819]: [GIN] 2025/03/29 - 16:13:39 | 200 |  2.669877938s |       127.0.0.1 | POST     "/api/generate"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.6.3

Originally created by @murzein on GitHub (Mar 29, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10040 Originally assigned to: @jessegross on GitHub. ### What is the issue? When using Gemma3, system memory is fully consumed over time. When using the qwen2.5 model, approximately 1.5Gb is consumed. ![Image](https://github.com/user-attachments/assets/0a333e7f-97c8-4142-b461-2336f2a8b1f1) ![Image](https://github.com/user-attachments/assets/601e291f-ae50-49be-a908-2d94e9b7f57f) ### Relevant log output ```shell Mar 29 16:13:35 ai systemd[1]: Stopping ollama.service - Ollama Service... Mar 29 16:13:35 ai systemd[1]: ollama.service: Deactivated successfully. Mar 29 16:13:35 ai systemd[1]: Stopped ollama.service - Ollama Service. Mar 29 16:13:35 ai systemd[1]: ollama.service: Consumed 1min 30.648s CPU time, 359.2M memory peak, 0B memory swap peak. Mar 29 16:13:35 ai systemd[1]: Started ollama.service - Ollama Service. Mar 29 16:13:35 ai ollama[143819]: 2025/03/29 16:13:35 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.648+05:00 level=INFO source=images.go:432 msg="total blobs: 19" Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.648+05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.649+05:00 level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.3)" Mar 29 16:13:35 ai ollama[143819]: time=2025-03-29T16:13:35.649+05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Mar 29 16:13:36 ai ollama[143819]: time=2025-03-29T16:13:36.035+05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.5 GiB" available="23.3 GiB" Mar 29 16:13:36 ai ollama[143819]: time=2025-03-29T16:13:36.892+05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b gpu=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d parallel=1 available=24997330944 required="9.7 GiB" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.164+05:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="29.7 GiB" free_swap="8.0 GiB" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[23.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.7 GiB" memory.required.partial="9.7 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[9.7 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB" projector.weights="814.6 MiB" projector.graph="0 B" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=INFO source=server.go:185 msg="enabling flash attention" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.439+05:00 level=WARN source=server.go:193 msg="kv cache type not supported by model" type="" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.489+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.501+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.505+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.513+05:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 35711" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.514+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.522+05:00 level=INFO source=runner.go:765 msg="starting ollama engine" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.522+05:00 level=INFO source=runner.go:828 msg="Server listening on 127.0.0.1:35711" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.575+05:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.575+05:00 level=INFO source=ggml.go:69 msg="" architecture=gemma3 file_type=Q4_K_M name="Gemma 3 12b It" description="" num_tensors=626 num_key_values=45 Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Mar 29 16:13:37 ai ollama[143819]: ggml_cuda_init: found 1 CUDA devices: Mar 29 16:13:37 ai ollama[143819]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Mar 29 16:13:37 ai ollama[143819]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Mar 29 16:13:37 ai ollama[143819]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.638+05:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.765+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.816+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CUDA0 size="6.8 GiB" Mar 29 16:13:37 ai ollama[143819]: time=2025-03-29T16:13:37.816+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CPU size="787.5 MiB" Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CPU buffer_type=CUDA_Host Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.824+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.828+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.831+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 29 16:13:38 ai ollama[143819]: time=2025-03-29T16:13:38.840+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 29 16:13:39 ai ollama[143819]: time=2025-03-29T16:13:39.020+05:00 level=INFO source=server.go:619 msg="llama runner started in 1.51 seconds" Mar 29 16:13:39 ai ollama[143819]: [GIN] 2025/03/29 - 16:13:39 | 200 | 2.669877938s | 127.0.0.1 | POST "/api/generate" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.6.3
GiteaMirror added the bug label 2026-04-29 01:55:31 -05:00
GiteaMirror commented 2026-04-29 01:55:33 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 29, 2025):

Output of

ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama)
<!-- gh-comment-id:2763637117 --> @rick-github commented on GitHub (Mar 29, 2025): Output of ``` ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama) ```
GiteaMirror commented 2026-04-29 01:55:33 -05:00
Author
Owner
Copy Link

@murzein commented on GitHub (Mar 30, 2025):

New test

Restart ollama server.
After 100 requests:

Image

After 600 requests:

Image

journalctl -u ollama --no-pager --follow --pager-end

Mar 30 11:02:53 ai systemd[1]: Stopping ollama.service - Ollama Service...
Mar 30 11:02:53 ai systemd[1]: ollama.service: Deactivated successfully.
Mar 30 11:02:53 ai systemd[1]: Stopped ollama.service - Ollama Service.
Mar 30 11:02:53 ai systemd[1]: ollama.service: Consumed 16min 20.107s CPU time, 8.5G memory peak, 0B memory swap peak.
Mar 30 11:02:53 ai systemd[1]: Started ollama.service - Ollama Service.
Mar 30 11:02:53 ai ollama[885282]: 2025/03/30 11:02:53 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=images.go:432 msg="total blobs: 19"
Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.3)"
Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Mar 30 11:02:54 ai ollama[885282]: time=2025-03-30T11:02:54.279+05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.5 GiB" available="23.3 GiB"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.093+05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b gpu=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d parallel=1 available=24997330944 required="9.7 GiB"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.361+05:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="29.8 GiB" free_swap="8.0 GiB"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[23.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.7 GiB" memory.required.partial="9.7 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[9.7 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB" projector.weights="814.6 MiB" projector.graph="0 B"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=INFO source=server.go:185 msg="enabling flash attention"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=WARN source=server.go:193 msg="kv cache type not supported by model" type=""
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.677+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.685+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 37719"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.704+05:00 level=INFO source=runner.go:765 msg="starting ollama engine"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.705+05:00 level=INFO source=runner.go:828 msg="Server listening on 127.0.0.1:37719"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.754+05:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.754+05:00 level=INFO source=ggml.go:69 msg="" architecture=gemma3 file_type=Q4_K_M name="Gemma 3 12b It" description="" num_tensors=626 num_key_values=45
Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: found 1 CUDA devices:
Mar 30 11:02:55 ai ollama[885282]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Mar 30 11:02:55 ai ollama[885282]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Mar 30 11:02:55 ai ollama[885282]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.809+05:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.947+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.977+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CUDA0 size="6.8 GiB"
Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.977+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CPU size="787.5 MiB"
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CPU buffer_type=CUDA_Host
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.958+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 30 11:03:00 ai ollama[885282]: time=2025-03-30T11:03:00.207+05:00 level=INFO source=server.go:619 msg="llama runner started in 4.51 seconds"

ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama)

 885282       1 11:02 /usr/local/bin/ollama serve
 885333  885282 11:02 /usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 37719
<!-- gh-comment-id:2764409710 --> @murzein commented on GitHub (Mar 30, 2025): ### New test Restart ollama server. **After 100 requests:** ![Image](https://github.com/user-attachments/assets/5c734203-a5ce-400e-bd82-15ec3ec4ebd9) **After 600 requests:** ![Image](https://github.com/user-attachments/assets/f526749c-54ba-4498-9287-7d4f08545a67) **journalctl -u ollama --no-pager --follow --pager-end** ``` Mar 30 11:02:53 ai systemd[1]: Stopping ollama.service - Ollama Service... Mar 30 11:02:53 ai systemd[1]: ollama.service: Deactivated successfully. Mar 30 11:02:53 ai systemd[1]: Stopped ollama.service - Ollama Service. Mar 30 11:02:53 ai systemd[1]: ollama.service: Consumed 16min 20.107s CPU time, 8.5G memory peak, 0B memory swap peak. Mar 30 11:02:53 ai systemd[1]: Started ollama.service - Ollama Service. Mar 30 11:02:53 ai ollama[885282]: 2025/03/30 11:02:53 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=images.go:432 msg="total blobs: 19" Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.3)" Mar 30 11:02:53 ai ollama[885282]: time=2025-03-30T11:02:53.832+05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Mar 30 11:02:54 ai ollama[885282]: time=2025-03-30T11:02:54.279+05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="23.5 GiB" available="23.3 GiB" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.093+05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b gpu=GPU-cf63f28b-b0a5-31c0-60a3-fde622d8ad0d parallel=1 available=24997330944 required="9.7 GiB" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.361+05:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="29.8 GiB" free_swap="8.0 GiB" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[23.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.7 GiB" memory.required.partial="9.7 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[9.7 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB" projector.weights="814.6 MiB" projector.graph="0 B" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=INFO source=server.go:185 msg="enabling flash attention" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.627+05:00 level=WARN source=server.go:193 msg="kv cache type not supported by model" type="" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.677+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.685+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.687+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.693+05:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 37719" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.694+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.704+05:00 level=INFO source=runner.go:765 msg="starting ollama engine" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.705+05:00 level=INFO source=runner.go:828 msg="Server listening on 127.0.0.1:37719" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.754+05:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.754+05:00 level=INFO source=ggml.go:69 msg="" architecture=gemma3 file_type=Q4_K_M name="Gemma 3 12b It" description="" num_tensors=626 num_key_values=45 Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Mar 30 11:02:55 ai ollama[885282]: ggml_cuda_init: found 1 CUDA devices: Mar 30 11:02:55 ai ollama[885282]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Mar 30 11:02:55 ai ollama[885282]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Mar 30 11:02:55 ai ollama[885282]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.809+05:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.947+05:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.977+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CUDA0 size="6.8 GiB" Mar 30 11:02:55 ai ollama[885282]: time=2025-03-30T11:02:55.977+05:00 level=INFO source=ggml.go:291 msg="model weights" buffer=CPU size="787.5 MiB" Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=INFO source=ggml.go:383 msg="compute graph" backend=CPU buffer_type=CUDA_Host Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.955+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.958+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.961+05:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 30 11:02:59 ai ollama[885282]: time=2025-03-30T11:02:59.972+05:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 30 11:03:00 ai ollama[885282]: time=2025-03-30T11:03:00.207+05:00 level=INFO source=server.go:619 msg="llama runner started in 4.51 seconds" ``` **ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama)** ``` 885282 1 11:02 /usr/local/bin/ollama serve 885333 885282 11:02 /usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-fc57f67efa46d711c346e587cbef7d049e95f3df8db2eb2271153343ef0acc7b --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --flash-attn --parallel 1 --port 37719 ```
GiteaMirror commented 2026-04-29 01:55:34 -05:00
Author
Owner
Copy Link

@superles commented on GitHub (Mar 30, 2025):

Same problem with gemma3:12b

ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama)

298497 1 20:49 /usr/local/bin/ollama serve 298560 298497 20:57 /usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-e8ad13eff07a78d89926e9e8b882317d082ef5bf9768ad7b50fcdbbcd63748de --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --parallel 4 --port 35149

<!-- gh-comment-id:2764684735 --> @superles commented on GitHub (Mar 30, 2025): Same problem with gemma3:12b ps wwho pid,ppid,start_time,cmd kstart_time p$(pidof ollama) `298497 1 20:49 /usr/local/bin/ollama serve 298560 298497 20:57 /usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-e8ad13eff07a78d89926e9e8b882317d082ef5bf9768ad7b50fcdbbcd63748de --ctx-size 8192 --batch-size 512 --n-gpu-layers 49 --threads 6 --parallel 4 --port 35149`
GiteaMirror commented 2026-04-29 01:55:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2025):

I was only able to replicate this in a docker container. Running bare metal, even with the binaries and configuration the same as the docker image, didn't produce the same results.

Image

Profiling the runner didn't show anything unusual in the memory usage. It's likely that the memory is being allocated in the C++ portions of the runner and so is invisible to the go profiling tools.

Image

$ go tool pprof -text http://runner:36313/pprof/heap 
Fetching profile over HTTP from http://runner:36313/pprof/heap
Saved profile in /home/rick/pprof/pprof.ollama.alloc_objects.alloc_space.inuse_objects.inuse_space.084.pb.gz
File: ollama
Build ID: 08fd1089b56b1ad7f54456f21a430c5223470f2f
Type: inuse_space
Time: Mar 30, 2025 at 9:52pm (CEST)
Showing nodes accounting for 78.75MB, 100% of 78.75MB total
      flat  flat%   sum%        cum   cum%
   30.39MB 38.59% 38.59%    39.89MB 50.65%  github.com/ollama/ollama/fs/ggml.readGGUFArray
   12.83MB 16.29% 54.88%    12.83MB 16.29%  github.com/ollama/ollama/model.(*Vocabulary).Encode.func1
    9.52MB 12.09% 66.97%     9.52MB 12.09%  runtime.allocm
    8.50MB 10.79% 77.76%     8.50MB 10.79%  github.com/ollama/ollama/fs/ggml.readGGUFString
    8.02MB 10.18% 87.94%     8.02MB 10.18%  github.com/ollama/ollama/fs/ggml.KV.Strings
    2.33MB  2.95% 90.90%     2.33MB  2.95%  github.com/ollama/ollama/fs/ggml.KV.Uints
    1.51MB  1.92% 92.81%     1.51MB  1.92%  regexp.onePassCopy
    1.16MB  1.47% 94.28%     1.16MB  1.47%  github.com/ollama/ollama/ml/backend/ggml.(*Tensor).Floats
       1MB  1.27% 95.55%     1.50MB  1.90%  runtime.gcBgMarkWorker
       1MB  1.27% 96.82%        1MB  1.27%  encoding/binary.Read
    0.50MB  0.64% 97.46%     0.50MB  0.64%  unicode.map.init.1
    0.50MB  0.64% 98.09%     0.50MB  0.64%  regexp/syntax.(*compiler).inst (inline)
    0.50MB  0.64% 98.73%     0.50MB  0.64%  google.golang.org/protobuf/internal/impl.legacyLoadMessageInfo
    0.50MB  0.64% 99.37%     0.50MB  0.64%  runtime.acquireSudog
    0.50MB  0.63%   100%     0.50MB  0.63%  github.com/ollama/ollama/ml/backend/ggml._Cfunc_GoString (inline)
         0     0%   100%     1.51MB  1.92%  github.com/go-playground/validator/v10.init
         0     0%   100%     0.50MB  0.64%  github.com/go-playground/validator/v10.init.0

However, straceing the runner showed that it is progressively mapping 128MB chunks into its address space:

mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1f4000000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1ec000000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1d4000000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1b4000000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1a0000000

It looks like at some point the map is not freed, so the allocator moves on to the next chunk:

$ grep -h 0x75f1f4000000 *  | cut -d'(' -f1 | sort | uniq -c
     43 mmap
     43 mprotect
     42 munmap

It seems the runtime is just busy extending the RSS. No clue as to why it happens (for me) only inside a container, whereas it seems to affect others who are running it out of systemd.

<!-- gh-comment-id:2764746825 --> @rick-github commented on GitHub (Mar 30, 2025): I was only able to replicate this in a docker container. Running bare metal, even with the binaries and configuration the same as the docker image, didn't produce the same results. ![Image](https://github.com/user-attachments/assets/19306e4e-81cc-4677-a2d2-3c61833a5f52) Profiling the runner didn't show anything unusual in the memory usage. It's likely that the memory is being allocated in the C++ portions of the runner and so is invisible to the go profiling tools. ![Image](https://github.com/user-attachments/assets/fb5ecbcc-4e54-4506-9c0b-48eacdcf1362) ```sh $ go tool pprof -text http://runner:36313/pprof/heap Fetching profile over HTTP from http://runner:36313/pprof/heap Saved profile in /home/rick/pprof/pprof.ollama.alloc_objects.alloc_space.inuse_objects.inuse_space.084.pb.gz File: ollama Build ID: 08fd1089b56b1ad7f54456f21a430c5223470f2f Type: inuse_space Time: Mar 30, 2025 at 9:52pm (CEST) Showing nodes accounting for 78.75MB, 100% of 78.75MB total flat flat% sum% cum cum% 30.39MB 38.59% 38.59% 39.89MB 50.65% github.com/ollama/ollama/fs/ggml.readGGUFArray 12.83MB 16.29% 54.88% 12.83MB 16.29% github.com/ollama/ollama/model.(*Vocabulary).Encode.func1 9.52MB 12.09% 66.97% 9.52MB 12.09% runtime.allocm 8.50MB 10.79% 77.76% 8.50MB 10.79% github.com/ollama/ollama/fs/ggml.readGGUFString 8.02MB 10.18% 87.94% 8.02MB 10.18% github.com/ollama/ollama/fs/ggml.KV.Strings 2.33MB 2.95% 90.90% 2.33MB 2.95% github.com/ollama/ollama/fs/ggml.KV.Uints 1.51MB 1.92% 92.81% 1.51MB 1.92% regexp.onePassCopy 1.16MB 1.47% 94.28% 1.16MB 1.47% github.com/ollama/ollama/ml/backend/ggml.(*Tensor).Floats 1MB 1.27% 95.55% 1.50MB 1.90% runtime.gcBgMarkWorker 1MB 1.27% 96.82% 1MB 1.27% encoding/binary.Read 0.50MB 0.64% 97.46% 0.50MB 0.64% unicode.map.init.1 0.50MB 0.64% 98.09% 0.50MB 0.64% regexp/syntax.(*compiler).inst (inline) 0.50MB 0.64% 98.73% 0.50MB 0.64% google.golang.org/protobuf/internal/impl.legacyLoadMessageInfo 0.50MB 0.64% 99.37% 0.50MB 0.64% runtime.acquireSudog 0.50MB 0.63% 100% 0.50MB 0.63% github.com/ollama/ollama/ml/backend/ggml._Cfunc_GoString (inline) 0 0% 100% 1.51MB 1.92% github.com/go-playground/validator/v10.init 0 0% 100% 0.50MB 0.64% github.com/go-playground/validator/v10.init.0 ``` However, `strace`ing the runner showed that it is progressively mapping 128MB chunks into its address space: ``` mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1f4000000 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1ec000000 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1d4000000 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1b4000000 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x75f1a0000000 ``` It looks like at some point the map is not freed, so the allocator moves on to the next chunk: ```sh $ grep -h 0x75f1f4000000 * | cut -d'(' -f1 | sort | uniq -c 43 mmap 43 mprotect 42 munmap ``` It seems the runtime is just busy extending the RSS. No clue as to why it happens (for me) only inside a container, whereas it seems to affect others who are running it out of `systemd`.
GiteaMirror commented 2026-04-29 01:55:36 -05:00
Author
Owner
Copy Link

@wisepmlin commented on GitHub (Mar 31, 2025):

Mac Studio M2 MAX的温度飘升到100度。好烫手，不知道会不会烧掉电脑。不敢运行Gemma3

<!-- gh-comment-id:2767152321 --> @wisepmlin commented on GitHub (Mar 31, 2025): Mac Studio M2 MAX的温度飘升到100度。好烫手，不知道会不会烧掉电脑。不敢运行Gemma3
GiteaMirror commented 2026-04-29 01:55:37 -05:00
Author
Owner
Copy Link

@michal0000000 commented on GitHub (Apr 3, 2025):

Tested v0.6.4 pre-release on rtx 5090 in Docker and the issue is still there.

<!-- gh-comment-id:2775831897 --> @michal0000000 commented on GitHub (Apr 3, 2025): Tested v0.6.4 pre-release on rtx 5090 in Docker and the issue is still there.
GiteaMirror commented 2026-04-29 01:55:38 -05:00
Author
Owner
Copy Link

@Slymi commented on GitHub (Apr 3, 2025):

v0.6.4 pre-release, I am having the same issue where Gemma 3 would OOM in docker (Truenas) after using it for a while.

What the kernel reported (part of it):
Cannot map memory with base addr 0x7f6d55400000 and size of 0x200 pages.

<!-- gh-comment-id:2775940510 --> @Slymi commented on GitHub (Apr 3, 2025): v0.6.4 pre-release, I am having the same issue where Gemma 3 would OOM in docker (Truenas) after using it for a while. What the kernel reported (part of it): Cannot map memory with base addr 0x7f6d55400000 and size of 0x200 pages.
GiteaMirror commented 2026-04-29 01:55:39 -05:00
Author
Owner
Copy Link

@murzein commented on GitHub (Apr 8, 2025):

When FLASH_ATTENTION is enabled, memory fills up very quickly.
If FLASH_ATTENTION is disabled, memory fills up slowly.

<!-- gh-comment-id:2786753930 --> @murzein commented on GitHub (Apr 8, 2025): When FLASH_ATTENTION is enabled, memory fills up very quickly. If FLASH_ATTENTION is disabled, memory fills up slowly.
GiteaMirror commented 2026-04-29 01:55:42 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Apr 9, 2025):

I see the cause of this issue - we are freeing the metadata for tensors that are inputs to the models (such as input tokens) but not the actual data itself. Previously, we recycled the data buffers across passes so we didn't have this issue. With flash attention, we have more and larger buffers, which is why it leaks faster.

The fix will still take a little while to avoid causing other side effects.

<!-- gh-comment-id:2787912767 --> @jessegross commented on GitHub (Apr 9, 2025): I see the cause of this issue - we are freeing the metadata for tensors that are inputs to the models (such as input tokens) but not the actual data itself. Previously, we recycled the data buffers across passes so we didn't have this issue. With flash attention, we have more and larger buffers, which is why it leaks faster. The fix will still take a little while to avoid causing other side effects.
GiteaMirror commented 2026-04-29 01:55:44 -05:00
Author
Owner
Copy Link

@NeuroinformaticaFBF commented on GitHub (Apr 10, 2025):

I'm having a similar issue. I'm using gemma3 to analyze 128 images. Almost at the end (image 124/128), I get:


Traceback (most recent call last):
  File "[my_home]/FOOD/SCRIPTS/auto_QC_csv_loop_v5.py", line 142, in <module>
    response = ollama.chat(
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 333, in chat
    return self._request(
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 178, in _request
    return cls(**self._request_raw(*args, **kwargs).json())
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 122, in _request_raw
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError: an error was encountered while running the model: GGML_ASSERT(ctx->mem_buffer != NULL) failed (status code: 500)

On the same set of images, but with a different prompt, I get:


Traceback (most recent call last):
  File "[my_home]/FOOD/SCRIPTS/auto_QC_csv_loop_v5.py", line 142, in <module>
    response = ollama.chat(
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 333, in chat
    return self._request(
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 178, in _request
    return cls(**self._request_raw(*args, **kwargs).json())
  File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 122, in _request_raw
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError: POST predict: Post "http://127.0.0.1:41691/completion": EOF (status code: 500)

Output of ollama ps:

NAME             ID              SIZE      PROCESSOR    UNTIL              
gemma3:latest    a2af6cc3eb7f    6.2 GB    100% GPU     2 minutes from now

Hope this helps

<!-- gh-comment-id:2791961546 --> @NeuroinformaticaFBF commented on GitHub (Apr 10, 2025): I'm having a similar issue. I'm using gemma3 to analyze 128 images. Almost at the end (image 124/128), I get: ``` Traceback (most recent call last): File "[my_home]/FOOD/SCRIPTS/auto_QC_csv_loop_v5.py", line 142, in <module> response = ollama.chat( File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 333, in chat return self._request( File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 178, in _request return cls(**self._request_raw(*args, **kwargs).json()) File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 122, in _request_raw raise ResponseError(e.response.text, e.response.status_code) from None ollama._types.ResponseError: an error was encountered while running the model: GGML_ASSERT(ctx->mem_buffer != NULL) failed (status code: 500) ``` On the same set of images, but with a different prompt, I get: ``` Traceback (most recent call last): File "[my_home]/FOOD/SCRIPTS/auto_QC_csv_loop_v5.py", line 142, in <module> response = ollama.chat( File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 333, in chat return self._request( File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 178, in _request return cls(**self._request_raw(*args, **kwargs).json()) File "[my_home]/ollama/lib/python3.10/site-packages/ollama/_client.py", line 122, in _request_raw raise ResponseError(e.response.text, e.response.status_code) from None ollama._types.ResponseError: POST predict: Post "http://127.0.0.1:41691/completion": EOF (status code: 500) ``` Output of `ollama ps`: ``` NAME ID SIZE PROCESSOR UNTIL gemma3:latest a2af6cc3eb7f 6.2 GB 100% GPU 2 minutes from now ``` Hope this helps
GiteaMirror commented 2026-04-29 01:55:44 -05:00
Author
Owner
Copy Link

@chunyao commented on GitHub (Apr 11, 2025):

is it fixed?

<!-- gh-comment-id:2795908409 --> @chunyao commented on GitHub (Apr 11, 2025): is it fixed?
GiteaMirror commented 2026-04-29 01:55:45 -05:00
Author
Owner
Copy Link

@pkarc commented on GitHub (Apr 15, 2025):

I tested it and its fixed, just build main branch as it is already merged. Thank you @jessegross

<!-- gh-comment-id:2803424911 --> @pkarc commented on GitHub (Apr 15, 2025): I tested it and its fixed, just build main branch as it is already merged. Thank you @jessegross
GiteaMirror commented 2026-04-29 01:55:45 -05:00
Author
Owner
Copy Link

@jetnet commented on GitHub (Apr 15, 2025):

I tested it and its fixed, just build main branch as it is already merged. Thank you @jessegross

I tested with 0.6.5, but still have the same issue - gemma3:27b crashes.

<!-- gh-comment-id:2804363460 --> @jetnet commented on GitHub (Apr 15, 2025): > I tested it and its fixed, just build main branch as it is already merged. Thank you [@jessegross](https://github.com/jessegross) I tested with 0.6.5, but still have the same issue - gemma3:27b crashes.
GiteaMirror commented 2026-04-29 01:55:46 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 15, 2025):

It's not released yet. Build from source if you want the fix.

<!-- gh-comment-id:2804371315 --> @rick-github commented on GitHub (Apr 15, 2025): It's not released yet. Build from source if you want the fix.
GiteaMirror commented 2026-04-29 01:55:46 -05:00
Author
Owner
Copy Link

@Kazunarit commented on GitHub (Apr 24, 2025):

The problem was fixed in my environment in v0.6.6. Thank you. (windows11,docker desktop,RTX4090,gemma3:12b)

<!-- gh-comment-id:2825813429 --> @Kazunarit commented on GitHub (Apr 24, 2025): The problem was fixed in my environment in v0.6.6. Thank you. (windows11,docker desktop,RTX4090,gemma3:12b)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#53097
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?