github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11815] gpt-oss:20b token speed falls dramatically with increased context size #85529

New Issue
Closed
opened 2026-05-10 00:26:12 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-10 00:26:12 -05:00
Owner
Copy Link

Originally created by @walltime1 on GitHub (Aug 8, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11815

What is the issue?

The speed of gpt-oss:20b token emission falls drastically when context increased higher than 16k

$ ollama run gpt-oss:20b --verbose
>>> what is an apple? give a short answer
Thinking...
The user asks: "what is an apple? give a short answer". They want a short answer. So just a brief definition: "An apple is a round fruit produced by an apple tree, sweet
to slightly tart, used fresh or cooked." That suffices.
...done thinking.

An apple is a sweet‑to‑tart, round fruit that grows on apple trees, commonly eaten fresh or used in cooking and baking.

total duration:       7.522438083s
load duration:        54.703583ms
prompt eval count:    76 token(s)
prompt eval duration: 4.95078825s
prompt eval rate:     15.35 tokens/s
eval count:           93 token(s)
eval duration:        2.515884708s
eval rate:            36.97 tokens/s

>>> /set parameter num_ctx 65536
Set parameter 'num_ctx' to '65536'

>>> what is an apple? give a short answer
Thinking...
The user is repeating the same question. They want a short answer. The previous answer was short: "An apple is a sweet‑to‑tart, round fruit that grows on apple trees,
commonly eaten fresh or used in cooking and baking." That's about 32 words. It's short. We can maybe make it even shorter, but it's fine. They didn't ask for a longer
answer. We can keep the answer. But maybe they'd want an even more concise answer: "A fruit from an apple tree." That might be too short. But the user wants a short
answer. We can respond: "An apple is a sweet, round fruit that grows on apple trees." That is concise. Let's provide that.
...done thinking.

An apple is a sweet, round fruit that grows on apple trees.

total duration:       34.924537792s
load duration:        5.785523s
prompt eval count:    124 token(s)
prompt eval duration: 6.756992083s
prompt eval rate:     18.35 tokens/s
eval count:           168 token(s)
eval duration:        22.331427959s
eval rate:            7.52 tokens/s

Relevant log output

time=2025-08-08T20:48:35.115+03:00 level=INFO source=routes.go:1304 msg="server config" env="map[HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/Users/walltime/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false http_proxy: https_proxy: no_proxy:]"
time=2025-08-08T20:48:35.121+03:00 level=INFO source=images.go:477 msg="total blobs: 39"
time=2025-08-08T20:48:35.122+03:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0"
time=2025-08-08T20:48:35.123+03:00 level=INFO source=routes.go:1357 msg="Listening on 127.0.0.1:11434 (version 0.11.4)"
time=2025-08-08T20:48:35.212+03:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=metal variant="" compute="" driver=0.0 name="" total="16.0 GiB" available="16.0 GiB"
time=2025-08-08T20:48:35.212+03:00 level=INFO source=routes.go:1398 msg="entering low vram mode" "total vram"="16.0 GiB" threshold="20.0 GiB"
[GIN] 2025/08/08 - 20:48:35 | 200 |     356.792µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/08/08 - 20:48:35 | 200 |   86.643791ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:49:00 | 200 |       15.75µs |       127.0.0.1 | GET      "/"
[GIN] 2025/08/08 - 20:49:01 | 200 |    6.931583ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:49:01 | 200 |     902.375µs |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:49:01 | 200 |    48.31775ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:49:04 | 200 |    1.052125ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:49:14 | 200 |      52.084µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/08/08 - 20:49:14 | 200 |     68.4555ms |       127.0.0.1 | POST     "/api/show"
time=2025-08-08T20:49:14.271+03:00 level=INFO source=sched.go:786 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 gpu=0 parallel=1 available=17179885568 required="13.9 GiB"
time=2025-08-08T20:49:14.271+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="11.5 GiB" free_swap="0 B"
time=2025-08-08T20:49:14.271+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="204.0 MiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="1.0 GiB" memory.graph.partial="1.0 GiB"
time=2025-08-08T20:49:14.294+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 4096 --batch-size 512 --n-gpu-layers 25 --threads 10 --parallel 1 --port 49195"
time=2025-08-08T20:49:14.296+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-08-08T20:49:14.296+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-08-08T20:49:14.296+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-08-08T20:49:14.303+03:00 level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-08-08T20:49:14.303+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49195"
time=2025-08-08T20:49:14.327+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
time=2025-08-08T20:49:14.329+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang)
time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:365 msg="offloading 24 repeating layers to GPU"
time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:371 msg="offloading output layer to GPU"
time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:376 msg="offloaded 25/25 layers to GPU"
time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="1.1 GiB"
time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=Metal size="11.7 GiB"
ggml_metal_init: allocating
ggml_metal_init: picking default device: Apple M4 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name:   Apple M4 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction   = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets    = false
ggml_metal_init: has bfloat            = true
ggml_metal_init: use bfloat            = true
ggml_metal_init: hasUnifiedMemory      = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="1.2 GiB"
time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="5.6 MiB"
time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B"
time=2025-08-08T20:49:14.547+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-08T20:49:19.077+03:00 level=INFO source=server.go:637 msg="llama runner started in 4.78 seconds"
[GIN] 2025/08/08 - 20:49:19 | 200 |  4.887598083s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/08/08 - 20:49:23 | 200 |   16.304916ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:49:40 | 200 |   10.567459ms |       127.0.0.1 | GET      "/api/tags"
time=2025-08-08T20:50:08.374+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="15.5 GiB" free_swap="0 B"
time=2025-08-08T20:50:08.375+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.3 GiB" memory.required.partial="0 B" memory.required.kv="1.6 GiB" memory.required.allocations="[0 B]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="16.0 GiB" memory.graph.partial="16.0 GiB"
time=2025-08-08T20:50:08.402+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 65536 --batch-size 512 --threads 10 --parallel 1 --port 49200"
time=2025-08-08T20:50:08.404+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-08-08T20:50:08.404+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-08-08T20:50:08.404+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-08-08T20:50:08.442+03:00 level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-08-08T20:50:08.442+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49200"
time=2025-08-08T20:50:08.467+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
time=2025-08-08T20:50:08.469+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang)
time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:365 msg="offloading 0 repeating layers to GPU"
time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:369 msg="offloading output layer to CPU"
time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:376 msg="offloaded 0/25 layers to GPU"
time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="12.8 GiB"
ggml_metal_init: allocating
ggml_metal_init: picking default device: Apple M4 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name:   Apple M4 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction   = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets    = false
ggml_metal_init: has bfloat            = true
ggml_metal_init: use bfloat            = true
ggml_metal_init: hasUnifiedMemory      = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
time=2025-08-08T20:50:08.656+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="0 B"
time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="16.0 GiB"
time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B"
time=2025-08-08T20:50:13.690+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.29 seconds"
[GIN] 2025/08/08 - 20:50:24 | 200 |    5.111417ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:50:29 | 200 | 21.210531708s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/08/08 - 20:50:47 | 200 |   93.418583ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:50:54 | 200 |   77.911083ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:50:58 | 200 |   74.110917ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:51:05 | 200 |    3.138541ms |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/08/08 - 20:51:42 | 200 |    2.263083ms |       127.0.0.1 | HEAD     "/"
[GIN] 2025/08/08 - 20:51:42 | 200 |   72.602334ms |       127.0.0.1 | POST     "/api/show"
time=2025-08-08T20:51:42.934+03:00 level=INFO source=sched.go:786 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 gpu=0 parallel=1 available=17179885568 required="13.9 GiB"
time=2025-08-08T20:51:42.935+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="16.3 GiB" free_swap="0 B"
time=2025-08-08T20:51:42.935+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="204.0 MiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="1.0 GiB" memory.graph.partial="1.0 GiB"
time=2025-08-08T20:51:42.961+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 4096 --batch-size 512 --n-gpu-layers 25 --threads 10 --parallel 1 --port 49210"
time=2025-08-08T20:51:42.963+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-08-08T20:51:42.963+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-08-08T20:51:42.964+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-08-08T20:51:42.970+03:00 level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-08-08T20:51:42.970+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49210"
time=2025-08-08T20:51:42.994+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
time=2025-08-08T20:51:42.999+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang)
time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:365 msg="offloading 24 repeating layers to GPU"
time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:371 msg="offloading output layer to GPU"
time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:376 msg="offloaded 25/25 layers to GPU"
time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=Metal size="11.7 GiB"
time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="1.1 GiB"
ggml_metal_init: allocating
ggml_metal_init: picking default device: Apple M4 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name:   Apple M4 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction   = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets    = false
ggml_metal_init: has bfloat            = true
ggml_metal_init: use bfloat            = true
ggml_metal_init: hasUnifiedMemory      = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="1.2 GiB"
time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="5.6 MiB"
time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B"
time=2025-08-08T20:51:43.214+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-08T20:51:47.984+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.02 seconds"
[GIN] 2025/08/08 - 20:51:47 | 200 |  5.744717042s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/08/08 - 20:51:55 | 200 |   81.545625ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/08/08 - 20:52:07 | 200 |  7.522508625s |       127.0.0.1 | POST     "/api/chat"
time=2025-08-08T20:52:19.093+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="17.1 GiB" free_swap="0 B"
time=2025-08-08T20:52:19.093+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.3 GiB" memory.required.partial="0 B" memory.required.kv="1.6 GiB" memory.required.allocations="[0 B]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="16.0 GiB" memory.graph.partial="16.0 GiB"
time=2025-08-08T20:52:19.119+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 65536 --batch-size 512 --threads 10 --parallel 1 --port 49213"
time=2025-08-08T20:52:19.121+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-08-08T20:52:19.121+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-08-08T20:52:19.122+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-08-08T20:52:19.154+03:00 level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-08-08T20:52:19.154+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49213"
time=2025-08-08T20:52:19.177+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
time=2025-08-08T20:52:19.179+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang)
time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:365 msg="offloading 0 repeating layers to GPU"
time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:369 msg="offloading output layer to CPU"
time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:376 msg="offloaded 0/25 layers to GPU"
time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="12.8 GiB"
ggml_metal_init: allocating
ggml_metal_init: picking default device: Apple M4 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name:   Apple M4 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction   = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets    = false
ggml_metal_init: has bfloat            = true
ggml_metal_init: use bfloat            = true
ggml_metal_init: hasUnifiedMemory      = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
time=2025-08-08T20:52:19.373+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="0 B"
time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="16.0 GiB"
time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B"
time=2025-08-08T20:52:24.142+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.02 seconds"
[GIN] 2025/08/08 - 20:52:53 | 200 | 34.924707708s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2025/08/08 - 20:53:46 | 200 |   16.342833ms |       127.0.0.1 | GET      "/api/tags"

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.11.4

Originally created by @walltime1 on GitHub (Aug 8, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11815 ### What is the issue? The speed of gpt-oss:20b token emission falls drastically when context increased higher than 16k ``` $ ollama run gpt-oss:20b --verbose >>> what is an apple? give a short answer Thinking... The user asks: "what is an apple? give a short answer". They want a short answer. So just a brief definition: "An apple is a round fruit produced by an apple tree, sweet to slightly tart, used fresh or cooked." That suffices. ...done thinking. An apple is a sweet‑to‑tart, round fruit that grows on apple trees, commonly eaten fresh or used in cooking and baking. total duration: 7.522438083s load duration: 54.703583ms prompt eval count: 76 token(s) prompt eval duration: 4.95078825s prompt eval rate: 15.35 tokens/s eval count: 93 token(s) eval duration: 2.515884708s eval rate: 36.97 tokens/s >>> /set parameter num_ctx 65536 Set parameter 'num_ctx' to '65536' >>> what is an apple? give a short answer Thinking... The user is repeating the same question. They want a short answer. The previous answer was short: "An apple is a sweet‑to‑tart, round fruit that grows on apple trees, commonly eaten fresh or used in cooking and baking." That's about 32 words. It's short. We can maybe make it even shorter, but it's fine. They didn't ask for a longer answer. We can keep the answer. But maybe they'd want an even more concise answer: "A fruit from an apple tree." That might be too short. But the user wants a short answer. We can respond: "An apple is a sweet, round fruit that grows on apple trees." That is concise. Let's provide that. ...done thinking. An apple is a sweet, round fruit that grows on apple trees. total duration: 34.924537792s load duration: 5.785523s prompt eval count: 124 token(s) prompt eval duration: 6.756992083s prompt eval rate: 18.35 tokens/s eval count: 168 token(s) eval duration: 22.331427959s eval rate: 7.52 tokens/s ``` ### Relevant log output ```shell time=2025-08-08T20:48:35.115+03:00 level=INFO source=routes.go:1304 msg="server config" env="map[HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/Users/walltime/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false http_proxy: https_proxy: no_proxy:]" time=2025-08-08T20:48:35.121+03:00 level=INFO source=images.go:477 msg="total blobs: 39" time=2025-08-08T20:48:35.122+03:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0" time=2025-08-08T20:48:35.123+03:00 level=INFO source=routes.go:1357 msg="Listening on 127.0.0.1:11434 (version 0.11.4)" time=2025-08-08T20:48:35.212+03:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=metal variant="" compute="" driver=0.0 name="" total="16.0 GiB" available="16.0 GiB" time=2025-08-08T20:48:35.212+03:00 level=INFO source=routes.go:1398 msg="entering low vram mode" "total vram"="16.0 GiB" threshold="20.0 GiB" [GIN] 2025/08/08 - 20:48:35 | 200 | 356.792µs | 127.0.0.1 | HEAD "/" [GIN] 2025/08/08 - 20:48:35 | 200 | 86.643791ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:49:00 | 200 | 15.75µs | 127.0.0.1 | GET "/" [GIN] 2025/08/08 - 20:49:01 | 200 | 6.931583ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:49:01 | 200 | 902.375µs | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:49:01 | 200 | 48.31775ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:49:04 | 200 | 1.052125ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:49:14 | 200 | 52.084µs | 127.0.0.1 | HEAD "/" [GIN] 2025/08/08 - 20:49:14 | 200 | 68.4555ms | 127.0.0.1 | POST "/api/show" time=2025-08-08T20:49:14.271+03:00 level=INFO source=sched.go:786 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 gpu=0 parallel=1 available=17179885568 required="13.9 GiB" time=2025-08-08T20:49:14.271+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="11.5 GiB" free_swap="0 B" time=2025-08-08T20:49:14.271+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="204.0 MiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="1.0 GiB" memory.graph.partial="1.0 GiB" time=2025-08-08T20:49:14.294+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 4096 --batch-size 512 --n-gpu-layers 25 --threads 10 --parallel 1 --port 49195" time=2025-08-08T20:49:14.296+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1 time=2025-08-08T20:49:14.296+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding" time=2025-08-08T20:49:14.296+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding" time=2025-08-08T20:49:14.303+03:00 level=INFO source=runner.go:925 msg="starting ollama engine" time=2025-08-08T20:49:14.303+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49195" time=2025-08-08T20:49:14.327+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30 time=2025-08-08T20:49:14.329+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang) time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:365 msg="offloading 24 repeating layers to GPU" time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:371 msg="offloading output layer to GPU" time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:376 msg="offloaded 25/25 layers to GPU" time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="1.1 GiB" time=2025-08-08T20:49:14.393+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=Metal size="11.7 GiB" ggml_metal_init: allocating ggml_metal_init: picking default device: Apple M4 Pro ggml_metal_load_library: using embedded metal library ggml_metal_init: GPU name: Apple M4 Pro ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction = true ggml_metal_init: simdgroup matrix mul. = true ggml_metal_init: has residency sets = false ggml_metal_init: has bfloat = true ggml_metal_init: use bfloat = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 17179.89 MB time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="1.2 GiB" time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="5.6 MiB" time=2025-08-08T20:49:14.440+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B" time=2025-08-08T20:49:14.547+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model" time=2025-08-08T20:49:19.077+03:00 level=INFO source=server.go:637 msg="llama runner started in 4.78 seconds" [GIN] 2025/08/08 - 20:49:19 | 200 | 4.887598083s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/08/08 - 20:49:23 | 200 | 16.304916ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:49:40 | 200 | 10.567459ms | 127.0.0.1 | GET "/api/tags" time=2025-08-08T20:50:08.374+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="15.5 GiB" free_swap="0 B" time=2025-08-08T20:50:08.375+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.3 GiB" memory.required.partial="0 B" memory.required.kv="1.6 GiB" memory.required.allocations="[0 B]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="16.0 GiB" memory.graph.partial="16.0 GiB" time=2025-08-08T20:50:08.402+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 65536 --batch-size 512 --threads 10 --parallel 1 --port 49200" time=2025-08-08T20:50:08.404+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1 time=2025-08-08T20:50:08.404+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding" time=2025-08-08T20:50:08.404+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding" time=2025-08-08T20:50:08.442+03:00 level=INFO source=runner.go:925 msg="starting ollama engine" time=2025-08-08T20:50:08.442+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49200" time=2025-08-08T20:50:08.467+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30 time=2025-08-08T20:50:08.469+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang) time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:365 msg="offloading 0 repeating layers to GPU" time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:369 msg="offloading output layer to CPU" time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:376 msg="offloaded 0/25 layers to GPU" time=2025-08-08T20:50:08.576+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="12.8 GiB" ggml_metal_init: allocating ggml_metal_init: picking default device: Apple M4 Pro ggml_metal_load_library: using embedded metal library ggml_metal_init: GPU name: Apple M4 Pro ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction = true ggml_metal_init: simdgroup matrix mul. = true ggml_metal_init: has residency sets = false ggml_metal_init: has bfloat = true ggml_metal_init: use bfloat = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 17179.89 MB time=2025-08-08T20:50:08.656+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model" time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="0 B" time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="16.0 GiB" time=2025-08-08T20:50:08.682+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B" time=2025-08-08T20:50:13.690+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.29 seconds" [GIN] 2025/08/08 - 20:50:24 | 200 | 5.111417ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:50:29 | 200 | 21.210531708s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/08/08 - 20:50:47 | 200 | 93.418583ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:50:54 | 200 | 77.911083ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:50:58 | 200 | 74.110917ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:51:05 | 200 | 3.138541ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/08/08 - 20:51:42 | 200 | 2.263083ms | 127.0.0.1 | HEAD "/" [GIN] 2025/08/08 - 20:51:42 | 200 | 72.602334ms | 127.0.0.1 | POST "/api/show" time=2025-08-08T20:51:42.934+03:00 level=INFO source=sched.go:786 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 gpu=0 parallel=1 available=17179885568 required="13.9 GiB" time=2025-08-08T20:51:42.935+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="16.3 GiB" free_swap="0 B" time=2025-08-08T20:51:42.935+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.9 GiB" memory.required.partial="13.9 GiB" memory.required.kv="204.0 MiB" memory.required.allocations="[13.9 GiB]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="1.0 GiB" memory.graph.partial="1.0 GiB" time=2025-08-08T20:51:42.961+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 4096 --batch-size 512 --n-gpu-layers 25 --threads 10 --parallel 1 --port 49210" time=2025-08-08T20:51:42.963+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1 time=2025-08-08T20:51:42.963+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding" time=2025-08-08T20:51:42.964+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding" time=2025-08-08T20:51:42.970+03:00 level=INFO source=runner.go:925 msg="starting ollama engine" time=2025-08-08T20:51:42.970+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49210" time=2025-08-08T20:51:42.994+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30 time=2025-08-08T20:51:42.999+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang) time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:365 msg="offloading 24 repeating layers to GPU" time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:371 msg="offloading output layer to GPU" time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:376 msg="offloaded 25/25 layers to GPU" time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=Metal size="11.7 GiB" time=2025-08-08T20:51:43.099+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="1.1 GiB" ggml_metal_init: allocating ggml_metal_init: picking default device: Apple M4 Pro ggml_metal_load_library: using embedded metal library ggml_metal_init: GPU name: Apple M4 Pro ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction = true ggml_metal_init: simdgroup matrix mul. = true ggml_metal_init: has residency sets = false ggml_metal_init: has bfloat = true ggml_metal_init: use bfloat = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 17179.89 MB time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="1.2 GiB" time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="5.6 MiB" time=2025-08-08T20:51:43.147+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B" time=2025-08-08T20:51:43.214+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model" time=2025-08-08T20:51:47.984+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.02 seconds" [GIN] 2025/08/08 - 20:51:47 | 200 | 5.744717042s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/08/08 - 20:51:55 | 200 | 81.545625ms | 127.0.0.1 | POST "/api/show" [GIN] 2025/08/08 - 20:52:07 | 200 | 7.522508625s | 127.0.0.1 | POST "/api/chat" time=2025-08-08T20:52:19.093+03:00 level=INFO source=server.go:135 msg="system memory" total="24.0 GiB" free="17.1 GiB" free_swap="0 B" time=2025-08-08T20:52:19.093+03:00 level=INFO source=server.go:175 msg=offload library=metal layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[16.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="13.3 GiB" memory.required.partial="0 B" memory.required.kv="1.6 GiB" memory.required.allocations="[0 B]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="16.0 GiB" memory.graph.partial="16.0 GiB" time=2025-08-08T20:52:19.119+03:00 level=INFO source=server.go:438 msg="starting llama server" cmd="/Applications/Ollama.app/Contents/Resources/ollama runner --ollama-engine --model /Users/walltime/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 65536 --batch-size 512 --threads 10 --parallel 1 --port 49213" time=2025-08-08T20:52:19.121+03:00 level=INFO source=sched.go:481 msg="loaded runners" count=1 time=2025-08-08T20:52:19.121+03:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding" time=2025-08-08T20:52:19.122+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding" time=2025-08-08T20:52:19.154+03:00 level=INFO source=runner.go:925 msg="starting ollama engine" time=2025-08-08T20:52:19.154+03:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:49213" time=2025-08-08T20:52:19.177+03:00 level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30 time=2025-08-08T20:52:19.179+03:00 level=INFO source=ggml.go:104 msg=system Metal.0.EMBED_LIBRARY=1 Metal.0.BF16=1 CPU.0.ARM_FMA=1 CPU.0.FP16_VA=1 CPU.0.DOTPROD=1 CPU.0.LLAMAFILE=1 CPU.0.ACCELERATE=1 compiler=cgo(clang) time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:365 msg="offloading 0 repeating layers to GPU" time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:369 msg="offloading output layer to CPU" time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:376 msg="offloaded 0/25 layers to GPU" time=2025-08-08T20:52:19.288+03:00 level=INFO source=ggml.go:379 msg="model weights" buffer=CPU size="12.8 GiB" ggml_metal_init: allocating ggml_metal_init: picking default device: Apple M4 Pro ggml_metal_load_library: using embedded metal library ggml_metal_init: GPU name: Apple M4 Pro ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction = true ggml_metal_init: simdgroup matrix mul. = true ggml_metal_init: has residency sets = false ggml_metal_init: has bfloat = true ggml_metal_init: use bfloat = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 17179.89 MB time=2025-08-08T20:52:19.373+03:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model" time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=Metal buffer_type=Metal size="0 B" time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=BLAS buffer_type=CPU size="16.0 GiB" time=2025-08-08T20:52:19.386+03:00 level=INFO source=ggml.go:668 msg="compute graph" backend=CPU buffer_type=CPU size="0 B" time=2025-08-08T20:52:24.142+03:00 level=INFO source=server.go:637 msg="llama runner started in 5.02 seconds" [GIN] 2025/08/08 - 20:52:53 | 200 | 34.924707708s | 127.0.0.1 | POST "/api/chat" [GIN] 2025/08/08 - 20:53:46 | 200 | 16.342833ms | 127.0.0.1 | GET "/api/tags" ``` ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.11.4
GiteaMirror added the bug label 2026-05-10 00:26:12 -05:00
GiteaMirror commented 2026-05-10 00:26:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 8, 2025):

The increased context causes an increase in graph size, larger than can fit on the GPU, so the model is loaded in system RAM where the slower CPU does inference.

<!-- gh-comment-id:3168968966 --> @rick-github commented on GitHub (Aug 8, 2025): The increased context causes an increase in graph size, larger than can fit on the GPU, so the model is loaded in system RAM where the slower CPU does inference.
GiteaMirror commented 2026-05-10 00:26:15 -05:00
Author
Owner
Copy Link

@walltime1 commented on GitHub (Aug 8, 2025):

The increased context causes an increase in graph size, larger than can fit on the GPU, so the model is loaded in system RAM where the slower CPU does inference.

Is this behaviour visible in provided relevant log? I would appreciate if you could point it out.

Initially I had assumed this were a bug because same change in the context size while using same model (although different quantisation) directly on llama.cpp haven't resulted in any performance degradation.

<!-- gh-comment-id:3169077865 --> @walltime1 commented on GitHub (Aug 8, 2025): > The increased context causes an increase in graph size, larger than can fit on the GPU, so the model is loaded in system RAM where the slower CPU does inference. Is this behaviour visible in provided relevant log? I would appreciate if you could point it out. Initially I had assumed this were a bug because same change in the context size while using same model (although different quantisation) directly on llama.cpp haven't resulted in any performance degradation.
GiteaMirror commented 2026-05-10 00:26:16 -05:00
Author
Owner
Copy Link

@walltime1 commented on GitHub (Aug 8, 2025):

Is this behaviour visible in provided relevant log? I would appreciate if you could point it out.

I think I've found that in the log, guess the issue is resolved.

<!-- gh-comment-id:3169083165 --> @walltime1 commented on GitHub (Aug 8, 2025): > Is this behaviour visible in provided relevant log? I would appreciate if you could point it out. I think I've found that in the log, guess the issue is resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#85529
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?