github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #14751] CUDA out of memory error when using Qwen3.5:27B after external application (e.g., Mattermost plugin) invokes the model #9537

New Issue
Closed
opened 2026-04-12 22:27:38 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 22:27:38 -05:00
Owner
Copy Link

Originally created by @cedarice on GitHub (Mar 10, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14751

What is the issue?

Description

When invoking the Qwen3.5 series models (specifically qwen3.5:27b) via Ollama, an intermittent CUDA out-of-memory error occurs. The issue appears to be triggered after an external application (such as the Mattermost Agent Plugin) calls the model. Subsequent manual inference attempts fail with a CUDA OOM error, even though ollama ps shows the model as actively loaded.

Steps to Reproduce

  1. Start Ollama service with GPU support (device 0).
  2. Use an external application (e.g., Mattermost Agent Plugin) to send a request to qwen3.5:27b.
  3. Observe that the application receives no response or an empty result from the model.
  4. In terminal, run: 
    ollama run qwen3.5:27b
  5. The model starts normally and accepts input.
  6. After submitting a prompt (e.g., who are you), the inference fails with a CUDA OOM error (see logs below).
  7. Run ollama ps — the model still appears as "running".

Expected Behavior

  • The model should handle inference requests consistently, whether invoked via API/plugin or CLI.
  • If a prior request exhausts VRAM, subsequent requests should either:
    • Gracefully release and reload the model, or
    • Return a clear error indicating resource unavailability (rather than a low-level CUDA crash).

Actual Behavior

After an external plugin invokes the model, manual CLI inference fails with:

Thinking...
OkayError: an error was encountered while running the model: CUDA error: out of memory
  current device: 0, in function evaluate_and_capture_cuda_graph at //ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:3831
  cudaGraphInstantiate(&cuda_ctx->cuda_graph->instance, cuda_ctx->cuda_graph->graph, __null, __null, 0)
//ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:94: CUDA error

Despite the error, ollama ps continues to list qwen3.5:27b as running, suggesting the model context was not properly cleaned up.

Qwen3 series (e.g., qwen3:32b, qwen3:14b) works as expected under the same conditions

Relevant log output

3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 111 gp=0xc00053a8c0 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e984a?, 0x3?, 0x63?, 0x4?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000534f38 sp=0xc000534f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000534fc8 sp=0xc000534f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000534fe0 sp=0xc000534fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000534fe8 sp=0xc000534fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 112 gp=0xc00053aa80 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xd4?, 0x8?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000535738 sp=0xc000535718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005357c8 sp=0xc000535738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc0005357e0 sp=0xc0005357c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005357e8 sp=0xc0005357e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 113 gp=0xc00053ac40 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a5d2a0f4?, 0x3?, 0xc3?, 0x4b?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000535f38 sp=0xc000535f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000535fc8 sp=0xc000535f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000535fe0 sp=0xc000535fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 114 gp=0xc00053ae00 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e93e8?, 0x1?, 0x5e?, 0xcf?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000540738 sp=0xc000540718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005407c8 sp=0xc000540738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc0005407e0 sp=0xc0005407c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005407e8 sp=0xc0005407e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 115 gp=0xc00053afc0 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e5f9b?, 0x3?, 0x4d?, 0xa2?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000540f38 sp=0xc000540f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000540fc8 sp=0xc000540f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000540fe0 sp=0xc000540fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000540fe8 sp=0xc000540fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 116 gp=0xc00053b180 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e3b7e?, 0x1?, 0x11?, 0x92?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000541738 sp=0xc000541718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005417c8 sp=0xc000541738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc0005417e0 sp=0xc0005417c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005417e8 sp=0xc0005417e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 117 gp=0xc00053b340 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57d0855?, 0x1?, 0x6c?, 0x8d?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000541f38 sp=0xc000541f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000541fc8 sp=0xc000541f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000541fe0 sp=0xc000541fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 118 gp=0xc00053b500 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0x26?, 0x79?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000542738 sp=0xc000542718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005427c8 sp=0xc000542738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc0005427e0 sp=0xc0005427c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005427e8 sp=0xc0005427e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 119 gp=0xc00053b6c0 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57f16b8?, 0x1?, 0xc0?, 0x72?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000542f38 sp=0xc000542f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000542fc8 sp=0xc000542f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000542fe0 sp=0xc000542fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000542fe8 sp=0xc000542fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 120 gp=0xc00053b880 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a5ddaecf?, 0x1?, 0xed?, 0xb4?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000543738 sp=0xc000543718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005437c8 sp=0xc000543738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc0005437e0 sp=0xc0005437c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005437e8 sp=0xc0005437e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 121 gp=0xc00053ba40 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x3?, 0x19?, 0x5?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000543f38 sp=0xc000543f18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc000543fc8 sp=0xc000543f38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc000543fe0 sp=0xc000543fc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 122 gp=0xc00053bc00 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xa1?, 0x20?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc00053c738 sp=0xc00053c718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc00053c7c8 sp=0xc00053c738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc00053c7e0 sp=0xc00053c7c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00053c7e8 sp=0xc00053c7e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 123 gp=0xc00053bdc0 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57dad07?, 0x1?, 0xf?, 0x4b?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc00053cf38 sp=0xc00053cf18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc00053cfc8 sp=0xc00053cf38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc00053cfe0 sp=0xc00053cfc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00053cfe8 sp=0xc00053cfe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 124 gp=0xc000544000 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xf6?, 0x8c?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc00053d738 sp=0xc00053d718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc00053d7c8 sp=0xc00053d738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc00053d7e0 sp=0xc00053d7c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00053d7e8 sp=0xc00053d7e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 125 gp=0xc0005441c0 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57effab?, 0x1?, 0xe0?, 0xdd?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc00053df38 sp=0xc00053df18 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc00053dfc8 sp=0xc00053df38 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc00053dfe0 sp=0xc00053dfc8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 126 gp=0xc000544380 m=nil [GC worker (idle)]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e5ff6?, 0x1?, 0x26?, 0x27?, 0x0?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc00053e738 sp=0xc00053e718 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1423 +0xe9 fp=0xc00053e7c8 sp=0xc00053e738 pc=0x564acafe7aa9
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x25 fp=0xc00053e7e0 sp=0xc00053e7c8 pc=0x564acafe7985
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00053e7e8 sp=0xc00053e7e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/mgc.go:1339 +0x105
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 20 gp=0xc00027b880 m=nil [chan receive]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x30?, 0x564acc813260?, 0x1?, 0x0?, 0xc0008bf798?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc0008bf750 sp=0xc0008bf730 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv(0xc000744310, 0x0, 0x1)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/chan.go:664 +0x445 fp=0xc0008bf7c8 sp=0xc0008bf750 pc=0x564acafd6085
3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv1(0x564acc333bbb?, 0x29?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/chan.go:506 +0x12 fp=0xc0008bf7f0 sp=0xc0008bf7c8 pc=0x564acafd5c12
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).forwardBatch(_, {0x2, {0x564acc8e4130, 0xc000730300}, {0x564acc8f1570, 0xc0006d5db8}, {0xc000b1c148, 0x1, 0x1}, {{0x564acc8f1570, ...}, ...}, ...})
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:476 +0xfa fp=0xc0008bfb58 sp=0xc0008bf7f0 pc=0x564acb65a7ba
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0002f90e0, {0x564acc8d7bd0, 0xc0001c3d60})
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:453 +0x18c fp=0xc0008bffb8 sp=0xc0008bfb58 pc=0x564acb65a46c
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:1442 +0x28 fp=0xc0008bffe0 sp=0xc0008bffb8 pc=0x564acb663f68
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0008bffe8 sp=0xc0008bffe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:1442 +0x4c9
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 21 gp=0xc00027ba40 m=nil [select]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0xc0000459e8?, 0x2?, 0x4?, 0x0?, 0xc000045810?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000045620 sp=0xc000045600 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.selectgo(0xc0000459e8, 0xc00004580c, 0xc00088c000?, 0x0, 0x1?, 0x1)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/select.go:351 +0x837 fp=0xc000045758 sp=0xc000045620 pc=0x564acb018c17
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc0002f90e0, {0x564acc8d5440, 0xc00079a0e0}, 0xc000476280)
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:974 +0xc86 fp=0xc000045ac0 sp=0xc000045758 pc=0x564acb65ee26
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x564acc8d5440?, 0xc00079a0e0?}, 0xc0008bfb40?)
3月 10 09:33:22 jzswai ollama[3872788]:         <autogenerated>:1 +0x36 fp=0xc000045af0 sp=0xc000045ac0 pc=0x564acb664456
3月 10 09:33:22 jzswai ollama[3872788]: net/http.HandlerFunc.ServeHTTP(0xc0005ba780?, {0x564acc8d5440?, 0xc00079a0e0?}, 0xc0008bfb60?)
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:2294 +0x29 fp=0xc000045b18 sp=0xc000045af0 pc=0x564acb33a8c9
3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*ServeMux).ServeHTTP(0x564acafdeac5?, {0x564acc8d5440, 0xc00079a0e0}, 0xc000476280)
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:2822 +0x1c4 fp=0xc000045b68 sp=0xc000045b18 pc=0x564acb33c7c4
3月 10 09:33:22 jzswai ollama[3872788]: net/http.serverHandler.ServeHTTP({0x564acc8d1730?}, {0x564acc8d5440?, 0xc00079a0e0?}, 0x1?)
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:3301 +0x8e fp=0xc000045b98 sp=0xc000045b68 pc=0x564acb35a24e
3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*conn).serve(0xc0001ae750, {0x564acc8d7b98, 0xc0005df800})
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:2102 +0x625 fp=0xc000045fb8 sp=0xc000045b98 pc=0x564acb338dc5
3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*Server).Serve.gowrap3()
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:3454 +0x28 fp=0xc000045fe0 sp=0xc000045fb8 pc=0x564acb33e688
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by net/http.(*Server).Serve in goroutine 1
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:3454 +0x485
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 1758 gp=0xc000583dc0 m=nil [IO wait]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0xc009445e00?, 0xc009445e50?, 0x0?, 0x80?, 0xb?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc0006f45d8 sp=0xc0006f45b8 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.netpollblock(0x564acb05d798?, 0xcafd34a6?, 0x4a?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/netpoll.go:575 +0xf7 fp=0xc0006f4610 sp=0xc0006f45d8 pc=0x564acafff097
3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.runtime_pollWait(0x7f9cb20525b8, 0x72)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/netpoll.go:351 +0x85 fp=0xc0006f4630 sp=0xc0006f4610 pc=0x564acb039105
3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*pollDesc).wait(0xc0005c1500?, 0xc0005df901?, 0x0)
3月 10 09:33:22 jzswai ollama[3872788]:         internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006f4658 sp=0xc0006f4630 pc=0x564acb0c1487
3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*pollDesc).waitRead(...)
3月 10 09:33:22 jzswai ollama[3872788]:         internal/poll/fd_poll_runtime.go:89
3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*FD).Read(0xc0005c1500, {0xc0005df901, 0x1, 0x1})
3月 10 09:33:22 jzswai ollama[3872788]:         internal/poll/fd_unix.go:165 +0x27a fp=0xc0006f46f0 sp=0xc0006f4658 pc=0x564acb0c277a
3月 10 09:33:22 jzswai ollama[3872788]: net.(*netFD).Read(0xc0005c1500, {0xc0005df901?, 0xc0007141d8?, 0xc0006f4770?})
3月 10 09:33:22 jzswai ollama[3872788]:         net/fd_posix.go:55 +0x25 fp=0xc0006f4738 sp=0xc0006f46f0 pc=0x564acb137da5
3月 10 09:33:22 jzswai ollama[3872788]: net.(*conn).Read(0xc00013ab00, {0xc0005df901?, 0xc009449c70?, 0xc009449cc0?})
3月 10 09:33:22 jzswai ollama[3872788]:         net/net.go:194 +0x45 fp=0xc0006f4780 sp=0xc0006f4738 pc=0x564acb146165
3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*connReader).backgroundRead(0xc0005df8f0)
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:690 +0x37 fp=0xc0006f47c8 sp=0xc0006f4780 pc=0x564acb332c97
3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*connReader).startBackgroundRead.gowrap2()
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:686 +0x25 fp=0xc0006f47e0 sp=0xc0006f47c8 pc=0x564acb332bc5
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0006f47e8 sp=0xc0006f47e0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by net/http.(*connReader).startBackgroundRead in goroutine 21
3月 10 09:33:22 jzswai ollama[3872788]:         net/http/server.go:686 +0xb6
3月 10 09:33:22 jzswai ollama[3872788]: goroutine 1439 gp=0xc0007b6e00 m=nil [chan receive]:
3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x30?, 0x564acc813260?, 0x1?, 0xba?, 0xc000b26b18?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/proc.go:435 +0xce fp=0xc000b26ad0 sp=0xc000b26ab0 pc=0x564acb039eee
3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv(0xc0007442a0, 0x0, 0x1)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/chan.go:664 +0x445 fp=0xc000b26b48 sp=0xc000b26ad0 pc=0x564acafd6085
3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv1(0x564acc3378ec?, 0x2c?)
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/chan.go:506 +0x12 fp=0xc000b26b70 sp=0xc000b26b48 pc=0x564acafd5c12
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).computeBatch(0xc0002f90e0, {0x2, {0x564acc8e4130, 0xc000730300}, {0x564acc8f1570, 0xc0006d5db8}, {0xc000b1c148, 0x1, 0x1}, {{0x564acc8f1570, ...}, ...}, ...})
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:646 +0x185 fp=0xc000b26ef0 sp=0xc000b26b70 pc=0x564acb65c2c5
3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run.gowrap1()
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:459 +0x58 fp=0xc000b26fe0 sp=0xc000b26ef0 pc=0x564acb65a698
3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({})
3月 10 09:33:22 jzswai ollama[3872788]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000b26fe8 sp=0xc000b26fe0 pc=0x564acb041e61
3月 10 09:33:22 jzswai ollama[3872788]: created by github.com/ollama/ollama/runner/ollamarunner.(*Server).run in goroutine 20
3月 10 09:33:22 jzswai ollama[3872788]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:459 +0x2cd
3月 10 09:33:22 jzswai ollama[3872788]: rax    0x0
3月 10 09:33:22 jzswai ollama[3872788]: rbx    0x7f9ad1fff640
3月 10 09:33:22 jzswai ollama[3872788]: rcx    0x7f9cb1e969fc
3月 10 09:33:22 jzswai ollama[3872788]: rdx    0x6
3月 10 09:33:22 jzswai ollama[3872788]: rdi    0x3cf144
3月 10 09:33:22 jzswai ollama[3872788]: rsi    0x3cf1a2
3月 10 09:33:22 jzswai ollama[3872788]: rbp    0x3cf1a2
3月 10 09:33:22 jzswai ollama[3872788]: rsp    0x7f9ad1ffdd20
3月 10 09:33:22 jzswai ollama[3872788]: r8     0x7f9ad1ffddf0
3月 10 09:33:22 jzswai ollama[3872788]: r9     0x7f94e9ba9e90
3月 10 09:33:22 jzswai ollama[3872788]: r10    0x8
3月 10 09:33:22 jzswai ollama[3872788]: r11    0x246
3月 10 09:33:22 jzswai ollama[3872788]: r12    0x6
3月 10 09:33:22 jzswai ollama[3872788]: r13    0x16
3月 10 09:33:22 jzswai ollama[3872788]: r14    0x7f9c5029242e
3月 10 09:33:22 jzswai ollama[3872788]: r15    0x7f9ac90d5508
3月 10 09:33:22 jzswai ollama[3872788]: rip    0x7f9cb1e969fc
3月 10 09:33:22 jzswai ollama[3872788]: rflags 0x246
3月 10 09:33:22 jzswai ollama[3872788]: cs     0x33
3月 10 09:33:22 jzswai ollama[3872788]: fs     0x0
3月 10 09:33:22 jzswai ollama[3872788]: gs     0x0
3月 10 09:33:22 jzswai ollama[3872788]: time=2026-03-10T09:33:22.799+08:00 level=ERROR source=server.go:303 msg="llama runner terminated" error="exit status 2"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.17.7

Originally created by @cedarice on GitHub (Mar 10, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14751 ### What is the issue? ## Description When invoking the Qwen3.5 series models (specifically `qwen3.5:27b`) via Ollama, an intermittent CUDA out-of-memory error occurs. The issue appears to be triggered after an external application (such as the Mattermost Agent Plugin) calls the model. Subsequent manual inference attempts fail with a CUDA OOM error, even though `ollama ps` shows the model as actively loaded. --- ## Steps to Reproduce 1. Start Ollama service with GPU support (device 0). 2. Use an external application (e.g., Mattermost Agent Plugin) to send a request to `qwen3.5:27b`. 3. Observe that the application receives no response or an empty result from the model. 4. In terminal, run: ```bash ollama run qwen3.5:27b ``` 5. The model starts normally and accepts input. 6. After submitting a prompt (e.g., `who are you`), the inference fails with a CUDA OOM error (see logs below). 7. Run `ollama ps` — the model still appears as "running". --- ## Expected Behavior - The model should handle inference requests consistently, whether invoked via API/plugin or CLI. - If a prior request exhausts VRAM, subsequent requests should either: - Gracefully release and reload the model, or - Return a clear error indicating resource unavailability (rather than a low-level CUDA crash). --- ## Actual Behavior After an external plugin invokes the model, manual CLI inference fails with: ``` Thinking... OkayError: an error was encountered while running the model: CUDA error: out of memory current device: 0, in function evaluate_and_capture_cuda_graph at //ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:3831 cudaGraphInstantiate(&cuda_ctx->cuda_graph->instance, cuda_ctx->cuda_graph->graph, __null, __null, 0) //ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:94: CUDA error ``` Despite the error, `ollama ps` continues to list `qwen3.5:27b` as running, suggesting the model context was not properly cleaned up. Qwen3 series (e.g., qwen3:32b, qwen3:14b) works as expected under the same conditions ### Relevant log output ```shell 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 111 gp=0xc00053a8c0 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e984a?, 0x3?, 0x63?, 0x4?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000534f38 sp=0xc000534f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000534fc8 sp=0xc000534f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000534fe0 sp=0xc000534fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000534fe8 sp=0xc000534fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 112 gp=0xc00053aa80 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xd4?, 0x8?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000535738 sp=0xc000535718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc0005357c8 sp=0xc000535738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc0005357e0 sp=0xc0005357c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005357e8 sp=0xc0005357e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 113 gp=0xc00053ac40 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a5d2a0f4?, 0x3?, 0xc3?, 0x4b?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000535f38 sp=0xc000535f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000535fc8 sp=0xc000535f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000535fe0 sp=0xc000535fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 114 gp=0xc00053ae00 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e93e8?, 0x1?, 0x5e?, 0xcf?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000540738 sp=0xc000540718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc0005407c8 sp=0xc000540738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc0005407e0 sp=0xc0005407c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005407e8 sp=0xc0005407e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 115 gp=0xc00053afc0 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e5f9b?, 0x3?, 0x4d?, 0xa2?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000540f38 sp=0xc000540f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000540fc8 sp=0xc000540f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000540fe0 sp=0xc000540fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000540fe8 sp=0xc000540fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 116 gp=0xc00053b180 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e3b7e?, 0x1?, 0x11?, 0x92?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000541738 sp=0xc000541718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc0005417c8 sp=0xc000541738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc0005417e0 sp=0xc0005417c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005417e8 sp=0xc0005417e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 117 gp=0xc00053b340 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57d0855?, 0x1?, 0x6c?, 0x8d?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000541f38 sp=0xc000541f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000541fc8 sp=0xc000541f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000541fe0 sp=0xc000541fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 118 gp=0xc00053b500 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0x26?, 0x79?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000542738 sp=0xc000542718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc0005427c8 sp=0xc000542738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc0005427e0 sp=0xc0005427c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005427e8 sp=0xc0005427e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 119 gp=0xc00053b6c0 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57f16b8?, 0x1?, 0xc0?, 0x72?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000542f38 sp=0xc000542f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000542fc8 sp=0xc000542f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000542fe0 sp=0xc000542fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000542fe8 sp=0xc000542fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 120 gp=0xc00053b880 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a5ddaecf?, 0x1?, 0xed?, 0xb4?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000543738 sp=0xc000543718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc0005437c8 sp=0xc000543738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc0005437e0 sp=0xc0005437c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005437e8 sp=0xc0005437e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 121 gp=0xc00053ba40 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x3?, 0x19?, 0x5?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000543f38 sp=0xc000543f18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc000543fc8 sp=0xc000543f38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc000543fe0 sp=0xc000543fc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 122 gp=0xc00053bc00 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xa1?, 0x20?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc00053c738 sp=0xc00053c718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc00053c7c8 sp=0xc00053c738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc00053c7e0 sp=0xc00053c7c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00053c7e8 sp=0xc00053c7e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 123 gp=0xc00053bdc0 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57dad07?, 0x1?, 0xf?, 0x4b?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc00053cf38 sp=0xc00053cf18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc00053cfc8 sp=0xc00053cf38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc00053cfe0 sp=0xc00053cfc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00053cfe8 sp=0xc00053cfe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 124 gp=0xc000544000 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x564acd3e44c0?, 0x1?, 0xf6?, 0x8c?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc00053d738 sp=0xc00053d718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc00053d7c8 sp=0xc00053d738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc00053d7e0 sp=0xc00053d7c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00053d7e8 sp=0xc00053d7e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 125 gp=0xc0005441c0 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57effab?, 0x1?, 0xe0?, 0xdd?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc00053df38 sp=0xc00053df18 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc00053dfc8 sp=0xc00053df38 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc00053dfe0 sp=0xc00053dfc8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 126 gp=0xc000544380 m=nil [GC worker (idle)]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x103b297a57e5ff6?, 0x1?, 0x26?, 0x27?, 0x0?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc00053e738 sp=0xc00053e718 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkWorker(0xc00016f570) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1423 +0xe9 fp=0xc00053e7c8 sp=0xc00053e738 pc=0x564acafe7aa9 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gcBgMarkStartWorkers.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x25 fp=0xc00053e7e0 sp=0xc00053e7c8 pc=0x564acafe7985 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00053e7e8 sp=0xc00053e7e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by runtime.gcBgMarkStartWorkers in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: runtime/mgc.go:1339 +0x105 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 20 gp=0xc00027b880 m=nil [chan receive]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x30?, 0x564acc813260?, 0x1?, 0x0?, 0xc0008bf798?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc0008bf750 sp=0xc0008bf730 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv(0xc000744310, 0x0, 0x1) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/chan.go:664 +0x445 fp=0xc0008bf7c8 sp=0xc0008bf750 pc=0x564acafd6085 3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv1(0x564acc333bbb?, 0x29?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/chan.go:506 +0x12 fp=0xc0008bf7f0 sp=0xc0008bf7c8 pc=0x564acafd5c12 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).forwardBatch(_, {0x2, {0x564acc8e4130, 0xc000730300}, {0x564acc8f1570, 0xc0006d5db8}, {0xc000b1c148, 0x1, 0x1}, {{0x564acc8f1570, ...}, ...}, ...}) 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:476 +0xfa fp=0xc0008bfb58 sp=0xc0008bf7f0 pc=0x564acb65a7ba 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0002f90e0, {0x564acc8d7bd0, 0xc0001c3d60}) 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:453 +0x18c fp=0xc0008bffb8 sp=0xc0008bfb58 pc=0x564acb65a46c 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:1442 +0x28 fp=0xc0008bffe0 sp=0xc0008bffb8 pc=0x564acb663f68 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0008bffe8 sp=0xc0008bffe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:1442 +0x4c9 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 21 gp=0xc00027ba40 m=nil [select]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0xc0000459e8?, 0x2?, 0x4?, 0x0?, 0xc000045810?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000045620 sp=0xc000045600 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.selectgo(0xc0000459e8, 0xc00004580c, 0xc00088c000?, 0x0, 0x1?, 0x1) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/select.go:351 +0x837 fp=0xc000045758 sp=0xc000045620 pc=0x564acb018c17 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc0002f90e0, {0x564acc8d5440, 0xc00079a0e0}, 0xc000476280) 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:974 +0xc86 fp=0xc000045ac0 sp=0xc000045758 pc=0x564acb65ee26 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x564acc8d5440?, 0xc00079a0e0?}, 0xc0008bfb40?) 3月 10 09:33:22 jzswai ollama[3872788]: <autogenerated>:1 +0x36 fp=0xc000045af0 sp=0xc000045ac0 pc=0x564acb664456 3月 10 09:33:22 jzswai ollama[3872788]: net/http.HandlerFunc.ServeHTTP(0xc0005ba780?, {0x564acc8d5440?, 0xc00079a0e0?}, 0xc0008bfb60?) 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:2294 +0x29 fp=0xc000045b18 sp=0xc000045af0 pc=0x564acb33a8c9 3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*ServeMux).ServeHTTP(0x564acafdeac5?, {0x564acc8d5440, 0xc00079a0e0}, 0xc000476280) 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:2822 +0x1c4 fp=0xc000045b68 sp=0xc000045b18 pc=0x564acb33c7c4 3月 10 09:33:22 jzswai ollama[3872788]: net/http.serverHandler.ServeHTTP({0x564acc8d1730?}, {0x564acc8d5440?, 0xc00079a0e0?}, 0x1?) 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:3301 +0x8e fp=0xc000045b98 sp=0xc000045b68 pc=0x564acb35a24e 3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*conn).serve(0xc0001ae750, {0x564acc8d7b98, 0xc0005df800}) 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:2102 +0x625 fp=0xc000045fb8 sp=0xc000045b98 pc=0x564acb338dc5 3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*Server).Serve.gowrap3() 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:3454 +0x28 fp=0xc000045fe0 sp=0xc000045fb8 pc=0x564acb33e688 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by net/http.(*Server).Serve in goroutine 1 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:3454 +0x485 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 1758 gp=0xc000583dc0 m=nil [IO wait]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0xc009445e00?, 0xc009445e50?, 0x0?, 0x80?, 0xb?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc0006f45d8 sp=0xc0006f45b8 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.netpollblock(0x564acb05d798?, 0xcafd34a6?, 0x4a?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/netpoll.go:575 +0xf7 fp=0xc0006f4610 sp=0xc0006f45d8 pc=0x564acafff097 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.runtime_pollWait(0x7f9cb20525b8, 0x72) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/netpoll.go:351 +0x85 fp=0xc0006f4630 sp=0xc0006f4610 pc=0x564acb039105 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*pollDesc).wait(0xc0005c1500?, 0xc0005df901?, 0x0) 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006f4658 sp=0xc0006f4630 pc=0x564acb0c1487 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*pollDesc).waitRead(...) 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll/fd_poll_runtime.go:89 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll.(*FD).Read(0xc0005c1500, {0xc0005df901, 0x1, 0x1}) 3月 10 09:33:22 jzswai ollama[3872788]: internal/poll/fd_unix.go:165 +0x27a fp=0xc0006f46f0 sp=0xc0006f4658 pc=0x564acb0c277a 3月 10 09:33:22 jzswai ollama[3872788]: net.(*netFD).Read(0xc0005c1500, {0xc0005df901?, 0xc0007141d8?, 0xc0006f4770?}) 3月 10 09:33:22 jzswai ollama[3872788]: net/fd_posix.go:55 +0x25 fp=0xc0006f4738 sp=0xc0006f46f0 pc=0x564acb137da5 3月 10 09:33:22 jzswai ollama[3872788]: net.(*conn).Read(0xc00013ab00, {0xc0005df901?, 0xc009449c70?, 0xc009449cc0?}) 3月 10 09:33:22 jzswai ollama[3872788]: net/net.go:194 +0x45 fp=0xc0006f4780 sp=0xc0006f4738 pc=0x564acb146165 3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*connReader).backgroundRead(0xc0005df8f0) 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:690 +0x37 fp=0xc0006f47c8 sp=0xc0006f4780 pc=0x564acb332c97 3月 10 09:33:22 jzswai ollama[3872788]: net/http.(*connReader).startBackgroundRead.gowrap2() 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:686 +0x25 fp=0xc0006f47e0 sp=0xc0006f47c8 pc=0x564acb332bc5 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0006f47e8 sp=0xc0006f47e0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by net/http.(*connReader).startBackgroundRead in goroutine 21 3月 10 09:33:22 jzswai ollama[3872788]: net/http/server.go:686 +0xb6 3月 10 09:33:22 jzswai ollama[3872788]: goroutine 1439 gp=0xc0007b6e00 m=nil [chan receive]: 3月 10 09:33:22 jzswai ollama[3872788]: runtime.gopark(0x30?, 0x564acc813260?, 0x1?, 0xba?, 0xc000b26b18?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/proc.go:435 +0xce fp=0xc000b26ad0 sp=0xc000b26ab0 pc=0x564acb039eee 3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv(0xc0007442a0, 0x0, 0x1) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/chan.go:664 +0x445 fp=0xc000b26b48 sp=0xc000b26ad0 pc=0x564acafd6085 3月 10 09:33:22 jzswai ollama[3872788]: runtime.chanrecv1(0x564acc3378ec?, 0x2c?) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/chan.go:506 +0x12 fp=0xc000b26b70 sp=0xc000b26b48 pc=0x564acafd5c12 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).computeBatch(0xc0002f90e0, {0x2, {0x564acc8e4130, 0xc000730300}, {0x564acc8f1570, 0xc0006d5db8}, {0xc000b1c148, 0x1, 0x1}, {{0x564acc8f1570, ...}, ...}, ...}) 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:646 +0x185 fp=0xc000b26ef0 sp=0xc000b26b70 pc=0x564acb65c2c5 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run.gowrap1() 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:459 +0x58 fp=0xc000b26fe0 sp=0xc000b26ef0 pc=0x564acb65a698 3月 10 09:33:22 jzswai ollama[3872788]: runtime.goexit({}) 3月 10 09:33:22 jzswai ollama[3872788]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000b26fe8 sp=0xc000b26fe0 pc=0x564acb041e61 3月 10 09:33:22 jzswai ollama[3872788]: created by github.com/ollama/ollama/runner/ollamarunner.(*Server).run in goroutine 20 3月 10 09:33:22 jzswai ollama[3872788]: github.com/ollama/ollama/runner/ollamarunner/runner.go:459 +0x2cd 3月 10 09:33:22 jzswai ollama[3872788]: rax 0x0 3月 10 09:33:22 jzswai ollama[3872788]: rbx 0x7f9ad1fff640 3月 10 09:33:22 jzswai ollama[3872788]: rcx 0x7f9cb1e969fc 3月 10 09:33:22 jzswai ollama[3872788]: rdx 0x6 3月 10 09:33:22 jzswai ollama[3872788]: rdi 0x3cf144 3月 10 09:33:22 jzswai ollama[3872788]: rsi 0x3cf1a2 3月 10 09:33:22 jzswai ollama[3872788]: rbp 0x3cf1a2 3月 10 09:33:22 jzswai ollama[3872788]: rsp 0x7f9ad1ffdd20 3月 10 09:33:22 jzswai ollama[3872788]: r8 0x7f9ad1ffddf0 3月 10 09:33:22 jzswai ollama[3872788]: r9 0x7f94e9ba9e90 3月 10 09:33:22 jzswai ollama[3872788]: r10 0x8 3月 10 09:33:22 jzswai ollama[3872788]: r11 0x246 3月 10 09:33:22 jzswai ollama[3872788]: r12 0x6 3月 10 09:33:22 jzswai ollama[3872788]: r13 0x16 3月 10 09:33:22 jzswai ollama[3872788]: r14 0x7f9c5029242e 3月 10 09:33:22 jzswai ollama[3872788]: r15 0x7f9ac90d5508 3月 10 09:33:22 jzswai ollama[3872788]: rip 0x7f9cb1e969fc 3月 10 09:33:22 jzswai ollama[3872788]: rflags 0x246 3月 10 09:33:22 jzswai ollama[3872788]: cs 0x33 3月 10 09:33:22 jzswai ollama[3872788]: fs 0x0 3月 10 09:33:22 jzswai ollama[3872788]: gs 0x0 3月 10 09:33:22 jzswai ollama[3872788]: time=2026-03-10T09:33:22.799+08:00 level=ERROR source=server.go:303 msg="llama runner terminated" error="exit status 2" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.17.7
GiteaMirror added the bug label 2026-04-12 22:27:38 -05:00
GiteaMirror commented 2026-04-12 22:27:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 10, 2026):

Please include the log from before the crash.

<!-- gh-comment-id:4030934907 --> @rick-github commented on GitHub (Mar 10, 2026): Please include the log from before the crash.
GiteaMirror commented 2026-04-12 22:27:40 -05:00
Author
Owner
Copy Link

@cedarice commented on GitHub (Mar 12, 2026):

ollama.json

<!-- gh-comment-id:4043113238 --> @cedarice commented on GitHub (Mar 12, 2026): [ollama.json](https://github.com/user-attachments/files/25921827/ollama.json)
GiteaMirror commented 2026-04-12 22:27:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 12, 2026):

Plain text logs, as in the OP.

<!-- gh-comment-id:4043129596 --> @rick-github commented on GitHub (Mar 12, 2026): Plain text logs, as in the OP.
GiteaMirror commented 2026-04-12 22:27:41 -05:00
Author
Owner
Copy Link

@cedarice commented on GitHub (Mar 12, 2026):

ollama.txt

<!-- gh-comment-id:4043431450 --> @cedarice commented on GitHub (Mar 12, 2026): [ollama.txt](https://github.com/user-attachments/files/25923001/ollama.txt)
GiteaMirror commented 2026-04-12 22:27:42 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 12, 2026):

3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=server.go:430 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-d4b8b4f4c350f5d322dc8235175eeae02d32c6f3fd70bdb9ea481e3abb7d7fc4 --port 37399"
3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=server.go:757 msg="loading model" "model layers"=65 requested=-1
3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.990+08:00 level=INFO source=ggml.go:136 msg="" architecture=qwen35 file_type=Q4_K_M name="" description="" num_tensors=1307 num_key_values=53

This model doesn't appear to be the same as the one in the ollama main library, where was it pulled from?

Error: an error was encountered while running the model: CUDA error: out of memory

See here for ways to mitigate OOM errors.

<!-- gh-comment-id:4047767264 --> @rick-github commented on GitHub (Mar 12, 2026): ``` 3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=server.go:430 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-d4b8b4f4c350f5d322dc8235175eeae02d32c6f3fd70bdb9ea481e3abb7d7fc4 --port 37399" 3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=server.go:757 msg="loading model" "model layers"=65 requested=-1 3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.990+08:00 level=INFO source=ggml.go:136 msg="" architecture=qwen35 file_type=Q4_K_M name="" description="" num_tensors=1307 num_key_values=53 ``` This model doesn't appear to be the same as the one in the ollama main library, where was it pulled from? ``` Error: an error was encountered while running the model: CUDA error: out of memory ``` See [here](https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288) for ways to mitigate OOM errors.
GiteaMirror commented 2026-04-12 22:27:42 -05:00
Author
Owner
Copy Link

@cedarice commented on GitHub (Mar 13, 2026):

I removed the qwen3.5:27b model and pulled it again using ollama pull qwen3.5:27b, but the issue persists. However, I haven't encountered any issues with other models in the qwen3 series.

<!-- gh-comment-id:4051369102 --> @cedarice commented on GitHub (Mar 13, 2026): I removed the qwen3.5:27b model and pulled it again using `ollama pull qwen3.5:27b`, but the issue persists. However, I haven't encountered any issues with other models in the qwen3 series.
GiteaMirror commented 2026-04-12 22:27:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 13, 2026):

3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=sched.go:496 msg="gpu memory" id=GPU-c3675116-fef2-b987-b9a3-97ab58d31710 library=CUDA available="21.5 GiB" free="22.0 GiB" minimum="457.0 MiB" overhead="0 B"
3月 12 08:23:08 jzswai ollama[3872788]: time=2026-03-12T08:23:08.214+08:00 level=INFO source=device.go:272 msg="total memory" size="21.4 GiB"

The GPU has 21.5G available and the model needs 21.4G so this seems like memory estimation is cutting it too close. This should be investigated but in the meantime the OOM mitigations may help.

<!-- gh-comment-id:4051545723 --> @rick-github commented on GitHub (Mar 13, 2026): ``` 3月 12 08:23:04 jzswai ollama[3872788]: time=2026-03-12T08:23:04.863+08:00 level=INFO source=sched.go:496 msg="gpu memory" id=GPU-c3675116-fef2-b987-b9a3-97ab58d31710 library=CUDA available="21.5 GiB" free="22.0 GiB" minimum="457.0 MiB" overhead="0 B" 3月 12 08:23:08 jzswai ollama[3872788]: time=2026-03-12T08:23:08.214+08:00 level=INFO source=device.go:272 msg="total memory" size="21.4 GiB" ``` The GPU has 21.5G available and the model needs 21.4G so this seems like memory estimation is cutting it too close. This should be investigated but in the meantime the OOM mitigations may help.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#9537
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?