github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #15950] [0.20.5] Runner accepts TCP connection but request never reaches work loop — same shape as resolved-in-0.20.2 #15258 #87845

New Issue
Open
opened 2026-05-10 06:26:46 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-10 06:26:46 -05:00
Owner
Copy Link

Originally created by @maximus-dss on GitHub (May 3, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15950

Summary

We hit the same shape as #15258 on Ollama 0.20.5, despite that issue being closed as resolved in 0.20.2 by its reporter. The pattern: large models pinned in memory for hours, then /api/generate hangs indefinitely with zero bytes returned, while listing endpoints (/api/version, /api/tags, /api/ps, /api/embeddings) keep responding normally.

Forensic analysis (stack samples + lsof) shows this is not a daemon-scheduler issue and not a deadlock primitive — it's a runner-side request-receive failure: the daemon dispatches correctly (TCP ESTABLISHED conn to runner), the runner accepts the connection at the kernel level, but the request payload never reaches the runner's main work loop. The main thread stays in its idle wait-for-work _pthread_cond_wait instead of waking to process the inbound request.

Filing this as a probable regression after #15258's 0.20.2 fix (or an adjacent variant the original fix didn't cover). May be related to but distinct from #15923 (which is multi-turn tool-calling crashes — error-emitting, not silent-hang) and #15350 (Gemma 4 Flash Attention hang).

Environment

  • Hardware: Mac Studio M3 Ultra, 256 GB unified memory (Mac15,14, 28 physical cores, hw.memsize=274877906944)
  • OS: macOS 26.4.1 (25E253)
  • Ollama: 0.20.5 (homebrew /opt/homebrew/opt/ollama/bin/ollama serve, supervised by com.ollama.serve LaunchAgent)
  • Env vars (per launchctl print):
    • OLLAMA_NUM_PARALLEL=4
    • OLLAMA_MAX_LOADED_MODELS=4
    • OLLAMA_FLASH_ATTENTION=1
    • OLLAMA_KV_CACHE_TYPE=q4_0
    • OLLAMA_KEEP_ALIVE=5m (daemon default — but see note below)
  • Effective keep-alive: A cron job (*/2 * * * *) actively pings each loaded model with keep_alive: -1 to pin them indefinitely. So at incident time, qwen3.5:122b-32k + glm-4.7-flash + gemma4 had been resident for ~3 hours of pinned idle.

Affected models

  • qwen3.5:122b-32k (84 GB resident, MoE) — STUCK
  • glm-4.7-flash:latest (45 GB resident, MoE) — STUCK
  • gemma4:latest (15 GB resident, dense) — UNAFFECTED, kept responding cleanly to direct /api/generate throughout the incident

The smaller dense model survived. Both larger MoE models hung. Pattern correlates with model size and/or MoE arch, but I don't have enough samples to claim either with confidence.

Symptom

After ~3 hours of pinned-idle state:

  • /api/version → 200, immediate
  • /api/tags → 200, lists all models
  • /api/ps → 200, lists qwen3.5 + glm + gemma4 as loaded with expires_at: "2318-08-12..." (the keep-alive=-1 sentinel)
  • /api/embeddings (untested at incident, but per #15258 these typically work)
  • /api/generate for qwen3.5:122b-32k → hangs indefinitely (60s curl timeout, then HTTP 000)
  • /api/generate for glm-4.7-flash:latest → same hang
  • /api/generate for gemma4:latest → 200 in 3-4s, normal output

This matches #15258's symptom exactly except for: (a) we're on 0.20.5 not 0.20.0, (b) only the largest two MoE models exhibited it, gemma4 was fine.

Forensic smoking gun

I captured sample <pid> 5 (5-sec stack profile) for both stuck runners while in the hung state:

Both runners' main threads:

Thread_NNNNNNNN   DispatchQueue_1: com.apple.main-thread  (serial)
  + N ???  (in <unknown binary>)  [...]
  +   N ???  (in ollama)  load address 0x10XXXXXXX + 0x8b5d4
  +     N ???  (in ollama)  load address 0x10XXXXXXX + 0x8c7d8
  +       N _pthread_cond_wait  (in libsystem_pthread.dylib) + 980
  +         N __psynch_cvwait   (in libsystem_kernel.dylib) + 8

The two relative offsets (+0x8b5d4 and +0x8c7d8) are identical across both stuck runners despite different load addresses. This is the runner's normal idle "wait for work" loop. Not a deadlock primitive — these threads are correctly sleeping waiting for work to be enqueued.

lsof comparison at incident time:

Runner TCP state
glm-4.7-flash (PID 58100) LISTEN only on its port
qwen3.5:122b-32k (PID 57826) LISTEN + 1 ESTABLISHED (localhost:64107->localhost:65255)

The qwen3.5 ESTABLISHED connection is the proof: the daemon DID dispatch a request to this runner (TCP handshake completed). The runner accepted at the kernel level (FD 5 is in ESTABLISHED state). But the runner's main work loop never woke to process it — the request payload was received into kernel buffers but never reached the work queue or signaled the cond_var.

Possible failure modes:

  1. The runner's HTTP-receive thread is alive but isn't notifying the work loop
  2. The runner's HTTP-receive thread is itself stuck (would also show in stack samples — and it does, ~33 of 38 threads in cond_wait state, consistent with all worker threads idle)
  3. A lost wake signal between the receive path and the main loop

Either way, the runner has the daemon's request bytes sitting in its socket buffer (or already read into userland) but is failing to act on them.

Mitigation that worked

Full daemon bootout + bootstrap:

launchctl bootout "gui/$(id -u)/com.ollama.serve"
sleep 3
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/com.ollama.serve.plist

After respawn, all three models cold-loaded on first request and responded normally.

Mitigation that did NOT work

kill -9 on the stuck runner PIDs alone. After SIGKILL:

  • Runner processes terminated correctly (verified via ps -p)
  • /api/ps continued listing the killed runners' models as loaded with expires_at in the future — stale daemon scheduler state
  • New /api/generate requests for those models hung waiting for the (now-dead) runner

Daemon's runner-tracking logic doesn't notice runner death until something else times out / cleans up, which apparently doesn't happen on its own. Only a daemon restart cleared the scheduler's fictional "loaded" entries.

This bears on the bug location: the daemon's runner-lifecycle bookkeeping is also fragile when runners die in this state, but that's a secondary concern — the primary bug is the runner-side request-receive failure.

Reproduction (untested)

I did NOT attempt reproduction — the test conditions would deliberately deadlock another runner and disrupt our production fleet. If you want me to attempt reproduction, please indicate what conditions to vary (KEEP_ALIVE pinning duration, model size, NUM_PARALLEL boundary, FLASH_ATTENTION on/off, etc.) and I'll set up a controlled run.

Forensic capture artifacts

Available on request:

  • sample-<pid>-{qwen,glm}.txt — 5-sec stack profiles per stuck runner (~115 KB each, 38 threads each)
  • lsof-<pid>-{qwen,glm}.txt — full FD inventory at stuck-time
  • api-ps-pre-kill.json — daemon's view of stuck state
  • system-memory.txt — vm_stat/sysctl vm.swapusage/top at incident time
  • ioreg-metal.txt — Metal GPU state
  • ollama-server-tail.log — server.log tail at incident time
  • com.ollama.serve.{pre,post}-restart.dump — launchctl print env captures

Related issues

  • #15258 — ancestor; same shape on 0.20.0 Apple Silicon, reporter said resolved in 0.20.2. We're hitting the same shape on 0.20.5.
  • #15923 — open; 0.20.5 + Apple Silicon. Different shape (multi-turn tool-calling crashes with explicit error signatures). Adjacent area, probably distinct cause.
  • #15350 — closed; Gemma 4 Flash Attention hang. We have FLASH_ATTENTION=1; may or may not be a contributing factor here.

What we'd find useful from maintainers

  1. Confirmation whether this is the same as #15258 returning, or a distinct adjacent bug
  2. Whether the 0.20.2 fix has unit-test coverage for the runner-side receive path that would catch a regression
  3. Guidance on safe reproduction (which env knob most reliably triggers it) before we attempt repro on a production machine
  4. Whether anyone else has filed this on 0.20.3-0.20.5 that we missed in our pre-filing search
Originally created by @maximus-dss on GitHub (May 3, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15950 ## Summary We hit the same shape as #15258 on **Ollama 0.20.5**, despite that issue being closed as resolved in 0.20.2 by its reporter. The pattern: large models pinned in memory for hours, then `/api/generate` hangs indefinitely with zero bytes returned, while listing endpoints (`/api/version`, `/api/tags`, `/api/ps`, `/api/embeddings`) keep responding normally. Forensic analysis (stack samples + lsof) shows this is **not a daemon-scheduler issue and not a deadlock primitive** — it's a **runner-side request-receive failure**: the daemon dispatches correctly (TCP ESTABLISHED conn to runner), the runner accepts the connection at the kernel level, but the request payload never reaches the runner's main work loop. The main thread stays in its idle wait-for-work `_pthread_cond_wait` instead of waking to process the inbound request. Filing this as a probable **regression after #15258's 0.20.2 fix** (or an adjacent variant the original fix didn't cover). May be related to but distinct from #15923 (which is multi-turn tool-calling crashes — error-emitting, not silent-hang) and #15350 (Gemma 4 Flash Attention hang). ## Environment - **Hardware:** Mac Studio M3 Ultra, 256 GB unified memory (Mac15,14, 28 physical cores, `hw.memsize=274877906944`) - **OS:** macOS 26.4.1 (25E253) - **Ollama:** 0.20.5 (homebrew `/opt/homebrew/opt/ollama/bin/ollama serve`, supervised by `com.ollama.serve` LaunchAgent) - **Env vars (per `launchctl print`):** - `OLLAMA_NUM_PARALLEL=4` - `OLLAMA_MAX_LOADED_MODELS=4` - `OLLAMA_FLASH_ATTENTION=1` - `OLLAMA_KV_CACHE_TYPE=q4_0` - `OLLAMA_KEEP_ALIVE=5m` (daemon default — but see note below) - **Effective keep-alive:** A cron job (`*/2 * * * *`) actively pings each loaded model with `keep_alive: -1` to pin them indefinitely. So at incident time, qwen3.5:122b-32k + glm-4.7-flash + gemma4 had been resident for ~3 hours of pinned idle. ## Affected models - **`qwen3.5:122b-32k`** (84 GB resident, MoE) — STUCK - **`glm-4.7-flash:latest`** (45 GB resident, MoE) — STUCK - **`gemma4:latest`** (15 GB resident, dense) — UNAFFECTED, kept responding cleanly to direct `/api/generate` throughout the incident The smaller dense model survived. Both larger MoE models hung. Pattern correlates with model size and/or MoE arch, but I don't have enough samples to claim either with confidence. ## Symptom After ~3 hours of pinned-idle state: - `/api/version` → 200, immediate - `/api/tags` → 200, lists all models - `/api/ps` → 200, lists qwen3.5 + glm + gemma4 as loaded with `expires_at: "2318-08-12..."` (the keep-alive=-1 sentinel) - `/api/embeddings` (untested at incident, but per #15258 these typically work) - **`/api/generate` for `qwen3.5:122b-32k` → hangs indefinitely** (60s curl timeout, then HTTP 000) - **`/api/generate` for `glm-4.7-flash:latest` → same hang** - `/api/generate` for `gemma4:latest` → 200 in 3-4s, normal output This matches #15258's symptom exactly except for: (a) we're on 0.20.5 not 0.20.0, (b) only the largest two MoE models exhibited it, gemma4 was fine. ## Forensic smoking gun I captured `sample <pid> 5` (5-sec stack profile) for both stuck runners while in the hung state: **Both runners' main threads:** ``` Thread_NNNNNNNN DispatchQueue_1: com.apple.main-thread (serial) + N ??? (in <unknown binary>) [...] + N ??? (in ollama) load address 0x10XXXXXXX + 0x8b5d4 + N ??? (in ollama) load address 0x10XXXXXXX + 0x8c7d8 + N _pthread_cond_wait (in libsystem_pthread.dylib) + 980 + N __psynch_cvwait (in libsystem_kernel.dylib) + 8 ``` The two relative offsets (`+0x8b5d4` and `+0x8c7d8`) are **identical** across both stuck runners despite different load addresses. This is the runner's normal idle "wait for work" loop. **Not a deadlock primitive — these threads are correctly sleeping waiting for work to be enqueued.** **lsof comparison at incident time:** | Runner | TCP state | |---|---| | `glm-4.7-flash` (PID 58100) | LISTEN only on its port | | `qwen3.5:122b-32k` (PID 57826) | LISTEN + 1 ESTABLISHED (`localhost:64107->localhost:65255`) | The qwen3.5 ESTABLISHED connection is the proof: **the daemon DID dispatch a request to this runner** (TCP handshake completed). The runner accepted at the kernel level (FD 5 is in ESTABLISHED state). But the runner's main work loop never woke to process it — the request payload was received into kernel buffers but never reached the work queue or signaled the cond_var. Possible failure modes: 1. The runner's HTTP-receive thread is alive but isn't notifying the work loop 2. The runner's HTTP-receive thread is itself stuck (would also show in stack samples — and it does, ~33 of 38 threads in cond_wait state, consistent with all worker threads idle) 3. A lost wake signal between the receive path and the main loop Either way, the runner has the daemon's request bytes sitting in its socket buffer (or already read into userland) but is failing to act on them. ## Mitigation that worked Full daemon `bootout` + `bootstrap`: ```bash launchctl bootout "gui/$(id -u)/com.ollama.serve" sleep 3 launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/com.ollama.serve.plist ``` After respawn, all three models cold-loaded on first request and responded normally. ## Mitigation that did NOT work `kill -9` on the stuck runner PIDs alone. After SIGKILL: - Runner processes terminated correctly (verified via `ps -p`) - `/api/ps` continued listing the killed runners' models as `loaded` with `expires_at` in the future — **stale daemon scheduler state** - New `/api/generate` requests for those models hung waiting for the (now-dead) runner Daemon's runner-tracking logic doesn't notice runner death until something else times out / cleans up, which apparently doesn't happen on its own. Only a daemon restart cleared the scheduler's fictional "loaded" entries. This bears on the bug location: the daemon's runner-lifecycle bookkeeping is also fragile when runners die in this state, but that's a secondary concern — the primary bug is the runner-side request-receive failure. ## Reproduction (untested) I did NOT attempt reproduction — the test conditions would deliberately deadlock another runner and disrupt our production fleet. If you want me to attempt reproduction, please indicate what conditions to vary (KEEP_ALIVE pinning duration, model size, NUM_PARALLEL boundary, FLASH_ATTENTION on/off, etc.) and I'll set up a controlled run. ## Forensic capture artifacts Available on request: - `sample-<pid>-{qwen,glm}.txt` — 5-sec stack profiles per stuck runner (~115 KB each, 38 threads each) - `lsof-<pid>-{qwen,glm}.txt` — full FD inventory at stuck-time - `api-ps-pre-kill.json` — daemon's view of stuck state - `system-memory.txt` — `vm_stat`/`sysctl vm.swapusage`/`top` at incident time - `ioreg-metal.txt` — Metal GPU state - `ollama-server-tail.log` — server.log tail at incident time - `com.ollama.serve.{pre,post}-restart.dump` — `launchctl print` env captures ## Related issues - **#15258** — ancestor; same shape on 0.20.0 Apple Silicon, reporter said resolved in 0.20.2. We're hitting the same shape on 0.20.5. - **#15923** — open; 0.20.5 + Apple Silicon. Different shape (multi-turn tool-calling crashes with explicit error signatures). Adjacent area, probably distinct cause. - **#15350** — closed; Gemma 4 Flash Attention hang. We have FLASH_ATTENTION=1; may or may not be a contributing factor here. ## What we'd find useful from maintainers 1. Confirmation whether this is the same as #15258 returning, or a distinct adjacent bug 2. Whether the 0.20.2 fix has unit-test coverage for the runner-side receive path that would catch a regression 3. Guidance on safe reproduction (which env knob most reliably triggers it) before we attempt repro on a production machine 4. Whether anyone else has filed this on 0.20.3-0.20.5 that we missed in our pre-filing search
GiteaMirror added the bug label 2026-05-10 06:26:46 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#87845
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?