github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-21 13:11:39 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3469] Ollama API Inference Hangs with Larger Models after a While #2137

New Issue
Closed
opened 2026-04-12 12:22:12 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-12 12:22:12 -05:00
Owner
Copy Link

Originally created by @batteryshark on GitHub (Apr 3, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3469

What is the issue?

While running models that take up most of my VRAM, inference works fine via api (including the OpenAI /chat/completions) up to a point and then it stops responding while making a bunch of log messages about "slot context shift"

What did you expect to see?

The API responding properly

Steps to reproduce

Load a large model like nous mixtral dpo and fill most of your VRAM, make several API calls to the model, and eventually it will stop responding.

Are there any recent changes that introduced the issue?

It's been like this for a while, I started noticing it more after using the native Windows version.

OS

Windows

Architecture

amd64

Platform

No response

Ollama version

0.1.30

GPU

Nvidia

GPU info

RTX 4090

CPU

Intel

Other software

No response

Originally created by @batteryshark on GitHub (Apr 3, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3469 ### What is the issue? While running models that take up most of my VRAM, inference works fine via api (including the OpenAI /chat/completions) up to a point and then it stops responding while making a bunch of log messages about "slot context shift" ### What did you expect to see? The API responding properly ### Steps to reproduce Load a large model like nous mixtral dpo and fill most of your VRAM, make several API calls to the model, and eventually it will stop responding. ### Are there any recent changes that introduced the issue? It's been like this for a while, I started noticing it more after using the native Windows version. ### OS Windows ### Architecture amd64 ### Platform _No response_ ### Ollama version 0.1.30 ### GPU Nvidia ### GPU info RTX 4090 ### CPU Intel ### Other software _No response_
GiteaMirror added the bugneeds more info labels 2026-04-12 12:22:13 -05:00
GiteaMirror commented 2026-04-12 12:22:14 -05:00
Author
Owner
Copy Link

@gamersover commented on GitHub (Apr 3, 2024):

I have encountered the same issue. When the messages are very long, the model hangs forever. From observing the stream response behavior, it appears to stop outputting tokens after a few tokens have been output.

<!-- gh-comment-id:2033738977 --> @gamersover commented on GitHub (Apr 3, 2024): I have encountered the same issue. When the messages are very long, the model hangs forever. From observing the stream response behavior, it appears to stop outputting tokens after a few tokens have been output.
GiteaMirror commented 2026-04-12 12:22:14 -05:00
Author
Owner
Copy Link

@mofanke commented on GitHub (Apr 3, 2024):

https://github.com/ollama/ollama/issues/2048 maybe this releated

<!-- gh-comment-id:2034265171 --> @mofanke commented on GitHub (Apr 3, 2024): https://github.com/ollama/ollama/issues/2048 maybe this releated
GiteaMirror commented 2026-04-12 12:22:14 -05:00
Author
Owner
Copy Link

@batteryshark commented on GitHub (Apr 3, 2024):

It could be - what's interesting is I've seen the behavior happen with temperature 0 and relatively short prompts as well that when rerun are fine. Maybe something is being cached between requests that's causing this? I've noticed it more in JSON mode and the OpenAI endpoint, but it doesn't seem to be limited to those.

<!-- gh-comment-id:2034376923 --> @batteryshark commented on GitHub (Apr 3, 2024): It could be - what's interesting is I've seen the behavior happen with temperature 0 and relatively short prompts as well that when rerun are fine. Maybe something is being cached between requests that's causing this? I've noticed it more in JSON mode and the OpenAI endpoint, but it doesn't seem to be limited to those.
GiteaMirror commented 2026-04-12 12:22:14 -05:00
Author
Owner
Copy Link

@batteryshark commented on GitHub (Apr 4, 2024):

As an update to this, it doesn't seem like llama.cpp has an issue so it's likely something specific to ollama.

<!-- gh-comment-id:2036134716 --> @batteryshark commented on GitHub (Apr 4, 2024): As an update to this, it doesn't seem like llama.cpp has an issue so it's likely something specific to ollama.
GiteaMirror commented 2026-04-12 12:22:15 -05:00
Author
Owner
Copy Link

@yaohaizhou commented on GitHub (Apr 10, 2024):

I think this may be caused by the mismatch of KV cache. I am still trying to find the final reason yet.
Before the context shift, many previous embeddings are stored in the KV cache.
It seems that after the context shift, lots of KV cache become missed for many iters. So in each time, the embeddings must be calculated again which costs lots of time.

BTW, this is found by Felix Wang

<!-- gh-comment-id:2047668929 --> @yaohaizhou commented on GitHub (Apr 10, 2024): I think this may be caused by the mismatch of KV cache. I am still trying to find the final reason yet. Before the context shift, many previous embeddings are stored in the KV cache. It seems that after the context shift, lots of KV cache become missed for many iters. So in each time, the embeddings must be calculated again which costs lots of time. BTW, this is found by [Felix Wang](https://github.com/philia897)
GiteaMirror commented 2026-04-12 12:22:15 -05:00
Author
Owner
Copy Link

@philia897 commented on GitHub (Apr 10, 2024):

@batteryshark
Encounter this issue when the size of cache_tokens reaches the context size.
In ext_server/server.cpp, line 1608:

llama_kv_cache_seq_rm (ctx, slot.id, n_keep , n_keep + n_discard);
llama_kv_cache_seq_add(ctx, slot.id, n_keep + n_discard, system_tokens.size() + slot.n_past, -n_discard);

for (size_t i = n_keep + n_discard; i < slot.cache_tokens.size(); i++)
{
    slot.cache_tokens[i - n_discard] = slot.cache_tokens[i];
}

slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard);

slot.n_past -= n_discard;

The cache_tokens for this slot is updated. However, in line 1775:

slot.n_past = common_part(slot.cache_tokens, prompt_tokens);

Seems the server tries to find the common part between cache_tokens and prompt_tokens. prompt_tokens remember the whole history but the cache_tokens is cut off during context shift. So you can imagine the length of common part will be 1 in most cases.

The consequence is that, in line 1831:

int p0 = (int) system_tokens.size() + slot.n_past;
LOG_INFO("kv cache rm [p0, end)", {
    { "slot_id", slot.id },
    { "task_id", slot.task_id },
    { "p0",      p0 }
});
llama_kv_cache_seq_rm(ctx, slot.id, p0, -1);

p0 will be 1, so almost all the kv cache sequences will be removed. From now on, kv cache can not work anymore because the processed tokens will fill the kv cache, then trigger the context shift, cutting off the cache_tokens. Then the kv cache will be reset because of the mismatch between the cache_tokens and prompt_tokens.

Hope this could help😊.

<!-- gh-comment-id:2047823704 --> @philia897 commented on GitHub (Apr 10, 2024): @batteryshark Encounter this issue when the size of cache_tokens reaches the context size. In `ext_server/server.cpp`, line 1608: ``` llama_kv_cache_seq_rm (ctx, slot.id, n_keep , n_keep + n_discard); llama_kv_cache_seq_add(ctx, slot.id, n_keep + n_discard, system_tokens.size() + slot.n_past, -n_discard); for (size_t i = n_keep + n_discard; i < slot.cache_tokens.size(); i++) { slot.cache_tokens[i - n_discard] = slot.cache_tokens[i]; } slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard); slot.n_past -= n_discard; ``` The `cache_tokens` for this slot is updated. However, in line 1775: ``` slot.n_past = common_part(slot.cache_tokens, prompt_tokens); ``` Seems the server tries to find the common part between `cache_tokens` and `prompt_tokens`. `prompt_tokens` remember the whole history but the `cache_tokens` is cut off during context shift. So you can imagine the length of common part will be 1 in most cases. The consequence is that, in line 1831: ``` int p0 = (int) system_tokens.size() + slot.n_past; LOG_INFO("kv cache rm [p0, end)", { { "slot_id", slot.id }, { "task_id", slot.task_id }, { "p0", p0 } }); llama_kv_cache_seq_rm(ctx, slot.id, p0, -1); ``` p0 will be 1, so almost all the kv cache sequences will be removed. From now on, kv cache can not work anymore because the processed tokens will fill the kv cache, then trigger the context shift, cutting off the cache_tokens. Then the kv cache will be reset because of the mismatch between the cache_tokens and prompt_tokens. Hope this could help😊.
GiteaMirror commented 2026-04-12 12:22:15 -05:00
Author
Owner
Copy Link

@peebles commented on GitHub (Apr 23, 2024):

Been seeing this too, after 0.1.29 -> 0.1.31 I think, but not sure. Works fine for a while (day or two), then infinite loop of

{
  "function": "update_slots",
  "level": "INFO",
  "line": 1601,
  "msg": "slot context shift",
  "n_cache_tokens": 16384,
  "n_ctx": 16384,
  "n_discard": 8191,
  "n_keep": 1,
  "n_left": 16382,
  "n_past": 16383,
  "n_system_tokens": 0,
  "slot_id": 0,
  "task_id": 8,
  "tid": "124603937681408",
  "timestamp": 1713703989
}

where "task_id" is the same. Messages come about 3 minutes apart. Restarting the server seems the only way out.

<!-- gh-comment-id:2071260463 --> @peebles commented on GitHub (Apr 23, 2024): Been seeing this too, after 0.1.29 -> 0.1.31 I think, but not sure. Works fine for a while (day or two), then infinite loop of ```json { "function": "update_slots", "level": "INFO", "line": 1601, "msg": "slot context shift", "n_cache_tokens": 16384, "n_ctx": 16384, "n_discard": 8191, "n_keep": 1, "n_left": 16382, "n_past": 16383, "n_system_tokens": 0, "slot_id": 0, "task_id": 8, "tid": "124603937681408", "timestamp": 1713703989 } ``` where "task_id" is the same. Messages come about 3 minutes apart. Restarting the server seems the only way out.
GiteaMirror commented 2026-04-12 12:22:15 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 23, 2024):

Please give the new 0.4.0 RC a try and see how it performs. We've rewritten the caching logic to improve performance and reliability which should hopefully resolve this problem.

https://github.com/ollama/ollama/releases

<!-- gh-comment-id:2433050807 --> @dhiltgen commented on GitHub (Oct 23, 2024): Please give the new 0.4.0 RC a try and see how it performs. We've rewritten the caching logic to improve performance and reliability which should hopefully resolve this problem. https://github.com/ollama/ollama/releases
GiteaMirror commented 2026-04-12 12:22:16 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 19, 2024):

I'm going to close this since the issue has been sitting for a while. We can reopen if it's still a problem.

<!-- gh-comment-id:2555820545 --> @pdevine commented on GitHub (Dec 19, 2024): I'm going to close this since the issue has been sitting for a while. We can reopen if it's still a problem.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2137
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?