github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9355] Keep alive failed when use the openai API #52620

New Issue
Closed
opened 2026-04-28 23:53:35 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-28 23:53:35 -05:00
Owner
Copy Link

Originally created by @Andylau-BIT on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9355

What is the issue?

I used ollama to deploy deepseek-r1:671b and the parameter "OLLAMA_KEEP_ALIVE=-1"have been set. Then I use openai API endpoint ["/v/chat/completation"] to request. The GPU memory released after a few minutes.

Image

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

v0.5.9

Originally created by @Andylau-BIT on GitHub (Feb 26, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9355 ### What is the issue? I used ollama to deploy deepseek-r1:671b and the parameter "OLLAMA_KEEP_ALIVE=-1"have been set. Then I use openai API endpoint ["/v/chat/completation"] to request. The GPU memory released after a few minutes. ![Image](https://github.com/user-attachments/assets/d064158c-7ed7-4768-8b24-aa9bc34180b7) ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version v0.5.9
GiteaMirror added the bug label 2026-04-28 23:53:35 -05:00
GiteaMirror commented 2026-04-28 23:53:39 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Feb 26, 2025):

How to set the parameter "keep_alive" in openai API request. Other parameter to present user change the parameter "keep_alive" must be better

<!-- gh-comment-id:2683893805 --> @Andylau-BIT commented on GitHub (Feb 26, 2025): How to set the parameter "keep_alive" in openai API request. Other parameter to present user change the parameter "keep_alive" must be better
GiteaMirror commented 2026-04-28 23:53:39 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 26, 2025):

Using the openAI endpoint does not reset keep_alive:

$ ollama run qwen2.5:0.5b ''
$ ollama ps
NAME            ID              SIZE      PROCESSOR    UNTIL   
qwen2.5:0.5b    a8b0c5157701    1.2 GB    100% GPU     Forever    
$ curl localhost:11434/v1/chat/completions -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"hello"}],"stream":false}'
{"id":"chatcmpl-431","object":"chat.completion","created":1740569824,"model":"qwen2.5:0.5b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"你好！(Tingba!)很高兴见到你。有什么可以帮助你的吗？"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":17,"total_tokens":47}}
$ ollama ps
NAME            ID              SIZE      PROCESSOR    UNTIL   
qwen2.5:0.5b    a8b0c5157701    1.2 GB    100% GPU     Forever
<!-- gh-comment-id:2684720915 --> @rick-github commented on GitHub (Feb 26, 2025): Using the openAI endpoint does not reset `keep_alive`: ```console $ ollama run qwen2.5:0.5b '' $ ollama ps NAME ID SIZE PROCESSOR UNTIL qwen2.5:0.5b a8b0c5157701 1.2 GB 100% GPU Forever $ curl localhost:11434/v1/chat/completions -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"hello"}],"stream":false}' {"id":"chatcmpl-431","object":"chat.completion","created":1740569824,"model":"qwen2.5:0.5b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"你好！(Tingba!)很高兴见到你。有什么可以帮助你的吗？"},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":17,"total_tokens":47}} $ ollama ps NAME ID SIZE PROCESSOR UNTIL qwen2.5:0.5b a8b0c5157701 1.2 GB 100% GPU Forever ```
GiteaMirror commented 2026-04-28 23:53:40 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Feb 26, 2025):

您的邮件我已经收到！

<!-- gh-comment-id:2684722131 --> @Andylau-BIT commented on GitHub (Feb 26, 2025): 您的邮件我已经收到！
GiteaMirror commented 2026-04-28 23:53:40 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Feb 26, 2025):

The actual phenomenon is that when I sent the request, I did not set keep_alive, but instead set OLLAMA_KEEP_ALIVE=-1 when starting the Ollama service. However, the model still exhibited the restart behavior shown in the figure. The parameter to present users setting "keepalive" is necessary

Image

<!-- gh-comment-id:2684785107 --> @Andylau-BIT commented on GitHub (Feb 26, 2025): The actual phenomenon is that when I sent the request, I did not set keep_alive, but instead set OLLAMA_KEEP_ALIVE=-1 when starting the Ollama service. However, the model still exhibited the restart behavior shown in the figure. The parameter to present users setting "keepalive" is necessary ![Image](https://github.com/user-attachments/assets/d1d494c8-9168-4f6d-88d9-e2badbe11d9a)
GiteaMirror commented 2026-04-28 23:53:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 26, 2025):

The actual phenomenon is that when I sent the request, I did not set keep_alive, but instead set OLLAMA_KEEP_ALIVE=-1 when starting the Ollama service

This is exactly how my example is set up: no keep_alive in the API call, and OLLAMA_KEEP_ALIVE=-1 in the server. Calling the openAI endpoint does not reset keep_alive.

<!-- gh-comment-id:2684818619 --> @rick-github commented on GitHub (Feb 26, 2025): > The actual phenomenon is that when I sent the request, I did not set keep_alive, but instead set OLLAMA_KEEP_ALIVE=-1 when starting the Ollama service This is exactly how my example is set up: no `keep_alive` in the API call, and `OLLAMA_KEEP_ALIVE=-1` in the server. Calling the openAI endpoint does not reset `keep_alive`.
GiteaMirror commented 2026-04-28 23:53:42 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Feb 26, 2025):

Alright, thank you. I have restarted the service and will observe the changes in VRAM usage.

<!-- gh-comment-id:2684833617 --> @Andylau-BIT commented on GitHub (Feb 26, 2025): Alright, thank you. I have restarted the service and will observe the changes in VRAM usage.
GiteaMirror commented 2026-04-28 23:53:44 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 26, 2025):

Server logs may show the reason for the release of VRAM. It could be the runner crashing due to https://github.com/ollama/ollama/issues/5975.

<!-- gh-comment-id:2684842417 --> @rick-github commented on GitHub (Feb 26, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may show the reason for the release of VRAM. It could be the runner crashing due to https://github.com/ollama/ollama/issues/5975.
GiteaMirror commented 2026-04-28 23:53:45 -05:00
Author
Owner
Copy Link

@ice6 commented on GitHub (Feb 27, 2025):

@Andylau-BIT same problem. check your server logs:

maybe the root problem is : llama.cpp:11942: The current context does not support K-shift.

Feb 27 09:10:34 web ollama[195429]: llama.cpp:11942: The current context does not support K-shift
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xfe4c88)[0x555ef95b6c88]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xfe534d)[0x555ef95b734d]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf5fcb4)[0x555ef9531cb4]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf62b61)[0x555ef9534b61]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf63777)[0x555ef9535777]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xe8d3e2)[0x555ef945f3e2]
Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0x2f4381)[0x555ef88c6381]
Feb 27 09:10:46 web ollama[195429]: SIGABRT: abort
...

check ollama's https://github.com/ollama/ollama/issues/5975 and lamma.cpp's https://github.com/ggml-org/llama.cpp/issues/8862 for more detail.

<!-- gh-comment-id:2687086767 --> @ice6 commented on GitHub (Feb 27, 2025): @Andylau-BIT same problem. check your server logs: maybe the root problem is : `llama.cpp:11942: The current context does not support K-shift`. ``` Feb 27 09:10:34 web ollama[195429]: llama.cpp:11942: The current context does not support K-shift Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xfe4c88)[0x555ef95b6c88] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xfe534d)[0x555ef95b734d] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf5fcb4)[0x555ef9531cb4] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf62b61)[0x555ef9534b61] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xf63777)[0x555ef9535777] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0xe8d3e2)[0x555ef945f3e2] Feb 27 09:10:46 web ollama[195429]: /usr/bin/ollama(+0x2f4381)[0x555ef88c6381] Feb 27 09:10:46 web ollama[195429]: SIGABRT: abort ... ``` check `ollama`'s https://github.com/ollama/ollama/issues/5975 and `lamma.cpp`'s https://github.com/ggml-org/llama.cpp/issues/8862 for more detail.
GiteaMirror commented 2026-04-28 23:53:46 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Feb 27, 2025):

So, an effective strategy is to increase the num_ctx ? #5975 。I am really obvious that how to set num_ctx so that it both prevents the model from crashing and avoids overly long contexts?

<!-- gh-comment-id:2687101983 --> @Andylau-BIT commented on GitHub (Feb 27, 2025): So, an effective strategy is to increase the num_ctx ? [#5975](https://github.com/ollama/ollama/issues/5975) 。I am really obvious that how to set num_ctx so that it both prevents the model from crashing and avoids overly long contexts?
GiteaMirror commented 2026-04-28 23:53:46 -05:00
Author
Owner
Copy Link

@ice6 commented on GitHub (Feb 27, 2025):

@Andylau-BIT according ggmls doc: https://github.com/ggml-org/llama.cpp/tree/master/examples/server

maybe set predict=-2 which means: -2 = until context filled

I have not tried, you can try. My hardware is too poor... it is very slow to have a try.

<!-- gh-comment-id:2687187338 --> @ice6 commented on GitHub (Feb 27, 2025): @Andylau-BIT according `ggml`s doc: https://github.com/ggml-org/llama.cpp/tree/master/examples/server maybe set `predict=-2` which means: `-2 = until context filled` I have not tried, you can try. My hardware is too poor... it is very slow to have a try.
GiteaMirror commented 2026-04-28 23:53:46 -05:00
Author
Owner
Copy Link

@tisfeng commented on GitHub (Mar 29, 2025):

If you're using the API, use the keep_alive parameter with the /api/generate and /api/chat endpoints to set the amount of time that a model stays in memory.
https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately

Hello, currently, do only /api/generate and /api/chat support the keep_alive parameter? It seems the OpenAI endpoint doesn't support keep_alive.

<!-- gh-comment-id:2762996301 --> @tisfeng commented on GitHub (Mar 29, 2025): > If you're using the API, use the keep_alive parameter with the /api/generate and /api/chat endpoints to set the amount of time that a model stays in memory. https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately Hello, currently, do only `/api/generate` and `/api/chat` support the `keep_alive` parameter? It seems the OpenAI endpoint doesn't support `keep_alive`.
GiteaMirror commented 2026-04-28 23:53:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 29, 2025):

The OpenAI endpoint doesn't support keep_alive.

<!-- gh-comment-id:2763285256 --> @rick-github commented on GitHub (Mar 29, 2025): The OpenAI endpoint doesn't support `keep_alive`.
GiteaMirror commented 2026-04-28 23:53:47 -05:00
Author
Owner
Copy Link

@Andylau-BIT commented on GitHub (Apr 13, 2025):

您的邮件我已经收到！

<!-- gh-comment-id:2800015028 --> @Andylau-BIT commented on GitHub (Apr 13, 2025): 您的邮件我已经收到！
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#52620
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?