github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #1573] Enable prompt cache #47374

New Issue
Closed
opened 2026-04-28 03:38:38 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-28 03:38:38 -05:00
Owner
Copy Link

Originally created by @K0IN on GitHub (Dec 17, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1573

I use ollama in an automated way, that's why I use the same prompt all the time.

That's why I thought we might allow ollama to use prompt_cache.

f7f468a97d/common/common.cpp (L1508C22-L1508C38)

Or is there already a way to control this / does ollama cache multiple prompts anyway?

Originally created by @K0IN on GitHub (Dec 17, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1573 I use ollama in an automated way, that's why I use the same prompt all the time. That's why I thought we might allow ollama to use prompt_cache. https://github.com/ggerganov/llama.cpp/blob/f7f468a97dceec2f8fe8b1ed7a2091083446ebc7/common/common.cpp#L1508C22-L1508C38 Or is there already a way to control this / does ollama cache multiple prompts anyway?
GiteaMirror commented 2026-04-28 03:38:39 -05:00
Author
Owner
Copy Link

@TheDudeFromCI commented on GitHub (Dec 19, 2023):

I think this would be extremely useful, especially on slower devices. At current, it takes me around 5 minutes to evaluate the prompt before generating any text because of hardware limitations. Having to wait for the prompt to be re-evaluated after each newly generated line takes over 90% of the total time spent generating text.

If the cache could be perhaps returned like the context tokens are, or even just a returned code to let the model know to read the cache from a specific temp file would greatly improve overall performance.

<!-- gh-comment-id:1862797903 --> @TheDudeFromCI commented on GitHub (Dec 19, 2023): I think this would be extremely useful, especially on slower devices. At current, it takes me around 5 minutes to evaluate the prompt before generating any text because of hardware limitations. Having to wait for the prompt to be re-evaluated after each newly generated line takes over 90% of the total time spent generating text. If the cache could be perhaps returned like the `context` tokens are, or even just a returned code to let the model know to read the cache from a specific temp file would greatly improve overall performance.
GiteaMirror commented 2026-04-28 03:38:39 -05:00
Author
Owner
Copy Link

@djmaze commented on GitHub (Dec 19, 2023):

I imagine a memory-based, fixed size LRU cache which stores prompt evaluations on a session-by-session basis. The data of the least recently used session would be evicted first.

That said, I don't know the ollama internals. Maybe it does not even have the concept of a client session?

<!-- gh-comment-id:1863551758 --> @djmaze commented on GitHub (Dec 19, 2023): I imagine a memory-based, fixed size LRU cache which stores prompt evaluations on a session-by-session basis. The data of the least recently used session would be evicted first. That said, I don't know the ollama internals. Maybe it does not even have the concept of a client session?
GiteaMirror commented 2026-04-28 03:38:40 -05:00
Author
Owner
Copy Link

@AndreiSva commented on GitHub (Dec 20, 2023):

yes please!!

<!-- gh-comment-id:1864012172 --> @AndreiSva commented on GitHub (Dec 20, 2023): yes please!!
GiteaMirror commented 2026-04-28 03:38:40 -05:00
Author
Owner
Copy Link

@K0IN commented on GitHub (Dec 20, 2023):

The link in my issue might be wrong, as far as i can tell ollama uses the example/server from llama.cpp which has its own cache_prompt flag, see https://github.com/ggerganov/llama.cpp/tree/master/examples/server#api-endpoints.

Also see the comment on n_predict.

I hacked something together to control this flag from the ollama api, but i don't see any difference so far.

I like the idea of @TheDudeFromCI, it is nice to let the API be stateless.

<!-- gh-comment-id:1865050524 --> @K0IN commented on GitHub (Dec 20, 2023): The link in my issue might be wrong, as far as i can tell ollama uses the example/server from llama.cpp which has its own cache_prompt flag, see https://github.com/ggerganov/llama.cpp/tree/master/examples/server#api-endpoints. Also see the comment on n_predict. I hacked something together to control this flag from the ollama api, but i don't see any difference so far. I like the idea of @TheDudeFromCI, it is nice to let the API be stateless.
GiteaMirror commented 2026-04-28 03:38:41 -05:00
Author
Owner
Copy Link

@K0IN commented on GitHub (Dec 20, 2023):

This might be just right, see comments made in https://github.com/ggerganov/llama.cpp/issues/4329

<!-- gh-comment-id:1865088918 --> @K0IN commented on GitHub (Dec 20, 2023): This might be just right, see comments made in https://github.com/ggerganov/llama.cpp/issues/4329
GiteaMirror commented 2026-04-28 03:38:41 -05:00
Author
Owner
Copy Link

@K0IN commented on GitHub (Dec 20, 2023):

@djmaze feel free to test out my pr, and give some feedback :)

<!-- gh-comment-id:1865113995 --> @K0IN commented on GitHub (Dec 20, 2023): @djmaze feel free to test out my pr, and give some feedback :)
GiteaMirror commented 2026-04-28 03:38:42 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jan 25, 2024):

This should be fixed in https://github.com/ollama/ollama/pull/2190, but feel free to re-open @K0IN

<!-- gh-comment-id:1911051607 --> @jmorganca commented on GitHub (Jan 25, 2024): This should be fixed in https://github.com/ollama/ollama/pull/2190, but feel free to re-open @K0IN
GiteaMirror commented 2026-04-28 03:38:44 -05:00
Author
Owner
Copy Link

@samos123 commented on GitHub (Feb 13, 2025):

Edit: looks like it's enabled by default

How to enable prefix caching?

<!-- gh-comment-id:2655385084 --> @samos123 commented on GitHub (Feb 13, 2025): Edit: looks like it's enabled by default How to enable prefix caching?
GiteaMirror commented 2026-04-28 03:38:44 -05:00
Author
Owner
Copy Link

@sairajv commented on GitHub (Mar 3, 2025):

@jmorganca Does ollama have prefix caching capability? I have searched issues but cannot find a definitive answer

<!-- gh-comment-id:2694981245 --> @sairajv commented on GitHub (Mar 3, 2025): @jmorganca Does ollama have prefix caching capability? I have searched issues but cannot find a definitive answer
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#47374
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?