github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2758] Switching back and forth between models will gradually reduce the available GPU memory. #1662

New Issue
Closed
opened 2026-04-12 11:38:06 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 11:38:06 -05:00
Owner
Copy Link

Originally created by @mofanke on GitHub (Feb 26, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2758

Originally assigned to: @dhiltgen on GitHub.

Operating System: Windows
GPU: NVIDIA with 6GB memory

Description:
While switching between Mistral 7B and Codellama 7B, I noticed a decrease in GPU available memory for layers offloaded to the GPU. Upon investigation, I captured the following debug log:

time=2024-02-26T11:18:53.800+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 3302M available memory"
time=2024-02-26T11:20:07.502+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2917M available memory"
time=2024-02-26T11:21:27.453+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2916M available memory"
time=2024-02-26T11:22:45.617+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2481M available memory"
time=2024-02-26T11:23:22.500+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2481M available memory"

This log indicates a consistent decrease in available GPU memory. Upon restarting the Ollama server, the available GPU memory returned to 3199MB.

Originally created by @mofanke on GitHub (Feb 26, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2758 Originally assigned to: @dhiltgen on GitHub. Operating System: Windows GPU: NVIDIA with 6GB memory Description: While switching between Mistral 7B and Codellama 7B, I noticed a decrease in GPU available memory for layers offloaded to the GPU. Upon investigation, I captured the following debug log: ```plaintext time=2024-02-26T11:18:53.800+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 3302M available memory" time=2024-02-26T11:20:07.502+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2917M available memory" time=2024-02-26T11:21:27.453+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2916M available memory" time=2024-02-26T11:22:45.617+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2481M available memory" time=2024-02-26T11:23:22.500+08:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2481M available memory" ``` This log indicates a consistent decrease in available GPU memory. Upon restarting the Ollama server, the available GPU memory returned to 3199MB.
GiteaMirror added the bug label 2026-04-12 11:38:06 -05:00
GiteaMirror commented 2026-04-12 11:38:07 -05:00
Author
Owner
Copy Link

@mofanke commented on GitHub (Feb 26, 2024):

disable cache_prompt works

<!-- gh-comment-id:1963664300 --> @mofanke commented on GitHub (Feb 26, 2024): disable cache_prompt works
GiteaMirror commented 2026-04-12 11:38:08 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Feb 27, 2024):

The KV cache code has been undergoing change upstream. By toggling back and forth between Mistral 7B and Codellama 7B I was able to get the system to wedge after only a few iterations, but after bumping llama.cpp I can get many hundreds of iterations without failure.

<!-- gh-comment-id:1965596318 --> @dhiltgen commented on GitHub (Feb 27, 2024): The KV cache code has been undergoing change upstream. By toggling back and forth between Mistral 7B and Codellama 7B I was able to get the system to wedge after only a few iterations, but after bumping llama.cpp I can get many hundreds of iterations without failure.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1662
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?