github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3114] Using INT4 Quantization to Save VRAM with ollama #27674

New Issue
Closed
opened 2026-04-22 05:12:31 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 05:12:31 -05:00
Owner
Copy Link

Originally created by @TraceRecursion on GitHub (Mar 13, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3114

Hello ollama team,

I am currently exploring the use of ollama to run models and am interested in implementing INT4 quantization to save on VRAM usage. I have read through the documentation but would appreciate some guidance on how to properly apply INT4 quantization during the model run.

Could you provide some insights or examples on how to integrate this quantization method with ollama? Specifically, I am looking for ways to reduce the VRAM footprint while maintaining performance.

Thank you for your assistance.

Best regards,
溯源递归

Originally created by @TraceRecursion on GitHub (Mar 13, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3114 Hello ollama team, I am currently exploring the use of ollama to run models and am interested in implementing INT4 quantization to save on VRAM usage. I have read through the documentation but would appreciate some guidance on how to properly apply INT4 quantization during the model run. Could you provide some insights or examples on how to integrate this quantization method with ollama? Specifically, I am looking for ways to reduce the VRAM footprint while maintaining performance. Thank you for your assistance. Best regards, 溯源递归
GiteaMirror commented 2026-04-22 05:12:33 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 13, 2024):

Why not just import a model that has already been quantised? e.g. in GGUF format. I think quantising the model on the fly (particularly for larger models) would take an enormous amount of time. Most of the models in the model library are already quantised to 4.xx bpw, which is equivalent to INT4.

<!-- gh-comment-id:1995893644 --> @orlyandico commented on GitHub (Mar 13, 2024): Why not just import a model that has already been quantised? e.g. in GGUF format. I think quantising the model on the fly (particularly for larger models) would take an enormous amount of time. Most of the models in the model library are already quantised to 4.xx bpw, which is equivalent to INT4.
GiteaMirror commented 2026-04-22 05:12:34 -05:00
Author
Owner
Copy Link

@easp commented on GitHub (Mar 14, 2024):

New quantizations would need to be implemented upstream in llama.cpp. There are already a variety of quantization approaches implemented there and the models in the ollama library are into a range of sizes using a few of those approaches. If you pull an ollama model without specifying the quantization it will pull a q4_0 version.

<!-- gh-comment-id:1996238709 --> @easp commented on GitHub (Mar 14, 2024): New quantizations would need to be implemented upstream in llama.cpp. There are already a variety of quantization approaches implemented there and the models in the ollama library are into a range of sizes using a few of those approaches. If you pull an ollama model without specifying the quantization it will pull a q4_0 version.
GiteaMirror commented 2026-04-22 05:12:35 -05:00
Author
Owner
Copy Link

@TraceRecursion commented on GitHub (Mar 14, 2024):

appreciate your help

<!-- gh-comment-id:1997673036 --> @TraceRecursion commented on GitHub (Mar 14, 2024): appreciate your help
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#27674
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?