github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1237] GPTQ / ExLlamaV2 (EXL2) quantisation #629

New Issue
Open
opened 2026-04-12 10:19:51 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 10:19:51 -05:00
Owner
Copy Link

Originally created by @0xdevalias on GitHub (Nov 22, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1237

Feature Description

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do as an enhancement.

Motivation

It sounds like it's a fast/useful quantisation method:

See Also

Originally created by @0xdevalias on GitHub (Nov 22, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1237 # Feature Description Please provide a detailed written description of what you were trying to do, and what you expected `llama.cpp` to do as an enhancement. # Motivation It sounds like it's a fast/useful quantisation method: - https://towardsdatascience.com/exllamav2-the-fastest-library-to-run-llms-32aeda294d26 - https://github.com/mlabonne/llm-course/blob/main/Quantize_models_with_ExLlamaV2.ipynb - https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34 - https://huggingface.co/blog/gptq-integration - https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ - > A detailed comparison between GPTQ, AWQ, EXL2, q4_K_M, q4_K_S, and load_in_4bit: perplexity, VRAM, speed, model size, and loading time. ## See Also - https://github.com/ggerganov/llama.cpp/issues/4165
GiteaMirror added the feature request label 2026-04-12 10:19:51 -05:00
GiteaMirror commented 2026-04-12 10:19:51 -05:00
Author
Owner
Copy Link

@wellmorq commented on GitHub (May 7, 2024):

Bump

<!-- gh-comment-id:2099012041 --> @wellmorq commented on GitHub (May 7, 2024): Bump
GiteaMirror commented 2026-04-12 10:19:51 -05:00
Author
Owner
Copy Link

@Greatz08 commented on GitHub (Jun 6, 2024):

no support for gptq and awq?

<!-- gh-comment-id:2152080403 --> @Greatz08 commented on GitHub (Jun 6, 2024): no support for gptq and awq?
GiteaMirror commented 2026-04-12 10:19:52 -05:00
Author
Owner
Copy Link

@fzyzcjy commented on GitHub (Aug 11, 2024):

Hi, is there any updates? Thanks!

<!-- gh-comment-id:2282749008 --> @fzyzcjy commented on GitHub (Aug 11, 2024): Hi, is there any updates? Thanks!
GiteaMirror commented 2026-04-12 10:19:53 -05:00
Author
Owner
Copy Link

@Readon commented on GitHub (Aug 13, 2024):

According to the diagram, it seems that exllama v2 is much better at generating speed.

<!-- gh-comment-id:2286535468 --> @Readon commented on GitHub (Aug 13, 2024): According to the [diagram](https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/), it seems that exllama v2 is much better at generating speed.
GiteaMirror commented 2026-04-12 10:19:53 -05:00
Author
Owner
Copy Link

@0xdevalias commented on GitHub (Aug 15, 2024):

According to the diagram, it seems that exllama v2 is much better at generating speed.

Also from that link:

Update 2: Gerganov has created a PR on llama.cpp that optimizes the llama.cpp evaluation/processing speeds and should make the values here obsolete. See the numbers and discussion here.

<!-- gh-comment-id:2290298215 --> @0xdevalias commented on GitHub (Aug 15, 2024): > According to the [diagram](https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/), it seems that exllama v2 is much better at generating speed. Also from that link: > Update 2: Gerganov has created a PR on llama.cpp that optimizes the llama.cpp evaluation/processing speeds **and should make the values here obsolete**. See the numbers and discussion here. - https://github.com/ggerganov/llama.cpp/pull/3776#issuecomment-1781472687
GiteaMirror commented 2026-04-12 10:19:53 -05:00
Author
Owner
Copy Link

@Readon commented on GitHub (Aug 15, 2024):

According to the diagram, it seems that exllama v2 is much better at generating speed.

Also from that link:

Update 2: Gerganov has created a PR on llama.cpp that optimizes the llama.cpp evaluation/processing speeds and should make the values here obsolete. See the numbers and discussion here.

Nice, this shows the improvement closer to exllama v2.

<!-- gh-comment-id:2291371661 --> @Readon commented on GitHub (Aug 15, 2024): > > According to the [diagram](https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/), it seems that exllama v2 is much better at generating speed. > > Also from that link: > > > Update 2: Gerganov has created a PR on llama.cpp that optimizes the llama.cpp evaluation/processing speeds **and should make the values here obsolete**. See the numbers and discussion here. > > * [cuda : improve text-generation and batched decoding performance ggerganov/llama.cpp#3776 (comment)](https://github.com/ggerganov/llama.cpp/pull/3776#issuecomment-1781472687) Nice, this shows the improvement closer to exllama v2.
GiteaMirror commented 2026-04-12 10:19:53 -05:00
Author
Owner
Copy Link

@gilbrotheraway commented on GitHub (Mar 31, 2025):

on my old ass p100 pascals gptq is about 20% faster than exl2, which in turn is way faster than gguf, the only reason we don't see many quants is because it takes more resources to quantize gptq

<!-- gh-comment-id:2766825462 --> @gilbrotheraway commented on GitHub (Mar 31, 2025): on my old ass p100 pascals gptq is about 20% faster than exl2, which in turn is way faster than gguf, the only reason we don't see many quants is because it takes more resources to quantize gptq
GiteaMirror commented 2026-04-12 10:19:54 -05:00
Author
Owner
Copy Link

@gilbrotheraway commented on GitHub (Mar 31, 2025):

btw this has been updated

https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/

gptq 20% faster accross the board confirmed

<!-- gh-comment-id:2766844834 --> @gilbrotheraway commented on GitHub (Mar 31, 2025): btw this has been updated https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ gptq 20% faster accross the board confirmed
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#629
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?