github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #3953] Support VLLM as a backend #64488

New Issue
Open
opened 2026-05-03 17:49:52 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-05-03 17:49:52 -05:00
Owner
Copy Link

Originally created by @kannon92 on GitHub (Apr 26, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3953

Hi,

I realize that this is a big ask but I am learning more and more about inferencing and I've heard that VLLM tends to have better performance for many GPU training.

OLLAMA is a great UX and I love the tight integration with llama.cpp. But it would be nice to start exploring how one could use OLLAMA models with vllm.

Originally created by @kannon92 on GitHub (Apr 26, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3953 Hi, I realize that this is a big ask but I am learning more and more about inferencing and I've heard that VLLM tends to have better performance for many GPU training. OLLAMA is a great UX and I love the tight integration with llama.cpp. But it would be nice to start exploring how one could use OLLAMA models with vllm.
GiteaMirror added the feature request label 2026-05-03 17:49:52 -05:00
GiteaMirror commented 2026-05-03 17:49:53 -05:00
Author
Owner
Copy Link

@JJassonn69 commented on GitHub (Apr 26, 2024):

it would be quite an interesting challenge, vllm supports concurrency natively so some things would surely be a plus.

<!-- gh-comment-id:2080043314 --> @JJassonn69 commented on GitHub (Apr 26, 2024): it would be quite an interesting challenge, vllm supports concurrency natively so some things would surely be a plus.
GiteaMirror commented 2026-05-03 17:49:54 -05:00
Author
Owner
Copy Link

@matbeedotcom commented on GitHub (May 31, 2024):

I feel like Ollama /needs/ to prove it can work with more than gguf models. Its proposed Modelfile method of defining how models are inferenced will actually be useful.

<!-- gh-comment-id:2142589960 --> @matbeedotcom commented on GitHub (May 31, 2024): I feel like Ollama /needs/ to prove it can work with more than gguf models. Its proposed Modelfile method of defining how models are inferenced will actually be useful.
GiteaMirror commented 2026-05-03 17:49:56 -05:00
Author
Owner
Copy Link

@angiopteris commented on GitHub (Sep 1, 2024):

Up the issue, I'm love to test other backend such as vllm to keep on improving perf

<!-- gh-comment-id:2323258751 --> @angiopteris commented on GitHub (Sep 1, 2024): Up the issue, I'm love to test other backend such as vllm to keep on improving perf
GiteaMirror commented 2026-05-03 17:49:58 -05:00
Author
Owner
Copy Link

@deiangi commented on GitHub (Dec 23, 2024):

I second this

<!-- gh-comment-id:2559822670 --> @deiangi commented on GitHub (Dec 23, 2024): I second this
GiteaMirror commented 2026-05-03 17:50:01 -05:00
Author
Owner
Copy Link

@ericcurtin commented on GitHub (Nov 24, 2025):

Implemented in Docker Model Runner:

https://github.com/docker/model-runner
https://www.docker.com/blog/docker-model-runner-integrates-vllm/

<!-- gh-comment-id:3570761057 --> @ericcurtin commented on GitHub (Nov 24, 2025): Implemented in Docker Model Runner: https://github.com/docker/model-runner https://www.docker.com/blog/docker-model-runner-integrates-vllm/
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#64488
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?