github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2396] llama.cpp now supports Vulkan #1394

New Issue
Closed
opened 2026-04-12 11:13:28 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 11:13:28 -05:00
Owner
Copy Link

Originally created by @ddpasa on GitHub (Feb 7, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2396

Originally assigned to: @dhiltgen on GitHub.

As of 10 days ago: 2307523d32

This is great news for people who non-CUDA cards.

What's necessary to support this with Ollama? I'm happy to help if you show me the pointers.

Originally created by @ddpasa on GitHub (Feb 7, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2396 Originally assigned to: @dhiltgen on GitHub. As of 10 days ago: https://github.com/ggerganov/llama.cpp/commit/2307523d322af762ae06648b29ec5a9eb1c73032 This is great news for people who non-CUDA cards. What's necessary to support this with Ollama? I'm happy to help if you show me the pointers.
GiteaMirror added the gpu label 2026-04-12 11:13:28 -05:00
GiteaMirror commented 2026-04-12 11:13:30 -05:00
Author
Owner
Copy Link

@ddpasa commented on GitHub (Feb 8, 2024):

I managed to compile ollama with the following code snippet gen_linux.sh and it builds a vulkan version:

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_VULKAN=1 -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_F16C=on -DCMAKE_BUILD_TYPE=Release -DLLAMA_SERVER_VERBOSE=on" go generate ./...
go build .

I'm now getting a very cryptic segfault. Debugging...

Edit: segfault fixed, I was forgetting to load libvulkan. Now it runs, but produces empty output. Continuing to debug...

Edit2: Phi-2 is running on Vulkan, but the outputs from the CPU version and the Vulkan version are different. Nice speedup though...

<!-- gh-comment-id:1933594219 --> @ddpasa commented on GitHub (Feb 8, 2024): I managed to compile ollama with the following code snippet gen_linux.sh and it builds a vulkan version: ``` OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_VULKAN=1 -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_F16C=on -DCMAKE_BUILD_TYPE=Release -DLLAMA_SERVER_VERBOSE=on" go generate ./... go build . ``` I'm now getting a very cryptic segfault. Debugging... Edit: segfault fixed, I was forgetting to load libvulkan. Now it runs, but produces empty output. Continuing to debug... Edit2: Phi-2 is running on Vulkan, but the outputs from the CPU version and the Vulkan version are different. Nice speedup though...
GiteaMirror commented 2026-04-12 11:13:31 -05:00
Author
Owner
Copy Link

@ddpasa commented on GitHub (Feb 9, 2024):

I was able to get llama.cpp compiled with the following, and confirm that it works. However, when I try to hack gen_commons.sh, I always get empty or grabled output. I'm not very familiar with how ollama builds llama.cpp, so I'm probably messing something up. Tagging @dhiltgen because he was kind enough to help me in the AVX thread.

working llama.cpp config:

mkdir build
cd build
cmake .. -DLLAMA_VULKAN=1
cmake --build . --config Release
# now test:
./build/bin/main -m ggml-model-q4_0.gguf -p "Hi you how are you" -n 50 -e -ngl 0 -t 4

ollama gen_commons.sh that compiles fine, but produces garbled output:

cmake -S ${LLAMACPP_DIR} -B ${BUILD_DIR} -DLLAMA_VULKAN=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=on -DLLAMA_SERVER_VERBOSE=on -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on 
cmake --build ${BUILD_DIR} ${CMAKE_TARGETS} -j8
mkdir -p ${BUILD_DIR}/lib/
g++ -fPIC -g -shared -o ${BUILD_DIR}/lib/libext_server.${LIB_EXT} \
        ${GCC_ARCH} \
        ${WHOLE_ARCHIVE} ${BUILD_DIR}/examples/server/libext_server.a ${NO_WHOLE_ARCHIVE} \
        ${BUILD_DIR}/common/libcommon.a \
        ${BUILD_DIR}/libllama.a \
        -Wl,-rpath,\$ORIGIN \
        -lpthread -ldl -lm -lvulkan \
        ${EXTRA_LIBS}
<!-- gh-comment-id:1936133022 --> @ddpasa commented on GitHub (Feb 9, 2024): I was able to get llama.cpp compiled with the following, and confirm that it works. However, when I try to hack [gen_commons.sh](https://github.com/ollama/ollama/blob/main/llm/generate/gen_common.sh#L85), I always get empty or grabled output. I'm not very familiar with how ollama builds llama.cpp, so I'm probably messing something up. Tagging @dhiltgen because he was kind enough to help me in the [AVX thread.](https://github.com/ollama/ollama/issues/2205) working llama.cpp config: ``` mkdir build cd build cmake .. -DLLAMA_VULKAN=1 cmake --build . --config Release # now test: ./build/bin/main -m ggml-model-q4_0.gguf -p "Hi you how are you" -n 50 -e -ngl 0 -t 4 ``` ollama gen_commons.sh that compiles fine, but produces garbled output: ``` cmake -S ${LLAMACPP_DIR} -B ${BUILD_DIR} -DLLAMA_VULKAN=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=on -DLLAMA_SERVER_VERBOSE=on -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on cmake --build ${BUILD_DIR} ${CMAKE_TARGETS} -j8 mkdir -p ${BUILD_DIR}/lib/ g++ -fPIC -g -shared -o ${BUILD_DIR}/lib/libext_server.${LIB_EXT} \ ${GCC_ARCH} \ ${WHOLE_ARCHIVE} ${BUILD_DIR}/examples/server/libext_server.a ${NO_WHOLE_ARCHIVE} \ ${BUILD_DIR}/common/libcommon.a \ ${BUILD_DIR}/libllama.a \ -Wl,-rpath,\$ORIGIN \ -lpthread -ldl -lm -lvulkan \ ${EXTRA_LIBS} ```
GiteaMirror commented 2026-04-12 11:13:31 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 21, 2024):

We have multiple issues tracking Vulkan support - lets use #2033

<!-- gh-comment-id:2012381264 --> @dhiltgen commented on GitHub (Mar 21, 2024): We have multiple issues tracking Vulkan support - lets use #2033
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label gpu
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1394
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?