github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8569] Linux: Compiling Ollama with AVX-512 and CUDA support #5534

New Issue
Closed
opened 2026-04-12 16:47:15 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-04-12 16:47:15 -05:00
Owner
Copy Link

Originally created by @graynoir on GitHub (Jan 24, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8569

Hi, I've been trying to compile Ollama with AVX-512 and Cuda support on Linux (Manjaro), however, despite multiple attempts and different custom cpu flags I don't seem to get it to work, and instead ollama serve falls back to level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu], when the binary package at least had runners="[cpu cpu_avx cpu_avx2 cuda_v12_avx], however no AVX-512. I don't know much about Go, I've read the development.md but all I was able to get from ollama serve after setting the custom flags was:

system info="CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 =1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8

after adding the following flags to the make command to build ollama:

make CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 -j16

But it seems neither AVX is working nor is the GPU being used, as the inference speed is much slower than that of the pre-compiled ollama binary.

Has anybody a successful build of Ollama with Cuda and AVX-512, and knows how to achieve that? Thanks in advance!
Additional information:

  • ollama version: 0.5.7-2
  • cuda version: 12.2
  • Nvidia driver version: 535.216.01
  • cpu: Intel Xeon W-11955m
  • gpu: RTX A5000
Originally created by @graynoir on GitHub (Jan 24, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8569 Hi, I've been trying to compile Ollama with AVX-512 and Cuda support on Linux (Manjaro), however, despite multiple attempts and different custom cpu flags I don't seem to get it to work, and instead `ollama serve` falls back to `level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]`, when the binary package at least had `runners="[cpu cpu_avx cpu_avx2 cuda_v12_avx]`, however no AVX-512. I don't know much about Go, I've read the `development.md` but all I was able to get from `ollama serve` after setting the custom flags was: ``` system info="CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 =1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 ``` after adding the following flags to the `make` command to build ollama: ``` make CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 -j16 ``` But it seems neither AVX is working nor is the GPU being used, as the inference speed is much slower than that of the pre-compiled ollama binary. Has anybody a successful build of Ollama with Cuda and AVX-512, and knows how to achieve that? Thanks in advance! Additional information: * ollama version: 0.5.7-2 * cuda version: 12.2 * Nvidia driver version: 535.216.01 * cpu: Intel Xeon W-11955m * gpu: RTX A5000
GiteaMirror commented 2026-04-12 16:47:17 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 24, 2025):

The makefile is supposed to detect a CUDA install and build the appropriate runners, but it can be hit and miss depending on where everything is installed. If you have docker installed it's much easier to use the dockerized build since all the dependencies are in docker images.

OLLAMA_SKIP_ROCM_GENERATE=1 \
  CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 \ 
  PLATFORM=linux/amd64 \
  ./scripts/build_linux.sh
<!-- gh-comment-id:2613540858 --> @rick-github commented on GitHub (Jan 24, 2025): The makefile is supposed to detect a CUDA install and build the appropriate runners, but it can be hit and miss depending on where everything is installed. If you have docker installed it's much easier to use the dockerized build since all the dependencies are in docker images. ```sh OLLAMA_SKIP_ROCM_GENERATE=1 \ CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 \ PLATFORM=linux/amd64 \ ./scripts/build_linux.sh ```
GiteaMirror commented 2026-04-12 16:47:19 -05:00
Author
Owner
Copy Link

@graynoir commented on GitHub (Jan 25, 2025):

Hi Rick, thank you for the suggestion, I tried building the dockerized build using the build_linux.sh script and the command you provided. With this, ollama serve shows:

system info="CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1| F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI= 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 |AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8

still I'm getting only

msg="Dynamic LLM libraries" runners=[cpu]

and the Nvidia gpu isn't being used either, as nvidia-smi and btop report no usage.

<!-- gh-comment-id:2614124057 --> @graynoir commented on GitHub (Jan 25, 2025): Hi Rick, thank you for the suggestion, I tried building the dockerized build using the `build_linux.sh` script and the command you provided. With this, `ollama serve` shows: ``` system info="CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1| F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI= 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 |AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 ``` still I'm getting only ``` msg="Dynamic LLM libraries" runners=[cpu] ``` and the Nvidia gpu isn't being used either, as `nvidia-smi` and `btop` report no usage.
GiteaMirror commented 2026-04-12 16:47:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 25, 2025):

The dockerized build created a tar.gz file in dist which should have contained the binary, runners and libraries. How did you install the programs in it?

<!-- gh-comment-id:2614125140 --> @rick-github commented on GitHub (Jan 25, 2025): The dockerized build created a tar.gz file in `dist` which should have contained the binary, runners and libraries. How did you install the programs in it?
GiteaMirror commented 2026-04-12 16:47:21 -05:00
Author
Owner
Copy Link

@graynoir commented on GitHub (Jan 25, 2025):

Yes, exactly, I extracted the tar.gz archive and ran ./ollama serve. There was only one ollama binary inside the archive, no other file.

<!-- gh-comment-id:2614126621 --> @graynoir commented on GitHub (Jan 25, 2025): Yes, exactly, I extracted the tar.gz archive and ran `./ollama serve`. There was only one `ollama` binary inside the archive, no other file.
GiteaMirror commented 2026-04-12 16:47:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 25, 2025):

Hmm, the command I gave above created a tar.gz with the following contents on my system:

$ tar ztf dist/ollama-linux-amd64.tgz 
./
./lib/
./lib/ollama/
./lib/ollama/libcublas.so.12.4.5.8
./lib/ollama/runners/
./lib/ollama/runners/cuda_v12/
./lib/ollama/runners/cuda_v12/ollama_llama_server
./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so
./lib/ollama/libcublasLt.so.12.4.5.8
./lib/ollama/libcublasLt.so.12
./lib/ollama/libcudart.so.12.4.127
./lib/ollama/libcudart.so.12
./lib/ollama/libcublas.so.12
./bin/
./bin/ollama

Were there any errors during your build? Have you tried a dockerized build before? Perhaps the last build re-used previously compiled objects in the build cache.

<!-- gh-comment-id:2614130511 --> @rick-github commented on GitHub (Jan 25, 2025): Hmm, the command I gave above created a tar.gz with the following contents on my system: ``` $ tar ztf dist/ollama-linux-amd64.tgz ./ ./lib/ ./lib/ollama/ ./lib/ollama/libcublas.so.12.4.5.8 ./lib/ollama/runners/ ./lib/ollama/runners/cuda_v12/ ./lib/ollama/runners/cuda_v12/ollama_llama_server ./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so ./lib/ollama/libcublasLt.so.12.4.5.8 ./lib/ollama/libcublasLt.so.12 ./lib/ollama/libcudart.so.12.4.127 ./lib/ollama/libcudart.so.12 ./lib/ollama/libcublas.so.12 ./bin/ ./bin/ollama ``` Were there any errors during your build? Have you tried a dockerized build before? Perhaps the last build re-used previously compiled objects in the build cache.
GiteaMirror commented 2026-04-12 16:47:22 -05:00
Author
Owner
Copy Link

@graynoir commented on GitHub (Jan 25, 2025):

Hmm, interesting, there weren't any errors during the build process as far as I could tell. To be honest, I haven't used docker much prior, the entire output of the console for the build process was the following

$ OLLAMA_SKIP_ROCM_GENERATE=1 \
  CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 \
  PLATFORM=linux/amd64 \
  ./scripts/build_linux.sh
rm -rf ./llama/build/linux-amd64 ./dist/linux-amd64/lib/ollama ./ollama ./dist/linux-amd64/bin/ollama
go clean -cache
Building Ollama
VERSION=0.5.7-4-gca2f984-dirty
PLATFORM=linux/amd64
[+] Building 396.2s (15/15) FINISHED               docker:default
 => [internal] load build definition from Dockerfile         0.1s
 => => transferring dockerfile: 8.53kB                       0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform   0.1s
 => [internal] load metadata for docker.io/rocm/dev-centos-  0.9s
 => [internal] load .dockerignore                            0.2s
 => => transferring context: 105B                            0.0s
 => [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-cen  0.0s
 => [internal] load build context                            0.3s
 => => transferring context: 43.10MB                         0.1s
 => CACHED [unified-builder-amd64 2/5] COPY ./scripts/rh_li  0.0s
 => CACHED [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1  0.0s
 => [unified-builder-amd64 4/5] RUN yum-config-manager --  289.0s
 => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/  0.3s
 => [build-amd64 1/4] COPY . .                               0.5s
 => [build-amd64 2/4] RUN --mount=type=cache,target=/root  101.2s
 => [build-amd64 3/4] RUN cd dist/linux-amd64 &&     tar -c  1.0s
 => [build-amd64 4/4] RUN if [ -z 1 ] ; then     cd dist/li  0.6s
 => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github  0.2s
 => exporting to client directory                            0.0s
 => => copying files 10.61MB

For the setup I followed ollama/docker.md | GitHub.

Extracting the archive yields only

$ tar -ztf ./dist/ollama-linux-amd64.tgz
./
./bin/
./bin/ollama
<!-- gh-comment-id:2614133138 --> @graynoir commented on GitHub (Jan 25, 2025): Hmm, interesting, there weren't any errors during the build process as far as I could tell. To be honest, I haven't used docker much prior, the entire output of the console for the build process was the following ```sh $ OLLAMA_SKIP_ROCM_GENERATE=1 \ CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 \ PLATFORM=linux/amd64 \ ./scripts/build_linux.sh rm -rf ./llama/build/linux-amd64 ./dist/linux-amd64/lib/ollama ./ollama ./dist/linux-amd64/bin/ollama go clean -cache Building Ollama VERSION=0.5.7-4-gca2f984-dirty PLATFORM=linux/amd64 [+] Building 396.2s (15/15) FINISHED docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 8.53kB 0.0s => WARN: FromPlatformFlagConstDisallowed: FROM --platform 0.1s => [internal] load metadata for docker.io/rocm/dev-centos- 0.9s => [internal] load .dockerignore 0.2s => => transferring context: 105B 0.0s => [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-cen 0.0s => [internal] load build context 0.3s => => transferring context: 43.10MB 0.1s => CACHED [unified-builder-amd64 2/5] COPY ./scripts/rh_li 0.0s => CACHED [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1 0.0s => [unified-builder-amd64 4/5] RUN yum-config-manager -- 289.0s => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/ 0.3s => [build-amd64 1/4] COPY . . 0.5s => [build-amd64 2/4] RUN --mount=type=cache,target=/root 101.2s => [build-amd64 3/4] RUN cd dist/linux-amd64 && tar -c 1.0s => [build-amd64 4/4] RUN if [ -z 1 ] ; then cd dist/li 0.6s => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github 0.2s => exporting to client directory 0.0s => => copying files 10.61MB ``` For the setup I followed [ollama/docker.md | GitHub](https://github.com/ollama/ollama/blob/main/docs/docker.md). Extracting the archive yields only ```sh $ tar -ztf ./dist/ollama-linux-amd64.tgz ./ ./bin/ ./bin/ollama ```
GiteaMirror commented 2026-04-12 16:47:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

I cleaned the docker build cache, cloned the repo and did a build in a clean environment:

$ docker system prune
Total reclaimed space: 343.8GB
$ git clone https://github.com/ollama/ollama.git
$ cd ollama
$ env - PATH=/usr/bin OLLAMA_SKIP_ROCM_GENERATE=1 CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 PLATFORM=linux/amd64 ./scripts/build_linux.sh
Building Ollama
VERSION=0.5.7-5-g453e4d0
PLATFORM=linux/amd64
[+] Building 1371.6s (15/15) FINISHED                                                                                                                                                               docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s
 => => transferring dockerfile: 8.53kB                                                                                                                                                                        0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 176)                                                                                        0.0s
 => [internal] load metadata for docker.io/rocm/dev-centos-7:6.1.2-complete                                                                                                                                   1.2s
 => [internal] load .dockerignore                                                                                                                                                                             0.0s
 => => transferring context: 105B                                                                                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                             0.2s
 => => transferring context: 43.21MB                                                                                                                                                                          0.2s
 => CACHED [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293                                                0.0s
 => [unified-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh /                                                                                                                                             0.0s
 => [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1.22.8 sh /rh_linux_deps.sh                                                                                                                               55.7s
 => [unified-builder-amd64 4/5] RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo &&     dnf clean all &&     dnf install -y     zsh  249.3s 
 => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/                                                                                                                                     0.0s 
 => [build-amd64 1/4] COPY . .                                                                                                                                                                                0.1s 
 => [build-amd64 2/4] RUN --mount=type=cache,target=/root/.ccache     if grep "^flags" /proc/cpuinfo|grep avx>/dev/null; then         make -j $(nproc) dist ;     else         make -j 5 dist ;     fi     1047.1s 
 => [build-amd64 3/4] RUN cd dist/linux-amd64 &&     tar -cf - . | pigz --best > ../ollama-linux-amd64.tgz                                                                                                   14.3s 
 => [build-amd64 4/4] RUN if [ -z 1 ] ; then     cd dist/linux-amd64-rocm &&     tar -cf - . | pigz --best > ../ollama-linux-amd64-rocm.tgz ;    fi                                                           0.1s 
 => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github.com/ollama/ollama/dist/ollama-linux-*.tgz /                                                                                                       0.6s 
 => exporting to client directory                                                                                                                                                                             1.0s 
 => => copying files 879.80MB                                                                                                                                                                                 1.0s 
$ tar ztf dist/ollama-linux-amd64.tgz 
./
./lib/
./lib/ollama/
./lib/ollama/libcublas.so.12.4.5.8
./lib/ollama/runners/
./lib/ollama/runners/cuda_v12/
./lib/ollama/runners/cuda_v12/ollama_llama_server
./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so
./lib/ollama/libcublasLt.so.12.4.5.8
./lib/ollama/libcublasLt.so.12
./lib/ollama/libcudart.so.12.4.127
./lib/ollama/libcudart.so.12
./lib/ollama/libcublas.so.12
./bin/
./bin/ollama
<!-- gh-comment-id:2614159809 --> @rick-github commented on GitHub (Jan 26, 2025): I cleaned the docker build cache, cloned the repo and did a build in a clean environment: ```console $ docker system prune Total reclaimed space: 343.8GB $ git clone https://github.com/ollama/ollama.git $ cd ollama $ env - PATH=/usr/bin OLLAMA_SKIP_ROCM_GENERATE=1 CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 PLATFORM=linux/amd64 ./scripts/build_linux.sh Building Ollama VERSION=0.5.7-5-g453e4d0 PLATFORM=linux/amd64 [+] Building 1371.6s (15/15) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 8.53kB 0.0s => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 176) 0.0s => [internal] load metadata for docker.io/rocm/dev-centos-7:6.1.2-complete 1.2s => [internal] load .dockerignore 0.0s => => transferring context: 105B 0.0s => [internal] load build context 0.2s => => transferring context: 43.21MB 0.2s => CACHED [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1b0a386fe06d3d9d5e19293 0.0s => [unified-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh / 0.0s => [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1.22.8 sh /rh_linux_deps.sh 55.7s => [unified-builder-amd64 4/5] RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo && dnf clean all && dnf install -y zsh 249.3s => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/ 0.0s => [build-amd64 1/4] COPY . . 0.1s => [build-amd64 2/4] RUN --mount=type=cache,target=/root/.ccache if grep "^flags" /proc/cpuinfo|grep avx>/dev/null; then make -j $(nproc) dist ; else make -j 5 dist ; fi 1047.1s => [build-amd64 3/4] RUN cd dist/linux-amd64 && tar -cf - . | pigz --best > ../ollama-linux-amd64.tgz 14.3s => [build-amd64 4/4] RUN if [ -z 1 ] ; then cd dist/linux-amd64-rocm && tar -cf - . | pigz --best > ../ollama-linux-amd64-rocm.tgz ; fi 0.1s => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github.com/ollama/ollama/dist/ollama-linux-*.tgz / 0.6s => exporting to client directory 1.0s => => copying files 879.80MB 1.0s $ tar ztf dist/ollama-linux-amd64.tgz ./ ./lib/ ./lib/ollama/ ./lib/ollama/libcublas.so.12.4.5.8 ./lib/ollama/runners/ ./lib/ollama/runners/cuda_v12/ ./lib/ollama/runners/cuda_v12/ollama_llama_server ./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so ./lib/ollama/libcublasLt.so.12.4.5.8 ./lib/ollama/libcublasLt.so.12 ./lib/ollama/libcudart.so.12.4.127 ./lib/ollama/libcudart.so.12 ./lib/ollama/libcublas.so.12 ./bin/ ./bin/ollama ```
GiteaMirror commented 2026-04-12 16:47:24 -05:00
Author
Owner
Copy Link

@graynoir commented on GitHub (Jan 26, 2025):

Thank you for your effort, so I tried again, and this time I got:

$ env - PATH=/usr/bin OLLAMA_SKIP_ROCM_GENERATE=1 CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 PLATFORM=linux/amd64 ./scripts/build_linux.sh
Building Ollama
VERSION=0.5.7-5-g453e4d0
PLATFORM=linux/amd64
[+] Building 12549.0s (15/15) FINISHED                                                        docker:default
 => [internal] load build definition from Dockerfile
 0.1s
 => => transferring dockerfile: 8.53kB
 0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 176) 0.1s
 => [internal] load metadata for docker.io/rocm/dev-centos-7:6.1.2-complete 1.2s
 => [internal] load .dockerignore
 0.1s
 => => transferring context: 105B
 0.0s
 => [internal] load build context
 0.1s
 => => transferring context: 40.89kB
 0.0s
 => [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1 0.0s
 => CACHED [unified-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh / 0.0s
 => CACHED [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1.22.8sh /rh_linux_deps.sh 0.0s
 => [unified-builder-amd64 4/5] RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rh  11150.7s
 => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/ 0.3s
 => [build-amd64 1/4] COPY . .
 0.4s
 => [build-amd64 2/4] RUN --mount=type=cache,target=/root/.ccache     if grep "^flags" /proc/cpuinfo|grep avx>/dev/null; then   1372.2s
 => [build-amd64 3/4] RUN cd dist/linux-amd64 &&     tar -cf - .| pigz --best > ../ollama-linux-amd64.tgz19.8s
 => [build-amd64 4/4] RUN if [ -z 1 ] ; then     cd dist/linux-amd64-rocm &&     tar -cf - . | pigz --best > ../ollama-linux-amd6 0.5s
 => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github.com/ollama/ollama/dist/ollama-linux-*.tgz / 0.6s
 => exporting to client directory
 0.7s
 => => copying files 879.83MB

Resulting in

$ tar -ztf ./dist/ollama-linux-amd64.tgz
./
./bin/
./bin/ollama
./lib/
./lib/ollama/
./lib/ollama/libcublas.so.12
./lib/ollama/libcublas.so.12.4.5.8
./lib/ollama/libcublasLt.so.12
./lib/ollama/libcublasLt.so.12.4.5.8
./lib/ollama/libcudart.so.12
./lib/ollama/libcudart.so.12.4.127
./lib/ollama/runners/
./lib/ollama/runners/cuda_v12/
./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so
./lib/ollama/runners/cuda_v12/ollama_llama_server

CUDA seems to be working, as I was able to offload 37/65 layers from deepkseek-r1-distill-qwen-32b to the GPU:

llm_load_tensors: offloaded 37/65 layers to GPU
...
total duration:       7m17.628188199s
load duration:        7.422162817s
prompt eval count:    2162 token(s)
prompt eval duration: 9.904s
prompt eval rate:     218.30 tokens/s
eval count:           1231 token(s)
eval duration:        7m0.299s
eval rate:            2.93 tokens/s

However, it seems that AVX-512 (or even AVX-2 in this case?) still is not working, as ollama serve reports:

"Dynamic LLM libraries" runners="[cpu cuda_v12]"
system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512= 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8

Compared to the pre-compiled binary from the Manjaro repo:

"Dynamic LLM libraries" runners="[cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx]"
<!-- gh-comment-id:2614492938 --> @graynoir commented on GitHub (Jan 26, 2025): Thank you for your effort, so I tried again, and this time I got: ```sh $ env - PATH=/usr/bin OLLAMA_SKIP_ROCM_GENERATE=1 CUSTOM_CPU_FLAGS=avx,avx2,avx512,avx512vbmi,avx512vnni,avx512bf16 PLATFORM=linux/amd64 ./scripts/build_linux.sh Building Ollama VERSION=0.5.7-5-g453e4d0 PLATFORM=linux/amd64 [+] Building 12549.0s (15/15) FINISHED docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 8.53kB 0.0s => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 176) 0.1s => [internal] load metadata for docker.io/rocm/dev-centos-7:6.1.2-complete 1.2s => [internal] load .dockerignore 0.1s => => transferring context: 105B 0.0s => [internal] load build context 0.1s => => transferring context: 40.89kB 0.0s => [unified-builder-amd64 1/5] FROM docker.io/rocm/dev-centos-7:6.1.2-complete@sha256:d32024d55afc71bd8e9b9eb20835a369b9df69c1f1 0.0s => CACHED [unified-builder-amd64 2/5] COPY ./scripts/rh_linux_deps.sh / 0.0s => CACHED [unified-builder-amd64 3/5] RUN GOLANG_VERSION=1.22.8sh /rh_linux_deps.sh 0.0s => [unified-builder-amd64 4/5] RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rh 11150.7s => [unified-builder-amd64 5/5] WORKDIR /go/src/github.com/ollama/ollama/ 0.3s => [build-amd64 1/4] COPY . . 0.4s => [build-amd64 2/4] RUN --mount=type=cache,target=/root/.ccache if grep "^flags" /proc/cpuinfo|grep avx>/dev/null; then 1372.2s => [build-amd64 3/4] RUN cd dist/linux-amd64 && tar -cf - .| pigz --best > ../ollama-linux-amd64.tgz19.8s => [build-amd64 4/4] RUN if [ -z 1 ] ; then cd dist/linux-amd64-rocm && tar -cf - . | pigz --best > ../ollama-linux-amd6 0.5s => [dist-amd64 1/1] COPY --from=build-amd64 /go/src/github.com/ollama/ollama/dist/ollama-linux-*.tgz / 0.6s => exporting to client directory 0.7s => => copying files 879.83MB ``` Resulting in ```sh $ tar -ztf ./dist/ollama-linux-amd64.tgz ./ ./bin/ ./bin/ollama ./lib/ ./lib/ollama/ ./lib/ollama/libcublas.so.12 ./lib/ollama/libcublas.so.12.4.5.8 ./lib/ollama/libcublasLt.so.12 ./lib/ollama/libcublasLt.so.12.4.5.8 ./lib/ollama/libcudart.so.12 ./lib/ollama/libcudart.so.12.4.127 ./lib/ollama/runners/ ./lib/ollama/runners/cuda_v12/ ./lib/ollama/runners/cuda_v12/libggml_cuda_v12.so ./lib/ollama/runners/cuda_v12/ollama_llama_server ``` CUDA seems to be working, as I was able to offload 37/65 layers from deepkseek-r1-distill-qwen-32b to the GPU: ``` llm_load_tensors: offloaded 37/65 layers to GPU ... total duration: 7m17.628188199s load duration: 7.422162817s prompt eval count: 2162 token(s) prompt eval duration: 9.904s prompt eval rate: 218.30 tokens/s eval count: 1231 token(s) eval duration: 7m0.299s eval rate: 2.93 tokens/s ``` However, it seems that AVX-512 (or even AVX-2 in this case?) still is not working, as `ollama serve` reports: ``` "Dynamic LLM libraries" runners="[cpu cuda_v12]" system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512= 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 ``` Compared to the pre-compiled binary from the Manjaro repo: ``` "Dynamic LLM libraries" runners="[cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx]" ```
GiteaMirror commented 2026-04-12 16:47:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

It looks like it built only one runner, but that runner has all the flags enabled. If you want one runner per flag, I think you'll have to do multiple builds. I only have a passing acquaintance with the ollama build system, somebody else might know a better way to do this.

<!-- gh-comment-id:2614495474 --> @rick-github commented on GitHub (Jan 26, 2025): It looks like it built only one runner, but that runner has all the flags enabled. If you want one runner per flag, I think you'll have to do multiple builds. I only have a passing acquaintance with the ollama build system, somebody else might know a better way to do this.
GiteaMirror commented 2026-04-12 16:47:25 -05:00
Author
Owner
Copy Link

@graynoir commented on GitHub (Jan 26, 2025):

Oh, I see, so all the features might be enabled in the "cpu" runner... Strange, I thought there was a performance gain from enabling AVX-512, but there seems to be none, tested with qwq-32b-preview:

  • For the pre-compiled binary:
system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8
...
llm_load_tensors: offloading 37 repeating layers to GPU
...
total duration:       20.878795065s
load duration:        13.259976ms
prompt eval count:    484 token(s)
prompt eval duration: 2.2s
prompt eval rate:     220.00 tokens/s
eval count:           56 token(s)
eval duration:        18.33s
eval rate:            3.06 tokens/s
  • For the most recent docker build:
system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512= 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8
...
llm_load_tensors: offloading 37 repeating layers to GPU
...
total duration:       20.281865602s
load duration:        19.864569ms
prompt eval count:    484 token(s)
prompt eval duration: 2.582s
prompt eval rate:     187.45 tokens/s
eval count:           52 token(s)
eval duration:        17.356s
eval rate:            3.00 tokens/s

Which makes me wonder if the AVX-512 features are actually enabled..

<!-- gh-comment-id:2614508334 --> @graynoir commented on GitHub (Jan 26, 2025): Oh, I see, so all the features might be enabled in the "cpu" runner... Strange, I thought there was a performance gain from enabling AVX-512, but there seems to be none, tested with qwq-32b-preview: * For the pre-compiled binary: ``` system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 ... llm_load_tensors: offloading 37 repeating layers to GPU ... total duration: 20.878795065s load duration: 13.259976ms prompt eval count: 484 token(s) prompt eval duration: 2.2s prompt eval rate: 220.00 tokens/s eval count: 56 token(s) eval duration: 18.33s eval rate: 3.06 tokens/s ``` * For the most recent docker build: ``` system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3= 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512= 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 ... llm_load_tensors: offloading 37 repeating layers to GPU ... total duration: 20.281865602s load duration: 19.864569ms prompt eval count: 484 token(s) prompt eval duration: 2.582s prompt eval rate: 187.45 tokens/s eval count: 52 token(s) eval duration: 17.356s eval rate: 3.00 tokens/s ``` Which makes me wonder if the AVX-512 features are actually enabled..
GiteaMirror commented 2026-04-12 16:47:25 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

It could be that avx512 just doesn't add much processing power, see a previous post.

<!-- gh-comment-id:2614609509 --> @rick-github commented on GitHub (Jan 26, 2025): It could be that avx512 just doesn't add much processing power, see a [previous post](https://github.com/ollama/ollama/issues/6312#issuecomment-2295320574).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5534
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?