[GH-ISSUE #2205] Support additional AVX instruction sets #47773

New Issue

Closed

opened 2026-04-28 05:17:10 -05:00 by GiteaMirror · 20 comments

GiteaMirror commented 2026-04-28 05:17:10 -05:00

Owner

Copy Link

Originally created by @ddpasa on GitHub (Jan 26, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2205

Originally assigned to: @dhiltgen on GitHub.

I have a intel CPU that supports a number of AVX features, but most of them are not picked up when using ollama. Below is the llama.log file:

system info: AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

However, when I look at lscpu, I see that avx512 and avx512_vnni are actually supported.

I'm running on Manjaro Linux with Ollama installed from official repos. It's an Intel Core i7-1065G7 with Iris Plus G7 onboard iGPU. (the iGPU works very well with ncnn vulkan inference)

Originally created by @ddpasa on GitHub (Jan 26, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2205 Originally assigned to: @dhiltgen on GitHub. I have a intel CPU that supports a number of AVX features, but most of them are not picked up when using ollama. Below is the llama.log file: system info: AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | However, when I look at lscpu, I see that avx512 and avx512_vnni are actually supported. I'm running on Manjaro Linux with Ollama installed from official repos. It's an Intel Core i7-1065G7 with Iris Plus G7 onboard iGPU. (the iGPU works very well with ncnn vulkan inference)

GiteaMirror added the buildfeature request labels 2026-04-28 05:17:10 -05:00

GiteaMirror closed this issue 2026-04-28 05:17:11 -05:00

GiteaMirror commented 2026-04-28 05:17:12 -05:00

Author

Owner

Copy Link

@dhiltgen commented on GitHub (Jan 26, 2024):

These settings are compiled into the Ollama binary at build time. We build a number of variants for CPU based use https://github.com/ollama/ollama/blob/main/llm/generate/gen_linux.sh#L69-L115 and then select from these based on what we detect at runtime.

What we've found when testing is AVX compared to no vector feature gives a ~400% speed boost. AVX2 adds another ~10% on top of that, and when we tried the AVX512 flags, performance wasn't improved on our test systems.

If you build from source, we've added a mechanism to set the flags at build time with OLLAMA_CUSTOM_CPU_DEFS which is described here https://github.com/ollama/ollama/blob/main/docs/development.md#advanced-cpu-settings

Each variant we add adds complexity and size to the system, so we're trying to make sure each one adds enough value to justify. I'm going to close this ticket for now as "working as designed" however, if you have a system where you're able to demonstrate a significant performance improvement by setting a different combination of compile flags, please re-open with more details on CPU model, and the performance benefit and we can consider adding a 4th CPU variant.

<!-- gh-comment-id:1912615902 --> @dhiltgen commented on GitHub (Jan 26, 2024): These settings are compiled into the Ollama binary at build time. We build a number of variants for CPU based use https://github.com/ollama/ollama/blob/main/llm/generate/gen_linux.sh#L69-L115 and then select from these based on what we detect at runtime. What we've found when testing is AVX compared to no vector feature gives a ~400% speed boost. AVX2 adds another ~10% on top of that, and when we tried the AVX512 flags, performance wasn't improved on our test systems. If you build from source, we've added a mechanism to set the flags at build time with `OLLAMA_CUSTOM_CPU_DEFS` which is described here https://github.com/ollama/ollama/blob/main/docs/development.md#advanced-cpu-settings Each variant we add adds complexity and size to the system, so we're trying to make sure each one adds enough value to justify. I'm going to close this ticket for now as "working as designed" however, if you have a system where you're able to demonstrate a significant performance improvement by setting a different combination of compile flags, please re-open with more details on CPU model, and the performance benefit and we can consider adding a 4th CPU variant.

GiteaMirror commented 2026-04-28 05:17:13 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Jan 27, 2024):

Hello @dhiltgen , thanks for your quick reply and detailed explanation.

As you suggested, I recompiled ollama from source (it was really easy!) with the following flags:
OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

The inference time with llava went down from approximately 8 minutes to less than 2 minutes! This is a major improvement in performance.

Is it not possible to compile these with all the flags enabled, and then at runtime perform a check to see what the cpu supports?

<!-- gh-comment-id:1913124918 --> @ddpasa commented on GitHub (Jan 27, 2024): Hello @dhiltgen , thanks for your quick reply and detailed explanation. As you suggested, I recompiled ollama from source (it was really easy!) with the following flags: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" The inference time with llava went down from approximately 8 minutes to less than 2 minutes! This is a major improvement in performance. Is it not possible to compile these with all the flags enabled, and then at runtime perform a check to see what the cpu supports?

GiteaMirror commented 2026-04-28 05:17:14 -05:00

Author

Owner

Copy Link

@dhiltgen commented on GitHub (Jan 27, 2024):

That's significant!

So just to clarify, before making this change, on your system we load the "cpu_avx2" variant, and your llava scenario took 8 minutes. With this refined set of custom CPU flags, the same scenario on CPU took 2 minutes. Correct?

If that's all correct, please share some more info so we can attempt to repro. What type of CPU are you using? What was your prompt?

<!-- gh-comment-id:1913231249 --> @dhiltgen commented on GitHub (Jan 27, 2024): That's significant! So just to clarify, before making this change, on your system we load the "cpu_avx2" variant, and your llava scenario took 8 minutes. With this refined set of custom CPU flags, the same scenario on CPU took 2 minutes. Correct? If that's all correct, please share some more info so we can attempt to repro. What type of CPU are you using? What was your prompt?

GiteaMirror commented 2026-04-28 05:17:15 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Jan 27, 2024):

That's significant!

So just to clarify, before making this change, on your system we load the "cpu_avx2" variant, and your llava scenario took 8 minutes. With this refined set of custom CPU flags, the same scenario on CPU took 2 minutes. Correct?

If that's all correct, please share some more info so we can attempt to repro. What type of CPU are you using? What was your prompt?

Hello @dhiltgen , yes that's exactly right.

I was originally using the ollama version from the manjaro repos. Just to exclude any funny business happening (maybe they compiled it wrong?), let me run a rigorous benchmark to see how different flags affect the runtime. Are there any flag combinations I should test?

<!-- gh-comment-id:1913300849 --> @ddpasa commented on GitHub (Jan 27, 2024): > That's significant! > > So just to clarify, before making this change, on your system we load the "cpu_avx2" variant, and your llava scenario took 8 minutes. With this refined set of custom CPU flags, the same scenario on CPU took 2 minutes. Correct? > > If that's all correct, please share some more info so we can attempt to repro. What type of CPU are you using? What was your prompt? Hello @dhiltgen , yes that's exactly right. I was originally using the ollama version from the manjaro repos. Just to exclude any funny business happening (maybe they compiled it wrong?), let me run a rigorous benchmark to see how different flags affect the runtime. Are there any flag combinations I should test?

GiteaMirror commented 2026-04-28 05:17:16 -05:00

Author

Owner

Copy Link

@dhiltgen commented on GitHub (Jan 27, 2024):

I'd recommend checking out an unmodified HEAD from main, compile that as is, and run a test. Set OLLAMA_DEBUG=1 for extra verbosity in the server logs, and make sure it's loading cpu_avx2. Then run a model with /set verbose so you can see TPS, and send a single prompt to get your baseline performance number.

Then build with your custom CPU flags, and repeat the experiment with the same model and prompt.

With the debug flag set, you'll see a line in the server log that looks something like this which will also help confirm everything got set up the way you intended.

[1706381855] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

<!-- gh-comment-id:1913306715 --> @dhiltgen commented on GitHub (Jan 27, 2024): I'd recommend checking out an unmodified HEAD from main, compile that as is, and run a test. Set OLLAMA_DEBUG=1 for extra verbosity in the server logs, and make sure it's loading cpu_avx2. Then run a model with `/set verbose` so you can see TPS, and send a single prompt to get your baseline performance number. Then build with your custom CPU flags, and repeat the experiment with the same model and prompt. With the debug flag set, you'll see a line in the server log that looks something like this which will also help confirm everything got set up the way you intended. ``` [1706381855] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | ```

GiteaMirror commented 2026-04-28 05:17:16 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Jan 28, 2024):

I conducted all tests with the Bakllava model here: https://ollama.ai/library/bakllava, using the same seed=100.

Details are below. The magic seems to be in VNNI. AVX512 helps a little, but it's not a gamechanger. VNNI makes a huge improvement. I don't know much about it, a very short search found this blog: https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Deep-Learning-Performance-Boost-by-Intel-VNNI/post/1335670

An unrelated issue is that removing FMA flag gives very cryptic compile errors and I could not get it working without FMA. That's why there is no test without FMA. Let me know if I should open a new issue for that.

The version from the Manjaro repos is even slower than my v1 version. I'll follow up on that separately with the manjaro folks to see what's going on. (maybe the difference is that my v1 has FMA but the default flags do not)

v1: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on"
v2: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on -DLLAMA_AVX512=on"
v3: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on"
v4: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

v1: 274.8689343929291 seconds
v2: 258.70444440841675 seconds
v3: 259.50786542892456 seconds
v4: 117.119699716568 seconds

System info:

from inxi -F
CPU Info: quad core model: Intel Core i7-1065G7 bits: 64 type: MT MCP cache: L2: 2 MiB
This is an Ice Lake generation laptop CPU

from lscpu

    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pd
                         pe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmul
                         qdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsa
                         ve avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_
                         ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt intel_pt avx512cd 
                         sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_e
                         pp hwp_pkg_req vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid sgx_
                         lc fsrm md_clear flush_l1d arch_capabilities

<!-- gh-comment-id:1913588361 --> @ddpasa commented on GitHub (Jan 28, 2024): I conducted all tests with the Bakllava model here: https://ollama.ai/library/bakllava, using the same seed=100. Details are below. The magic seems to be in VNNI. AVX512 helps a little, but it's not a gamechanger. VNNI makes a huge improvement. I don't know much about it, a very short search found this blog: https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Deep-Learning-Performance-Boost-by-Intel-VNNI/post/1335670 An unrelated issue is that removing FMA flag gives very cryptic compile errors and I could not get it working without FMA. That's why there is no test without FMA. Let me know if I should open a new issue for that. The version from the Manjaro repos is even slower than my v1 version. I'll follow up on that separately with the manjaro folks to see what's going on. (maybe the difference is that my v1 has FMA but the default flags do not) v1: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on" v2: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on -DLLAMA_AVX512=on" v3: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_FMA=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on" v4: OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" v1: 274.8689343929291 seconds v2: 258.70444440841675 seconds v3: 259.50786542892456 seconds v4: 117.119699716568 seconds System info: from inxi -F `CPU Info: quad core model: Intel Core i7-1065G7 bits: 64 type: MT MCP cache: L2: 2 MiB` This is an Ice Lake generation laptop CPU from lscpu ``` Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pd pe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmul qdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsa ve avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_e pp hwp_pkg_req vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid sgx_ lc fsrm md_clear flush_l1d arch_capabilities ```

GiteaMirror commented 2026-04-28 05:17:16 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Jan 28, 2024):

@dhiltgen , let me know if you need more tests or additional information.

<!-- gh-comment-id:1913588496 --> @ddpasa commented on GitHub (Jan 28, 2024): @dhiltgen , let me know if you need more tests or additional information.

GiteaMirror commented 2026-04-28 05:17:17 -05:00

Author

Owner

Copy Link

@jukofyork commented on GitHub (Jan 28, 2024):

If you build from source, we've added a mechanism to set the flags at build time with OLLAMA_CUSTOM_CPU_DEFS which is described here https://github.com/ollama/ollama/blob/main/docs/development.md#advanced-cpu-settings

Oh I wish I'd known about this! I made a shell script to do this but the location kept moving in the source tree and had to use rgrep to find it... Then one day I forgot to cd into the Ollama source folder and my script committed suicide by running sed on itself 😁

<!-- gh-comment-id:1913712205 --> @jukofyork commented on GitHub (Jan 28, 2024): > If you build from source, we've added a mechanism to set the flags at build time with `OLLAMA_CUSTOM_CPU_DEFS` which is described here https://github.com/ollama/ollama/blob/main/docs/development.md#advanced-cpu-settings Oh I wish I'd known about this! I made a shell script to do this but the location kept moving in the source tree and had to use rgrep to find it... Then one day I forgot to cd into the Ollama source folder and my script committed suicide by running sed on itself 😁

GiteaMirror commented 2026-04-28 05:17:17 -05:00

Author

Owner

Copy Link

@dhiltgen commented on GitHub (Jan 28, 2024):

@ddpasa this sounds promising. I've tried to reproduce, and haven't had luck yet. I'm sending a "why is the sky blue" prompt to llama2 with the ollama CLI and /set verbose set. I've tried on both a recent Intel and AMD system.

My baseline which is our current cpu_avx2 variant on a 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

[1706476798] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

yields:

eval rate:            8.43 tokens/s

My test which I believe I configured the same as your experiment 4

[1706477370] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

yields:

eval rate:            8.45 tokens/s

Are you only seeing the performance improvement on multimodal models?

<!-- gh-comment-id:1913734777 --> @dhiltgen commented on GitHub (Jan 28, 2024): @ddpasa this sounds promising. I've tried to reproduce, and haven't had luck yet. I'm sending a "why is the sky blue" prompt to llama2 with the ollama CLI and `/set verbose` set. I've tried on both a recent Intel and AMD system. My baseline which is our current `cpu_avx2` variant on a `11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz` ``` [1706476798] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | ``` yields: ``` eval rate: 8.43 tokens/s ``` My test which I believe I configured the same as your experiment 4 ``` [1706477370] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | ``` yields: ``` eval rate: 8.45 tokens/s ``` Are you only seeing the performance improvement on multimodal models?

GiteaMirror commented 2026-04-28 05:17:18 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Jan 29, 2024):

Hello @dhiltgen

Are you only seeing the performance improvement on multimodal models?

I ran inference with the Phi-2 model here and I think you're right! The v4 version with vnni is still faster than v1 with optimizations only up to avx2, but just marginally so. I don't see the same dramatic improvement I see in Bakllava.

v1: (only up to avx2)
39.85 ms per token, 25.09 tokens per second
v4: (with vnni)
36.60 ms per token, 27.32 tokens per second

I queried both ollama versions with:
curl http://localhost:11434/api/generate -d '{"model": "phi", "prompt": "Why is the sky blue?", "stream": false, "options": {"seed": 100}}'

For multimodel models like Llava and Bakllava the image encoding part is pretty expensive. Maybe it's helping there?

<!-- gh-comment-id:1914172364 --> @ddpasa commented on GitHub (Jan 29, 2024): Hello @dhiltgen > Are you only seeing the performance improvement on multimodal models? I ran inference with the [Phi-2 model here](https://ollama.ai/library/phi) and I think you're right! The v4 version with vnni is still faster than v1 with optimizations only up to avx2, but just marginally so. I don't see the same dramatic improvement I see in [Bakllava](https://ollama.ai/library/bakllava). v1: (only up to avx2) 39.85 ms per token, 25.09 tokens per second v4: (with vnni) 36.60 ms per token, 27.32 tokens per second I queried both ollama versions with: `curl http://localhost:11434/api/generate -d '{"model": "phi", "prompt": "Why is the sky blue?", "stream": false, "options": {"seed": 100}}'` For multimodel models like Llava and Bakllava the image encoding part is pretty expensive. Maybe it's helping there?

GiteaMirror commented 2026-04-28 05:17:19 -05:00

Author

Owner

Copy Link

@dhiltgen commented on GitHub (Jan 29, 2024):

That's a good datapoint. Let me explore multimodal performance a bit more.

<!-- gh-comment-id:1915103164 --> @dhiltgen commented on GitHub (Jan 29, 2024): That's a good datapoint. Let me explore multimodal performance a bit more.

GiteaMirror commented 2026-04-28 05:17:19 -05:00

Author

Owner

Copy Link

@itsdotscience commented on GitHub (Feb 22, 2024):

AMD

Might be able to help with regard to both AMD AVX512 and the no improvement over AVX2 on some systems. Lets start with AMD because I've been eager to have someone try this on some AMD hardware. I moved the longer boring bits to the end, feel free to skip if you want to just give it a try.

For improved AMD cpu perf (using llama.cpp int his case as its blas/lapack friendly, for now) on inference you can try https://www.amd.com/en/developer/aocl.html#downloads either the package or the tar ball. Unsure about where the package will place them but the install.sh updates the libs and other env vars if on linux. Then in the root of a checkout of the llama.cpp git run (if previouslly used, make clean or otherwise remove CMakeCache.txt and CMakeFiles first)

cmake . -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON

if it doesn't find the libs add

-DBLAS_LIBRARIES=/lib/libblis-mt.so

on win replace

libblis-mt.so with LibBlis-Win-MT-dll.dll

no not a typo. it really has -dll in the file name just before the extension. if it cant find the includes add

-DBLAS_INCLUDE_DIRS=<path to install location/include>

if testing in the presence of an AMD or NVIDIA GPU you should likely add

-DLLAMA_HIPBLAS=OFF or -DLLAMA_CUBLAS=OFF just to ensure you get the AOCL-BLAS results when you run 'benchmark' from the 'bin' dir, default to single thread so you may want to ramp that up and observe if there is any change here as well. Finally build it.

cmake --build . --config Release .

---

Simply put regarding AVX512 being no more performant or even slow than AVX2 in some systems is complicated. The most common reason is the lower end chips just fused two 256 width register together to get the 512 width. High end xeon and the top range on the 2017-~2019 X Series aka SkylakeX 10+ core models not only have a 512 width unit but two of them per core

Going back to why AMD chips tend to not end up utilizing their AVX instruction sets most often its due to Intel MKL which bundles a bunch of kernels in compiled code then leverages the CPUID instruction to pick the "best" one. For Intel branded chips this more or less works nicely. For none they get the "most compatible" kernel, read "basic" "slow" . MKL Obviously isn't the only way for something to leverage the instruction sets it's one of the easiest. This is especially true if underlying components wrap around BLAS/LAPACK . Much of llamacpp does, however 'main' no longer does but 'server' does. Likewise 'benchmark' which should make it easy to figure out if we're on the right track.

Enter 'AMD Optimizing CPU Libraries' aka AOCL.

https://www.amd.com/en/developer/aocl.html

There's quite a few but for improving cpu only and proving a benefit to the AMD AVX512 instructions we're gonna need an optimized routine to do matrix-matrix multiplication and other ops. A BLAS/BLIS implementation will generally include this and AMD's AOCL-BLAS has what should be a compliant a/z/s/gemm routine.

https://www.amd.com/en/developer/aocl/dense.html

---

All that said, the matmul routines in llama.cpp aren't bad at all, other than when I can leverage 2x512 registers and the like the stock mkl as blas vs builtin are on par gflop wise but the built in routine tended to have lower overhead. Using the same as above, setting the BLAS_VENDOR and using https://cmake.org/cmake/help/latest/module/FindBLAS.html as a reference you can play around with various libs, variations and options if so inclined.

Hope that helps! I'm very eager to see if AOCL delivered anything meaningful :)

<!-- gh-comment-id:1958505548 --> @itsdotscience commented on GitHub (Feb 22, 2024): > AMD Might be able to help with regard to both AMD AVX512 and the no improvement over AVX2 on some systems. Lets start with AMD because I've been eager to have someone try this on some AMD hardware. I moved the longer boring bits to the end, feel free to skip if you want to just give it a try. For improved AMD cpu perf (using llama.cpp int his case as its blas/lapack friendly, for now) on inference you can try https://www.amd.com/en/developer/aocl.html#downloads either the package or the tar ball. Unsure about where the package will place them but the install.sh updates the libs and other env vars if on linux. Then in the root of a checkout of the llama.cpp git run (if previouslly used, make clean or otherwise remove CMakeCache.txt and CMakeFiles first) cmake . -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON if it doesn't find the libs add -DBLAS_LIBRARIES=<path to install location>/lib/libblis-mt.so on win replace libblis-mt.so with LibBlis-Win-MT-dll.dll no not a typo. it really has -dll in the file name just before the extension. if it cant find the includes add -DBLAS_INCLUDE_DIRS=<path to install location/include> if testing in the presence of an AMD or NVIDIA GPU you should likely add -DLLAMA_HIPBLAS=OFF or -DLLAMA_CUBLAS=OFF just to ensure you get the AOCL-BLAS results when you run 'benchmark' from the 'bin' dir, default to single thread so you may want to ramp that up and observe if there is any change here as well. Finally build it. cmake --build . --config Release . ------ Simply put regarding AVX512 being no more performant or even slow than AVX2 in some systems is complicated. The most common reason is the lower end chips just fused two 256 width register together to get the 512 width. High end xeon and the top range on the 2017-~2019 X Series aka SkylakeX 10+ core models not only have a 512 width unit but two of them per core Going back to why AMD chips tend to not end up utilizing their AVX instruction sets most often its due to Intel MKL which bundles a bunch of kernels in compiled code then leverages the CPUID instruction to pick the "best" one. For Intel branded chips this more or less works nicely. For none they get the "most compatible" kernel, read "basic" "slow" . MKL Obviously isn't the only way for something to leverage the instruction sets it's one of the easiest. This is especially true if underlying components wrap around BLAS/LAPACK . Much of llamacpp does, however 'main' no longer does but 'server' does. Likewise 'benchmark' which should make it easy to figure out if we're on the right track. Enter 'AMD Optimizing CPU Libraries' aka AOCL. https://www.amd.com/en/developer/aocl.html There's quite a few but for improving cpu only and proving a benefit to the AMD AVX512 instructions we're gonna need an optimized routine to do matrix-matrix multiplication and other ops. A BLAS/BLIS implementation will generally include this and AMD's AOCL-BLAS has what should be a compliant a/z/s/gemm routine. https://www.amd.com/en/developer/aocl/dense.html ---- All that said, the matmul routines in llama.cpp aren't bad at all, other than when I can leverage 2x512 registers and the like the stock mkl as blas vs builtin are on par gflop wise but the built in routine tended to have lower overhead. Using the same as above, setting the BLAS_VENDOR and using https://cmake.org/cmake/help/latest/module/FindBLAS.html as a reference you can play around with various libs, variations and options if so inclined. Hope that helps! I'm very eager to see if AOCL delivered anything meaningful :)

GiteaMirror commented 2026-04-28 05:17:19 -05:00

Author

Owner

Copy Link

@Neurrone commented on GitHub (Mar 3, 2024):

@itsdotscience I've tried compiling it with AOCL but the build fails immediately. I'm on Windows 11 running this on a Ryzen 7840U.

On installing the libraries, I see there are ILP64 and LP64 variants, I wasn't sure which one to choose but both resulted in the same error.

I generated the build files with this command: cmake .. --fresh -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES="C:\Program Files\AMD\AOCL-Windows\amd-blis\lib\ILP64\AOCL-LibBlis-Win-MT-dll.dll" -DBLAS_INCLUDE_DIRS="C:\Program Files\AMD\AOCL-Windows\amd-blis\include\ILP64"

D:\source\llama.cpp\build>cmake --build . --config Release
MSBuild version 17.8.5+b5265ef37 for .NET Framework

  build_info.vcxproj -> D:\source\llama.cpp\build\common\build_info.dir\Release\build_info.lib
  ggml.c
D:\source\llama.cpp\ggml.c(358,6): error C2371: 'ggml_guid_matches': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(683,19):
  see declaration of 'ggml_guid_matches'

D:\source\llama.cpp\ggml.c(2183,6): error C2371: 'ggml_is_numa': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(696,22):
  see declaration of 'ggml_is_numa'

D:\source\llama.cpp\ggml.c(2263,16): error C2371: 'ggml_is_quantized': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(723,32):
  see declaration of 'ggml_is_quantized'

D:\source\llama.cpp\ggml.c(2293,6): error C2371: 'ggml_is_scalar': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(731,29):
  see declaration of 'ggml_is_scalar'

D:\source\llama.cpp\ggml.c(2299,6): error C2371: 'ggml_is_vector': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(732,29):
  see declaration of 'ggml_is_vector'

D:\source\llama.cpp\ggml.c(2305,6): error C2371: 'ggml_is_matrix': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(733,29):
  see declaration of 'ggml_is_matrix'

D:\source\llama.cpp\ggml.c(2311,6): error C2371: 'ggml_is_3d': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(734,29):
  see declaration of 'ggml_is_3d'

D:\source\llama.cpp\ggml.c(2377,16): error C2371: 'ggml_is_transposed': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(728,29):
  see declaration of 'ggml_is_transposed'

D:\source\llama.cpp\ggml.c(2381,16): error C2371: 'ggml_is_contiguous': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(729,29):
  see declaration of 'ggml_is_contiguous'

D:\source\llama.cpp\ggml.c(2400,16): error C2371: 'ggml_is_permuted': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(730,29):
  see declaration of 'ggml_is_permuted'

D:\source\llama.cpp\ggml.c(2415,6): error C2371: 'ggml_are_same_shape': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(737,19):
  see declaration of 'ggml_are_same_shape'

D:\source\llama.cpp\ggml.c(2625,6): error C2371: 'ggml_get_no_alloc': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(750,22):
  see declaration of 'ggml_get_no_alloc'

D:\source\llama.cpp\ggml.c(2629,56): warning C4028: formal parameter 2 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(4097,61): warning C4028: formal parameter 4 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(5414,31): warning C4028: formal parameter 6 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(5435,31): warning C4028: formal parameter 15 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(5647,26): warning C4028: formal parameter 10 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(5945,31): warning C4028: formal parameter 5 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(6010,31): warning C4028: formal parameter 6 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(15782,6): error C2371: 'ggml_hash_contains': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml-impl.h(248,8):
  see declaration of 'ggml_hash_contains'

D:\source\llama.cpp\ggml.c(16885,115): warning C4028: formal parameter 4 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(16940,53): warning C4028: formal parameter 2 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(16948,89): warning C4028: formal parameter 3 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(18740,57): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(18831,61): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(18973,65): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(19116,61): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj]
D:\source\llama.cpp\ggml.c(19702,6): error C2371: 'ggml_quantize_requires_imatrix': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcx
proj]
  D:\source\llama.cpp\ggml.h(2166,19):
  see declaration of 'ggml_quantize_requires_imatrix'

D:\source\llama.cpp\ggml.c(20583,6): error C2371: 'gguf_get_val_bool': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj]
  D:\source\llama.cpp\ggml.h(2233,27):
  see declaration of 'gguf_get_val_bool'

<!-- gh-comment-id:1975134075 --> @Neurrone commented on GitHub (Mar 3, 2024): @itsdotscience I've tried compiling it with AOCL but the build fails immediately. I'm on Windows 11 running this on a Ryzen 7840U. On installing the libraries, I see there are ILP64 and LP64 variants, I wasn't sure which one to choose but both resulted in the same error. I generated the build files with this command: `cmake .. --fresh -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES="C:\Program Files\AMD\AOCL-Windows\amd-blis\lib\ILP64\AOCL-LibBlis-Win-MT-dll.dll" -DBLAS_INCLUDE_DIRS="C:\Program Files\AMD\AOCL-Windows\amd-blis\include\ILP64"` ```bash D:\source\llama.cpp\build>cmake --build . --config Release MSBuild version 17.8.5+b5265ef37 for .NET Framework build_info.vcxproj -> D:\source\llama.cpp\build\common\build_info.dir\Release\build_info.lib ggml.c D:\source\llama.cpp\ggml.c(358,6): error C2371: 'ggml_guid_matches': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(683,19): see declaration of 'ggml_guid_matches' D:\source\llama.cpp\ggml.c(2183,6): error C2371: 'ggml_is_numa': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(696,22): see declaration of 'ggml_is_numa' D:\source\llama.cpp\ggml.c(2263,16): error C2371: 'ggml_is_quantized': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(723,32): see declaration of 'ggml_is_quantized' D:\source\llama.cpp\ggml.c(2293,6): error C2371: 'ggml_is_scalar': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(731,29): see declaration of 'ggml_is_scalar' D:\source\llama.cpp\ggml.c(2299,6): error C2371: 'ggml_is_vector': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(732,29): see declaration of 'ggml_is_vector' D:\source\llama.cpp\ggml.c(2305,6): error C2371: 'ggml_is_matrix': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(733,29): see declaration of 'ggml_is_matrix' D:\source\llama.cpp\ggml.c(2311,6): error C2371: 'ggml_is_3d': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(734,29): see declaration of 'ggml_is_3d' D:\source\llama.cpp\ggml.c(2377,16): error C2371: 'ggml_is_transposed': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(728,29): see declaration of 'ggml_is_transposed' D:\source\llama.cpp\ggml.c(2381,16): error C2371: 'ggml_is_contiguous': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(729,29): see declaration of 'ggml_is_contiguous' D:\source\llama.cpp\ggml.c(2400,16): error C2371: 'ggml_is_permuted': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(730,29): see declaration of 'ggml_is_permuted' D:\source\llama.cpp\ggml.c(2415,6): error C2371: 'ggml_are_same_shape': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(737,19): see declaration of 'ggml_are_same_shape' D:\source\llama.cpp\ggml.c(2625,6): error C2371: 'ggml_get_no_alloc': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(750,22): see declaration of 'ggml_get_no_alloc' D:\source\llama.cpp\ggml.c(2629,56): warning C4028: formal parameter 2 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(4097,61): warning C4028: formal parameter 4 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(5414,31): warning C4028: formal parameter 6 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(5435,31): warning C4028: formal parameter 15 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(5647,26): warning C4028: formal parameter 10 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(5945,31): warning C4028: formal parameter 5 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(6010,31): warning C4028: formal parameter 6 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(15782,6): error C2371: 'ggml_hash_contains': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml-impl.h(248,8): see declaration of 'ggml_hash_contains' D:\source\llama.cpp\ggml.c(16885,115): warning C4028: formal parameter 4 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(16940,53): warning C4028: formal parameter 2 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(16948,89): warning C4028: formal parameter 3 different from declaration [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(18740,57): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(18831,61): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(18973,65): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(19116,61): warning C4133: 'function': incompatible types - from 'bool *' to 'bool *' [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.c(19702,6): error C2371: 'ggml_quantize_requires_imatrix': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcx proj] D:\source\llama.cpp\ggml.h(2166,19): see declaration of 'ggml_quantize_requires_imatrix' D:\source\llama.cpp\ggml.c(20583,6): error C2371: 'gguf_get_val_bool': redefinition; different basic types [D:\source\llama.cpp\build\ggml.vcxproj] D:\source\llama.cpp\ggml.h(2233,27): see declaration of 'gguf_get_val_bool' ```

GiteaMirror commented 2026-04-28 05:17:20 -05:00

Author

Owner

Copy Link

@m4r1k commented on GitHub (Mar 20, 2024):

I'd like to provide some extra data points.

All my tests have been conducted on Ubuntu 22.04.4, kernel 6.5.0-1016-gcp. The underneath hardware is a GCP C3 instance using Intel Sapphire Rapids, specifically Xeon 8481C while the machine type is a c3-standard-8, translation: 8 vCPU / 4 pCores, SMT enabled, and 32GB of memory.

AVX512 and AMX are available:

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities

All deps were installed through apt / snap:

# gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# cmake --version
cmake version 3.22.1

# go version
go version go1.22.1 linux/amd64

Last but not least, ollama was compiled from main commit 7ed3e94 and I've used the Gemma 7B model.

Taking @ddpasa four versions, here the results:

• v1 189.14 ms per token, 5.29 tokens per second
• v2 188.88 ms per token, 5.29 tokens per second
• v3 189.26 ms per token, 5.28 tokens per second
• v4 188.99 ms per token, 5.29 tokens per second

As you can see the throughput is the same across all four iterations. Checking system info, it shows that AVX512 and VNNI were not used:

• v1 system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
• v2 system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
• v3 system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
• v4 system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |

I then did the following patch:

diff --git a/gpu/cpu_common.go b/gpu/cpu_common.go
index 3b299e4..607a19e 100644
--- a/gpu/cpu_common.go
+++ b/gpu/cpu_common.go
@@ -7,6 +7,10 @@ import (
 )

 func GetCPUVariant() string {
+       if cpu.X86.HasAVX512 {
+               slog.Info("CPU has AVX512")
+               return "avx512"
+       }
        if cpu.X86.HasAVX2 {
                slog.Info("CPU has AVX2")
                return "avx2"

At this point, across the four versions, AVX512 was recognized according to system info: system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |.

Throughout wise, whether compiled with or without AVX512/VNNI, it yields the very same outcome, although ~6% slower:

• v1 199.59 ms per token, 5.01 tokens per second
• v2 200.23 ms per token, 4.99 tokens per second
• v3 200.55 ms per token, 4.99 tokens per second
• v4 200.37 ms per token, 4.99 tokens per second

The last thing done was creating a new AVX512 CPU target:

diff --git a/llm/generate/gen_linux.sh b/llm/generate/gen_linux.sh
index 591fc80..2af1c2d 100755
--- a/llm/generate/gen_linux.sh
+++ b/llm/generate/gen_linux.sh
@@ -115,6 +115,15 @@ if [ -z "${OLLAMA_SKIP_CPU_GENERATE}" ]; then
             build
             compress_libs
         fi
+
+        if [ -z "${OLLAMA_CPU_TARGET}" -o "${OLLAMA_CPU_TARGET}" = "cpu_avx512" ]; then
+            init_vars
+            CMAKE_DEFS="${COMMON_CPU_DEFS} -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_FMA=on -DLLAMA_F16C=on ${CMAKE_DEFS}"
+            BUILD_DIR="${LLAMACPP_DIR}/build/linux/${ARCH}/cpu_avx512"
+            echo "Building AVX512 w/ VNNI CPU"
+            build
+            compress_libs
+        fi
     fi
 else
     echo "Skipping CPU generation step as requested"

Ollama was compiled with OLLAMA_CPU_TARGET="cpu_avx512" go generate ./... and now when serving it reports Dynamic LLM libraries [cpu_avx2 cpu_avx512 cpu_avx cpu], despite that, the TPS is kinda low: 188.85 ms per token, 5.30 tokens per second.

What's your take? To me it feels like either Gemma doesn't take advantage of AVX512/VNNI or something else is going on with ollama / llama.cpp preventing to speed up inference.

All tests were done using the same prompt and seed:

# curl http://localhost:11434/api/generate -d '{"model": "gemma:7b", "prompt": "Why is the sky blue?",  "stream": false, "options": {"seed": 100}}'

<!-- gh-comment-id:2009571248 --> @m4r1k commented on GitHub (Mar 20, 2024): I'd like to provide some extra data points. All my tests have been conducted on Ubuntu 22.04.4, kernel `6.5.0-1016-gcp`. The underneath hardware is a GCP C3 instance using Intel Sapphire Rapids, specifically [Xeon 8481C](https://cloud.google.com/compute/docs/cpu-platforms#:~:text=Intel%C2%AE%20Xeon%C2%AE%20Platinum%208481C%20Processor) while the machine type is a `c3-standard-8`, translation: 8 vCPU / 4 pCores, SMT enabled, and 32GB of memory. AVX512 and AMX are available: ``` fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities ``` All deps were installed through apt / snap: ```bash # gcc --version gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # cmake --version cmake version 3.22.1 # go version go version go1.22.1 linux/amd64 ``` Last but not least, ollama was compiled from `main` commit `7ed3e94` and I've used the Gemma 7B model. Taking @ddpasa four versions, here the results: * **v1** `189.14 ms per token, 5.29 tokens per second` * **v2** `188.88 ms per token, 5.29 tokens per second` * **v3** `189.26 ms per token, 5.28 tokens per second` * **v4** `188.99 ms per token, 5.29 tokens per second` As you can see the throughput is the same across all four iterations. Checking system info, it shows that AVX512 and VNNI were not used: * **v1** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` * **v2** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` * **v3** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` * **v4** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` I then did the following patch: ```diff diff --git a/gpu/cpu_common.go b/gpu/cpu_common.go index 3b299e4..607a19e 100644 --- a/gpu/cpu_common.go +++ b/gpu/cpu_common.go @@ -7,6 +7,10 @@ import ( ) func GetCPUVariant() string { + if cpu.X86.HasAVX512 { + slog.Info("CPU has AVX512") + return "avx512" + } if cpu.X86.HasAVX2 { slog.Info("CPU has AVX2") return "avx2" ``` At this point, across the four versions, AVX512 was recognized according to system info: `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`. Throughout wise, whether compiled with or without AVX512/VNNI, it yields the very same outcome, although ~6% slower: * **v1** `199.59 ms per token, 5.01 tokens per second` * **v2** `200.23 ms per token, 4.99 tokens per second` * **v3** `200.55 ms per token, 4.99 tokens per second` * **v4** `200.37 ms per token, 4.99 tokens per second` The last thing done was creating a new AVX512 CPU target: ```diff diff --git a/llm/generate/gen_linux.sh b/llm/generate/gen_linux.sh index 591fc80..2af1c2d 100755 --- a/llm/generate/gen_linux.sh +++ b/llm/generate/gen_linux.sh @@ -115,6 +115,15 @@ if [ -z "${OLLAMA_SKIP_CPU_GENERATE}" ]; then build compress_libs fi + + if [ -z "${OLLAMA_CPU_TARGET}" -o "${OLLAMA_CPU_TARGET}" = "cpu_avx512" ]; then + init_vars + CMAKE_DEFS="${COMMON_CPU_DEFS} -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_FMA=on -DLLAMA_F16C=on ${CMAKE_DEFS}" + BUILD_DIR="${LLAMACPP_DIR}/build/linux/${ARCH}/cpu_avx512" + echo "Building AVX512 w/ VNNI CPU" + build + compress_libs + fi fi else echo "Skipping CPU generation step as requested" ``` Ollama was compiled with `OLLAMA_CPU_TARGET="cpu_avx512" go generate ./...` and now when serving it reports `Dynamic LLM libraries [cpu_avx2 cpu_avx512 cpu_avx cpu]`, despite that, the TPS is kinda low: `188.85 ms per token, 5.30 tokens per second`. What's your take? To me it feels like either Gemma doesn't take advantage of AVX512/VNNI or something else is going on with ollama / llama.cpp preventing to speed up inference. All tests were done using the same prompt and seed: ```bash # curl http://localhost:11434/api/generate -d '{"model": "gemma:7b", "prompt": "Why is the sky blue?", "stream": false, "options": {"seed": 100}}' ```

GiteaMirror commented 2026-04-28 05:17:20 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Mar 20, 2024):

I'd like to provide some extra data points.

All my tests have been conducted on Ubuntu 22.04.4, kernel 6.5.0-1016-gcp. The underneath hardware is a GCP C3 instance using Intel Sapphire Rapids, specifically Xeon 8481C while the machine type is a c3-standard-8, translation: 8 vCPU / 4 pCores, SMT enabled, and 32GB of memory.

AVX512 and AMX are available:

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities

All deps were installed through apt / snap:

# gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# cmake --version
cmake version 3.22.1

# go version
go version go1.22.1 linux/amd64

Last but not least, ollama was compiled from main commit 7ed3e94 and I've used the Gemma 7B model.

Taking @ddpasa four versions, here the results:

* **v1** `189.14 ms per token,     5.29 tokens per second`

* **v2** `188.88 ms per token,     5.29 tokens per second`

* **v3** `189.26 ms per token,     5.28 tokens per second`

* **v4** `188.99 ms per token,     5.29 tokens per second`

As you can see the throughput is the same across all four iterations. Checking system info, it shows that AVX512 and VNNI were not used:

* **v1** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`

* **v2** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`

* **v3** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`

* **v4** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`

I then did the following patch:

diff --git a/gpu/cpu_common.go b/gpu/cpu_common.go
index 3b299e4..607a19e 100644
--- a/gpu/cpu_common.go
+++ b/gpu/cpu_common.go
@@ -7,6 +7,10 @@ import (
 )

 func GetCPUVariant() string {
+       if cpu.X86.HasAVX512 {
+               slog.Info("CPU has AVX512")
+               return "avx512"
+       }
        if cpu.X86.HasAVX2 {
                slog.Info("CPU has AVX2")
                return "avx2"

At this point, across the four versions, AVX512 was recognized according to system info: system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |.

Throughout wise, whether compiled with or without AVX512/VNNI, it yields the very same outcome, although ~6% slower:

* **v1** `199.59 ms per token,     5.01 tokens per second`

* **v2** `200.23 ms per token,     4.99 tokens per second`

* **v3** `200.55 ms per token,     4.99 tokens per second`

* **v4** `200.37 ms per token,     4.99 tokens per second`

The last thing done was creating a new AVX512 CPU target:

diff --git a/llm/generate/gen_linux.sh b/llm/generate/gen_linux.sh
index 591fc80..2af1c2d 100755
--- a/llm/generate/gen_linux.sh
+++ b/llm/generate/gen_linux.sh
@@ -115,6 +115,15 @@ if [ -z "${OLLAMA_SKIP_CPU_GENERATE}" ]; then
             build
             compress_libs
         fi
+
+        if [ -z "${OLLAMA_CPU_TARGET}" -o "${OLLAMA_CPU_TARGET}" = "cpu_avx512" ]; then
+            init_vars
+            CMAKE_DEFS="${COMMON_CPU_DEFS} -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_FMA=on -DLLAMA_F16C=on ${CMAKE_DEFS}"
+            BUILD_DIR="${LLAMACPP_DIR}/build/linux/${ARCH}/cpu_avx512"
+            echo "Building AVX512 w/ VNNI CPU"
+            build
+            compress_libs
+        fi
     fi
 else
     echo "Skipping CPU generation step as requested"

Ollama was compiled with OLLAMA_CPU_TARGET="cpu_avx512" go generate ./... and now when serving it reports Dynamic LLM libraries [cpu_avx2 cpu_avx512 cpu_avx cpu], despite that, the TPS is kinda low: 188.85 ms per token, 5.30 tokens per second.

What's your take? To me it feels like either Gemma doesn't take advantage of AVX512/VNNI or something else is going on with ollama / llama.cpp preventing to speed up inference.

All tests were done using the same prompt and seed:

# curl http://localhost:11434/api/generate -d '{"model": "gemma:7b", "prompt": "Why is the sky blue?",  "stream": false, "options": {"seed": 100}}'

Hello @m4r1k , can you retry with a multimodal model, such as llava or bakllava? I think our conclusion is that VNNI instuctions make a huge difference in processing the image, which is why we see an improvement in multimodal models. But they don't make any difference with text only language models.

<!-- gh-comment-id:2009628698 --> @ddpasa commented on GitHub (Mar 20, 2024): > I'd like to provide some extra data points. > > All my tests have been conducted on Ubuntu 22.04.4, kernel `6.5.0-1016-gcp`. The underneath hardware is a GCP C3 instance using Intel Sapphire Rapids, specifically [Xeon 8481C](https://cloud.google.com/compute/docs/cpu-platforms#:~:text=Intel%C2%AE%20Xeon%C2%AE%20Platinum%208481C%20Processor) while the machine type is a `c3-standard-8`, translation: 8 vCPU / 4 pCores, SMT enabled, and 32GB of memory. > > AVX512 and AMX are available: > > ``` > fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities > ``` > > All deps were installed through apt / snap: > > ```shell > # gcc --version > gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 > Copyright (C) 2021 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > # cmake --version > cmake version 3.22.1 > > # go version > go version go1.22.1 linux/amd64 > ``` > > Last but not least, ollama was compiled from `main` commit `7ed3e94` and I've used the Gemma 7B model. > > Taking @ddpasa four versions, here the results: > > * **v1** `189.14 ms per token, 5.29 tokens per second` > > * **v2** `188.88 ms per token, 5.29 tokens per second` > > * **v3** `189.26 ms per token, 5.28 tokens per second` > > * **v4** `188.99 ms per token, 5.29 tokens per second` > > > As you can see the throughput is the same across all four iterations. Checking system info, it shows that AVX512 and VNNI were not used: > > * **v1** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` > > * **v2** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` > > * **v3** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` > > * **v4** `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` > > > I then did the following patch: > > ```diff > diff --git a/gpu/cpu_common.go b/gpu/cpu_common.go > index 3b299e4..607a19e 100644 > --- a/gpu/cpu_common.go > +++ b/gpu/cpu_common.go > @@ -7,6 +7,10 @@ import ( > ) > > func GetCPUVariant() string { > + if cpu.X86.HasAVX512 { > + slog.Info("CPU has AVX512") > + return "avx512" > + } > if cpu.X86.HasAVX2 { > slog.Info("CPU has AVX2") > return "avx2" > ``` > > At this point, across the four versions, AVX512 was recognized according to system info: `system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |`. > > Throughout wise, whether compiled with or without AVX512/VNNI, it yields the very same outcome, although ~6% slower: > > * **v1** `199.59 ms per token, 5.01 tokens per second` > > * **v2** `200.23 ms per token, 4.99 tokens per second` > > * **v3** `200.55 ms per token, 4.99 tokens per second` > > * **v4** `200.37 ms per token, 4.99 tokens per second` > > > The last thing done was creating a new AVX512 CPU target: > > ```diff > diff --git a/llm/generate/gen_linux.sh b/llm/generate/gen_linux.sh > index 591fc80..2af1c2d 100755 > --- a/llm/generate/gen_linux.sh > +++ b/llm/generate/gen_linux.sh > @@ -115,6 +115,15 @@ if [ -z "${OLLAMA_SKIP_CPU_GENERATE}" ]; then > build > compress_libs > fi > + > + if [ -z "${OLLAMA_CPU_TARGET}" -o "${OLLAMA_CPU_TARGET}" = "cpu_avx512" ]; then > + init_vars > + CMAKE_DEFS="${COMMON_CPU_DEFS} -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on -DLLAMA_FMA=on -DLLAMA_F16C=on ${CMAKE_DEFS}" > + BUILD_DIR="${LLAMACPP_DIR}/build/linux/${ARCH}/cpu_avx512" > + echo "Building AVX512 w/ VNNI CPU" > + build > + compress_libs > + fi > fi > else > echo "Skipping CPU generation step as requested" > ``` > > Ollama was compiled with `OLLAMA_CPU_TARGET="cpu_avx512" go generate ./...` and now when serving it reports `Dynamic LLM libraries [cpu_avx2 cpu_avx512 cpu_avx cpu]`, despite that, the TPS is kinda low: `188.85 ms per token, 5.30 tokens per second`. > > What's your take? To me it feels like either Gemma doesn't take advantage of AVX512/VNNI or something else is going on with ollama / llama.cpp preventing to speed up inference. > > All tests were done using the same prompt and seed: > > ```shell > # curl http://localhost:11434/api/generate -d '{"model": "gemma:7b", "prompt": "Why is the sky blue?", "stream": false, "options": {"seed": 100}}' > ``` Hello @m4r1k , can you retry with a multimodal model, such as llava or bakllava? I think our conclusion is that VNNI instuctions make a huge difference in processing the image, which is why we see an improvement in multimodal models. But they don't make any difference with text only language models.

GiteaMirror commented 2026-04-28 05:17:20 -05:00

Author

Owner

Copy Link

@m4r1k commented on GitHub (Mar 21, 2024):

Hey @ddpasa, follows some tests with both LLaVA 1.6 7b and BakLLaVA latest version 7b.

In terms of prompts, I asked the ollama's API sample with an image as well as the usual "Why the sky is blue?".

Follows the results for LLaVA:

• V1: 143.73 ms per token, 6.96 tokens per second for the question (generation eval time), and 59850.81 ms per token, 0.02 tokens per second for the image recognition (prompt eval time)
• V4: 140.75 ms per token, 7.10 tokens per second for the question (generation eval time), and 59161.23 ms per token, 0.02 tokens per second for the image recognition (prompt eval time)

Follows the results for BakLLaVA:

• V1: 143.05 ms per token, 6.99 tokens per second for the question (generation eval time), and 60477.51 ms per token, 0.02 tokens per second for the image recognition (prompt eval time)
• V4: 139.94 ms per token, 7.15 tokens per second for the question (generation eval time), and 59392.86 ms per token, 0.02 tokens per second for the image recognition (prompt eval time)

Common aspects:

• V1 system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
• V4 system info: AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
• System and tags same as former post
• Stream disabled
• Seed 100
• In the v4 case I did not apply my customer patches

Perhaps I'm making some sort of mistakes but I cannot find large differences..

<!-- gh-comment-id:2013087742 --> @m4r1k commented on GitHub (Mar 21, 2024): Hey @ddpasa, follows some tests with both LLaVA 1.6 7b and BakLLaVA latest version 7b. In terms of prompts, I asked the [ollama's API sample](https://ollama.com/library/llava#:~:text=of%20the%20picture.-,API%20Usage,-curl%20http%3A//localhost) with an image as well as the usual "Why the sky is blue?". Follows the results for LLaVA: - **V1**: `143.73 ms per token, 6.96 tokens per second` for the question (generation eval time), and `59850.81 ms per token, 0.02 tokens per second` for the image recognition (prompt eval time) - **V4**: `140.75 ms per token, 7.10 tokens per second` for the question (generation eval time), and `59161.23 ms per token, 0.02 tokens per second` for the image recognition (prompt eval time) Follows the results for BakLLaVA: - **V1**: `143.05 ms per token, 6.99 tokens per second` for the question (generation eval time), and `60477.51 ms per token, 0.02 tokens per second` for the image recognition (prompt eval time) - **V4**: `139.94 ms per token, 7.15 tokens per second` for the question (generation eval time), and `59392.86 ms per token, 0.02 tokens per second` for the image recognition (prompt eval time) Common aspects: - V1 system info: `AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` - V4 system info: `AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |` - System and tags same as former post - Stream disabled - Seed 100 - In the v4 case I did **not** apply my customer patches Perhaps I'm making some sort of mistakes but I cannot find large differences..

GiteaMirror commented 2026-04-28 05:17:21 -05:00

Author

Owner

Copy Link

@MarkWard0110 commented on GitHub (Apr 21, 2024):

If you find little or no difference when testing, please check if your CPU is throttling due to a temperature that is too high. AVX instructions increase the CPU's temperature. There is even a BIOS setting to reduce CPU frequency when AVX instructions are used. The motherboard may have this off, but it might be throttling the CPU if the cooler can't handle the load.

With the release build of Ollama I have been struggling with keeping the CPU cool.

With just handling thermal.
Multimodel mixtral:8x7b goes from ~10.7 TPS to ~13.7 TPS (generation)
command-r:35b goes from ~4.74 TPS to ~6.00 TPS (generation)

<!-- gh-comment-id:2068043012 --> @MarkWard0110 commented on GitHub (Apr 21, 2024): If you find little or no difference when testing, please check if your CPU is throttling due to a temperature that is too high. AVX instructions increase the CPU's temperature. There is even a BIOS setting to reduce CPU frequency when AVX instructions are used. The motherboard may have this off, but it might be throttling the CPU if the cooler can't handle the load. With the release build of Ollama I have been struggling with keeping the CPU cool. With just handling thermal. Multimodel mixtral:8x7b goes from ~10.7 TPS to ~13.7 TPS (generation) command-r:35b goes from ~4.74 TPS to ~6.00 TPS (generation)

GiteaMirror commented 2026-04-28 05:17:22 -05:00

Author

Owner

Copy Link

@MarkWard0110 commented on GitHub (Apr 30, 2024):

Does the CPU instruction flags like OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_F16C=on -DLLAMA_FMA=on" have an affect on the libraries it is compiled with like BLAS and LAMMAFILE?

When a model is split between a GPU and CPU what build is it running to execute the model? Is the GPU build the runner that is executing both the GPU and CPU or does the CPU offloaded layers execute with one of the dedicated CPU builds?

<!-- gh-comment-id:2085532781 --> @MarkWard0110 commented on GitHub (Apr 30, 2024): Does the CPU instruction flags like `OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_F16C=on -DLLAMA_FMA=on"` have an affect on the libraries it is compiled with like BLAS and LAMMAFILE? When a model is split between a GPU and CPU what build is it running to execute the model? Is the GPU build the runner that is executing both the GPU and CPU or does the CPU offloaded layers execute with one of the dedicated CPU builds?

GiteaMirror commented 2026-04-28 05:17:28 -05:00

Author

Owner

Copy Link

@MaciejMogilany commented on GitHub (May 10, 2024):

AMD

Might be able to help with regard to both AMD AVX512 and the no improvement over AVX2 on some systems. Lets start with AMD because I've been eager to have someone try this on some AMD hardware. I moved the longer boring bits to the end, feel free to skip if you want to just give it a try.

For improved AMD cpu perf (using llama.cpp int his case as its blas/lapack friendly, for now) on inference you can try https://www.amd.com/en/developer/aocl.html#downloads either the package or the tar ball. Unsure about where the package will place them but the install.sh updates the libs and other env vars if on linux. Then in the root of a checkout of the llama.cpp git run (if previouslly used, make clean or otherwise remove CMakeCache.txt and CMakeFiles first)

cmake . -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON

if it doesn't find the libs add

-DBLAS_LIBRARIES=/lib/libblis-mt.so

on win replace

libblis-mt.so with LibBlis-Win-MT-dll.dll

Going back to why AMD chips tend to not end up utilizing their AVX instruction sets most often its due to Intel MKL which bundles a bunch of kernels in compiled code then leverages the CPUID instruction to pick the "best" one. For Intel branded chips this more or less works nicely. For none they get the "most compatible" kernel, read "basic" "slow" . MKL Obviously isn't the only way for something to leverage the instruction sets it's one of the easiest. This is especially true if underlying components wrap around BLAS/LAPACK . Much of llamacpp does, however 'main' no longer does but 'server' does. Likewise 'benchmark' which should make it easy to figure out if we're on the right track.

Enter 'AMD Optimizing CPU Libraries' aka AOCL.

https://www.amd.com/en/developer/aocl.html

There's quite a few but for improving cpu only and proving a benefit to the AMD AVX512 instructions we're gonna need an optimized routine to do matrix-matrix multiplication and other ops. A BLAS/BLIS implementation will generally include this and AMD's AOCL-BLAS has what should be a compliant a/z/s/gemm routine.

https://www.amd.com/en/developer/aocl/dense.html

Hope that helps! I'm very eager to see if AOCL delivered anything meaningful :)

Only difference is that AOCL-mt use all cores in preprocessing phase 8 real ones and 8 HT cores, official ollama build use only 8 cores. AOCL + AVX512 give slower prompt ingestion

In test from phoronix AVX512 gives 2x improvements on OpenVINO but not there. I think that AVX512 are not used in 4 bit qantization so at the bottom of my test is phi3 at fp16, still noting

https://www.phoronix.com/review/amd-zen4-avx512/5

run on mini pc AMD Ryzen 7 7840HS

inxi -F
System:
Host: dev Kernel: 6.8.9-arch1-2 arch: x86_64 bits: 64
Desktop: GNOME v: 46.1 Distro: EndeavourOS
Machine:
Type: Mini-pc System: Micro (HK) Tech product: Venus series v: 1.0
serial:
Mobo: Shenzhen Meigao Equipment model: F7BSD v: 1.1
serial: UEFI: American Megatrends LLC. v: 1.05
date: 12/15/2023
CPU:
Info: 8-core model: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics bits: 64
type: MT MCP cache: L2: 8 MiB
Speed (MHz): avg: 1460 min/max: 400/5293:5137:5764:6080:5449:5924:5608
cores: 1: 3763 2: 400 3: 400 4: 400 5: 3551 6: 3333 7: 400 8: 400 9: 4557
10: 400 11: 3767 12: 400 13: 400 14: 400 15: 400 16: 400
Graphics:
Device-1: AMD Phoenix1 driver: amdgpu v: kernel
Display: wayland server: X.Org v: 23.2.6 with: Xwayland v: 23.2.6
compositor: gnome-shell driver: X: loaded: amdgpu
unloaded: modesetting,radeon dri: radeonsi gpu: amdgpu
resolution: 3440x1440~75Hz
API: EGL v: 1.5 drivers: radeonsi,swrast
platforms: wayland,x11,surfaceless,device
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.0.6-arch1.2
renderer: AMD Radeon Graphics (radeonsi gfx1103_r1 LLVM 17.0.6 DRM 3.57
6.8.9-arch1-2)
API: Vulkan v: 1.3.279 drivers: amd surfaces: xcb,xlib,wayland

...

Drives:
Local Storage: total: 931.51 GiB used: 547.62 GiB (58.8%)
ID-1: /dev/nvme0n1 vendor: Smart Modular Tech. model: SHPP41-1000GM
size: 931.51 GiB

...

Info:
Memory: total: 96 GiB note: est. available: 93.1 GiB used: 6.42 GiB (6.9%)
Processes: 407 Uptime: 36m Shell: Bash inxi: 3.3.34

TESTS

---

ollama version is 0.1.34
ollama run llama3:70b-instruct --verbose

article sumarization

total duration: 11m33.722657067s
load duration: 803.251µs
prompt eval count: 1521 token(s)
prompt eval duration: 6m45.393753s
prompt eval rate: 3.75 tokens/s
eval count: 371 token(s)
eval duration: 4m48.219751s
eval rate: 1.29 tokens/s

ollama from git repo date of 08.05.2024
./ollama run llama3:70b-instruct --verbose

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "

article sumarization

total duration: 12m1.363525552s
load duration: 844.333µs
prompt eval count: 1517 token(s)
prompt eval duration: 8m24.139798s
prompt eval rate: 3.01 tokens/s
eval count: 281 token(s)
eval duration: 3m37.077587s
eval rate: 1.29 tokens/s

---

ollama version is 0.1.34
ollama run phi3 --verbose

article sumarization

total duration: 1m5.838295646s
load duration: 1.177418ms
prompt eval count: 2004 token(s)
prompt eval duration: 35.807216s
prompt eval rate: 55.97 tokens/s
eval count: 413 token(s)
eval duration: 29.933218s
eval rate: 13.80 tokens/s

ollama from git repo date of 08.05.2024
./ollama run phi3 --verbose

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "

article sumarization

total duration: 1m32.598432218s
load duration: 874.609µs
prompt eval count: 2004 token(s)
prompt eval duration: 48.579079s
prompt eval rate: 41.25 tokens/s
eval count: 616 token(s)
eval duration: 43.917537s
eval rate: 14.03 tokens/s

---

ollama version is 0.1.34
ollama run llava-llama3:latest --verbose

screnshoot description

total duration: 44.579711139s
load duration: 5.358803ms
prompt eval duration: 19.443385s
prompt eval rate: 0.00 tokens/s
eval count: 246 token(s)
eval duration: 24.950062s
eval rate: 9.86 tokens/s

ollama from git repo date of 08.05.2024
./ollama run llava-llama3:latest --verbose

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "

screnshoot description

total duration: 32.495976862s
load duration: 5.25242ms
prompt eval duration: 31.126234s
prompt eval rate: 0.00 tokens/s
eval count: 13 token(s)
eval duration: 1.186779s
eval rate: 10.95 tokens/s

---

ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose

article sumarize

total duration: 3m35.789563621s
load duration: 925.737µs
prompt eval count: 1999 token(s)
prompt eval duration: 37.907926s
prompt eval rate: 52.73 tokens/s
eval count: 1107 token(s)
eval duration: 2m57.780921s
eval rate: 6.23 tokens/s

ollama from git repo date of 08.05.2024
./ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "

article sumarize

total duration: 2m11.17614835s
load duration: 871.753µs
prompt eval count: 2004 token(s)
prompt eval duration: 49.166916s
prompt eval rate: 40.76 tokens/s
eval count: 504 token(s)
eval duration: 1m21.911449s
eval rate: 6.15 tokens/s

ollama from git repo date of 08.05.2024 compiled with intel MKL library
./ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose

OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DLLAMA_NATIVE=ON -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on"

system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

article sumarize

total duration: 1m11.621474743s
load duration: 1.074064ms
prompt eval count: 1287 token(s)
prompt eval duration: 36.504465s
prompt eval rate: 35.26 tokens/s
eval count: 224 token(s)
eval duration: 34.985369s
eval rate: 6.40 tokens/s

<!-- gh-comment-id:2104108733 --> @MaciejMogilany commented on GitHub (May 10, 2024): > > AMD > > Might be able to help with regard to both AMD AVX512 and the no improvement over AVX2 on some systems. Lets start with AMD because I've been eager to have someone try this on some AMD hardware. I moved the longer boring bits to the end, feel free to skip if you want to just give it a try. > > For improved AMD cpu perf (using llama.cpp int his case as its blas/lapack friendly, for now) on inference you can try https://www.amd.com/en/developer/aocl.html#downloads either the package or the tar ball. Unsure about where the package will place them but the install.sh updates the libs and other env vars if on linux. Then in the root of a checkout of the llama.cpp git run (if previouslly used, make clean or otherwise remove CMakeCache.txt and CMakeFiles first) > > cmake . -DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON > > if it doesn't find the libs add > > -DBLAS_LIBRARIES=/lib/libblis-mt.so > > on win replace > > libblis-mt.so with LibBlis-Win-MT-dll.dll > > Going back to why AMD chips tend to not end up utilizing their AVX instruction sets most often its due to Intel MKL which bundles a bunch of kernels in compiled code then leverages the CPUID instruction to pick the "best" one. For Intel branded chips this more or less works nicely. For none they get the "most compatible" kernel, read "basic" "slow" . MKL Obviously isn't the only way for something to leverage the instruction sets it's one of the easiest. This is especially true if underlying components wrap around BLAS/LAPACK . Much of llamacpp does, however 'main' no longer does but 'server' does. Likewise 'benchmark' which should make it easy to figure out if we're on the right track. > > Enter 'AMD Optimizing CPU Libraries' aka AOCL. > > https://www.amd.com/en/developer/aocl.html > > There's quite a few but for improving cpu only and proving a benefit to the AMD AVX512 instructions we're gonna need an optimized routine to do matrix-matrix multiplication and other ops. A BLAS/BLIS implementation will generally include this and AMD's AOCL-BLAS has what should be a compliant a/z/s/gemm routine. > > https://www.amd.com/en/developer/aocl/dense.html > > Hope that helps! I'm very eager to see if AOCL delivered anything meaningful :) Only difference is that AOCL-mt use all cores in preprocessing phase 8 real ones and 8 HT cores, official ollama build use only 8 cores. AOCL + AVX512 give slower prompt ingestion In test from phoronix AVX512 gives 2x improvements on OpenVINO but not there. I think that AVX512 are not used in 4 bit qantization so at the bottom of my test is phi3 at fp16, still noting https://www.phoronix.com/review/amd-zen4-avx512/5 run on mini pc AMD Ryzen 7 7840HS inxi -F System: Host: dev Kernel: 6.8.9-arch1-2 arch: x86_64 bits: 64 Desktop: GNOME v: 46.1 Distro: EndeavourOS Machine: Type: Mini-pc System: Micro (HK) Tech product: Venus series v: 1.0 serial: <superuser required> Mobo: Shenzhen Meigao Equipment model: F7BSD v: 1.1 serial: <superuser required> UEFI: American Megatrends LLC. v: 1.05 date: 12/15/2023 CPU: Info: 8-core model: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics bits: 64 type: MT MCP cache: L2: 8 MiB Speed (MHz): avg: 1460 min/max: 400/5293:5137:5764:6080:5449:5924:5608 cores: 1: 3763 2: 400 3: 400 4: 400 5: 3551 6: 3333 7: 400 8: 400 9: 4557 10: 400 11: 3767 12: 400 13: 400 14: 400 15: 400 16: 400 Graphics: Device-1: AMD Phoenix1 driver: amdgpu v: kernel Display: wayland server: X.Org v: 23.2.6 with: Xwayland v: 23.2.6 compositor: gnome-shell driver: X: loaded: amdgpu unloaded: modesetting,radeon dri: radeonsi gpu: amdgpu resolution: 3440x1440~75Hz API: EGL v: 1.5 drivers: radeonsi,swrast platforms: wayland,x11,surfaceless,device API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.0.6-arch1.2 renderer: AMD Radeon Graphics (radeonsi gfx1103_r1 LLVM 17.0.6 DRM 3.57 6.8.9-arch1-2) API: Vulkan v: 1.3.279 drivers: amd surfaces: xcb,xlib,wayland ... Drives: Local Storage: total: 931.51 GiB used: 547.62 GiB (58.8%) ID-1: /dev/nvme0n1 vendor: Smart Modular Tech. model: SHPP41-1000GM size: 931.51 GiB ... Info: Memory: total: 96 GiB note: est. available: 93.1 GiB used: 6.42 GiB (6.9%) Processes: 407 Uptime: 36m Shell: Bash inxi: 3.3.34 TESTS ----------------------------------------------- ollama version is 0.1.34 ollama run llama3:70b-instruct --verbose article sumarization total duration: 11m33.722657067s load duration: 803.251µs prompt eval count: 1521 token(s) prompt eval duration: 6m45.393753s prompt eval rate: 3.75 tokens/s eval count: 371 token(s) eval duration: 4m48.219751s eval rate: 1.29 tokens/s ollama from git repo date of 08.05.2024 ./ollama run llama3:70b-instruct --verbose OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" "system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " article sumarization total duration: 12m1.363525552s load duration: 844.333µs prompt eval count: 1517 token(s) prompt eval duration: 8m24.139798s prompt eval rate: 3.01 tokens/s eval count: 281 token(s) eval duration: 3m37.077587s eval rate: 1.29 tokens/s ---------------------------------------------------- ollama version is 0.1.34 ollama run phi3 --verbose article sumarization total duration: 1m5.838295646s load duration: 1.177418ms prompt eval count: 2004 token(s) prompt eval duration: 35.807216s prompt eval rate: 55.97 tokens/s eval count: 413 token(s) eval duration: 29.933218s eval rate: 13.80 tokens/s ollama from git repo date of 08.05.2024 ./ollama run phi3 --verbose OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" "system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " article sumarization total duration: 1m32.598432218s load duration: 874.609µs prompt eval count: 2004 token(s) prompt eval duration: 48.579079s prompt eval rate: 41.25 tokens/s eval count: 616 token(s) eval duration: 43.917537s eval rate: 14.03 tokens/s ------------------------------------------------------ ollama version is 0.1.34 ollama run llava-llama3:latest --verbose screnshoot description total duration: 44.579711139s load duration: 5.358803ms prompt eval duration: 19.443385s prompt eval rate: 0.00 tokens/s eval count: 246 token(s) eval duration: 24.950062s eval rate: 9.86 tokens/s ollama from git repo date of 08.05.2024 ./ollama run llava-llama3:latest --verbose OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" "system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " screnshoot description total duration: 32.495976862s load duration: 5.25242ms prompt eval duration: 31.126234s prompt eval rate: 0.00 tokens/s eval count: 13 token(s) eval duration: 1.186779s eval rate: 10.95 tokens/s ----------------------------------------------------------------- ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose article sumarize total duration: 3m35.789563621s load duration: 925.737µs prompt eval count: 1999 token(s) prompt eval duration: 37.907926s prompt eval rate: 52.73 tokens/s eval count: 1107 token(s) eval duration: 2m57.780921s eval rate: 6.23 tokens/s ollama from git repo date of 08.05.2024 ./ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS_VENDOR=AOCL-mt -DLLAMA_BLAS=ON -DBLAS_LIBRARIES=/usr/lib/libblis-mt.so -DBLAS_INCLUDE_DIRS=/usr/include/blis -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" "system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " article sumarize total duration: 2m11.17614835s load duration: 871.753µs prompt eval count: 2004 token(s) prompt eval duration: 49.166916s prompt eval rate: 40.76 tokens/s eval count: 504 token(s) eval duration: 1m21.911449s eval rate: 6.15 tokens/s ollama from git repo date of 08.05.2024 compiled with intel MKL library ./ollama run phi3:3.8b-mini-instruct-4k-fp16 --verbose OLLAMA_CUSTOM_CPU_DEFS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DLLAMA_NATIVE=ON -DLLAMA_AVX=on -DLLAMA_AVX2=on -DLLAMA_AVX512=on -DLLAMA_FMA=on -DLLAMA_AVX512_VBMI=on -DLLAMA_AVX512_VNNI=on" system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | article sumarize total duration: 1m11.621474743s load duration: 1.074064ms prompt eval count: 1287 token(s) prompt eval duration: 36.504465s prompt eval rate: 35.26 tokens/s eval count: 224 token(s) eval duration: 34.985369s eval rate: 6.40 tokens/s

GiteaMirror commented 2026-04-28 05:17:36 -05:00

Author

Owner

Copy Link

@ddpasa commented on GitHub (Dec 26, 2024):

Does this mean that building ollama from source will automatically enable avx512_vnni if it is available? vnni makes a huge difference on visual models.

<!-- gh-comment-id:2562798654 --> @ddpasa commented on GitHub (Dec 26, 2024): Does this mean that building ollama from source will automatically enable avx512_vnni if it is available? vnni makes a huge difference on visual models.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

v0.23.1

v0.23.1-rc0

v0.23.0

v0.23.0-rc0

v0.22.1

v0.22.1-rc1

v0.22.1-rc0

v0.22.0

v0.22.0-rc1

v0.21.3-rc0

v0.21.2-rc1

v0.21.2

v0.21.2-rc0

v0.21.1

v0.21.1-rc1

v0.21.1-rc0

v0.21.0

v0.21.0-rc1

v0.21.0-rc0

v0.20.8-rc0

v0.20.7

v0.20.7-rc1

v0.20.7-rc0

v0.20.6

v0.20.6-rc1

v0.20.6-rc0

v0.20.5

v0.20.5-rc2

v0.20.5-rc1

v0.20.5-rc0

v0.20.4

v0.20.4-rc2

v0.20.4-rc1

v0.20.4-rc0

v0.20.3

v0.20.3-rc0

v0.20.2

v0.20.1

v0.20.1-rc2

v0.20.1-rc1

v0.20.1-rc0

v0.20.0

v0.20.0-rc1

v0.20.0-rc0

v0.19.0

v0.19.0-rc2

v0.19.0-rc1

v0.19.0-rc0

v0.18.4-rc1

v0.18.4-rc0

v0.18.3

v0.18.3-rc2

v0.18.3-rc1

v0.18.3-rc0

v0.18.2

v0.18.2-rc1

v0.18.2-rc0

v0.18.1

v0.18.1-rc1

v0.18.1-rc0

v0.18.0

v0.18.0-rc2

v0.18.0-rc1

v0.18.0-rc0

v0.17.8-rc4

v0.17.8-rc3

v0.17.8-rc2

v0.17.8-rc1

v0.17.8-rc0

v0.17.7

v0.17.7-rc2

v0.17.7-rc1

v0.17.7-rc0

v0.17.6

v0.17.5

v0.17.4

v0.17.3

v0.17.2

v0.17.1

v0.17.1-rc2

v0.17.1-rc1

v0.17.1-rc0

v0.17.0

v0.17.0-rc2

v0.17.0-rc1

v0.17.0-rc0

v0.16.3

v0.16.3-rc2

v0.16.3-rc1

v0.16.3-rc0

v0.16.2

v0.16.2-rc0

v0.16.1

v0.16.0

v0.16.0-rc2

v0.16.0-rc0

v0.16.0-rc1

v0.15.6

v0.15.5

v0.15.5-rc5

v0.15.5-rc4

v0.15.5-rc3

v0.15.5-rc2

v0.15.5-rc1

v0.15.5-rc0

v0.15.4

v0.15.3

v0.15.2

v0.15.1

v0.15.1-rc1

v0.15.1-rc0

v0.15.0-rc6

v0.15.0

v0.15.0-rc5

v0.15.0-rc4

v0.15.0-rc3

v0.15.0-rc2

v0.15.0-rc1

v0.15.0-rc0

v0.14.3

v0.14.3-rc3

v0.14.3-rc2

v0.14.3-rc1

v0.14.3-rc0

v0.14.2

v0.14.2-rc1

v0.14.2-rc0

v0.14.1

v0.14.0-rc11

v0.14.0

v0.14.0-rc10

v0.14.0-rc9

v0.14.0-rc8

v0.14.0-rc7

v0.14.0-rc6

v0.14.0-rc5

v0.14.0-rc4

v0.14.0-rc3

v0.14.0-rc2

v0.14.0-rc1

v0.14.0-rc0

v0.13.5

v0.13.5-rc1

v0.13.5-rc0

v0.13.4-rc2

v0.13.4

v0.13.4-rc1

v0.13.4-rc0

v0.13.3

v0.13.3-rc1

v0.13.3-rc0

v0.13.2

v0.13.2-rc2

v0.13.2-rc1

v0.13.2-rc0

v0.13.1

v0.13.1-rc2

v0.13.1-rc1

v0.13.1-rc0

v0.13.0

v0.13.0-rc0

v0.12.11

v0.12.11-rc1

v0.12.11-rc0

v0.12.10

v0.12.10-rc1

v0.12.10-rc0

v0.12.9-rc0

v0.12.9

v0.12.8

v0.12.8-rc0

v0.12.7

v0.12.7-rc1

v0.12.7-rc0

v0.12.7-citest0

v0.12.6

v0.12.6-rc1

v0.12.6-rc0

v0.12.5

v0.12.5-rc0

v0.12.4

v0.12.4-rc7

v0.12.4-rc6

v0.12.4-rc5

v0.12.4-rc4

v0.12.4-rc3

v0.12.4-rc2

v0.12.4-rc1

v0.12.4-rc0

v0.12.3

v0.12.2

v0.12.2-rc0

v0.12.1

v0.12.1-rc1

v0.12.1-rc2

v0.12.1-rc0

v0.12.0

v0.12.0-rc1

v0.12.0-rc0

v0.11.11

v0.11.11-rc3

v0.11.11-rc2

v0.11.11-rc1

v0.11.11-rc0

v0.11.10

v0.11.9

v0.11.9-rc0

v0.11.8

v0.11.8-rc0

v0.11.7-rc1

v0.11.7-rc0

v0.11.7

v0.11.6

v0.11.6-rc0

v0.11.5-rc4

v0.11.5-rc3

v0.11.5

v0.11.5-rc5

v0.11.5-rc2

v0.11.5-rc1

v0.11.5-rc0

v0.11.4

v0.11.4-rc0

v0.11.3

v0.11.3-rc0

v0.11.2

v0.11.1

v0.11.0-rc0

v0.11.0-rc1

v0.11.0-rc2

v0.11.0

v0.10.2-int1

v0.10.1

v0.10.0

v0.10.0-rc4

v0.10.0-rc3

v0.10.0-rc2

v0.10.0-rc1

v0.10.0-rc0

v0.9.7-rc1

v0.9.7-rc0

v0.9.6

v0.9.6-rc0

v0.9.6-ci0

v0.9.5

v0.9.4-rc5

v0.9.4-rc6

v0.9.4

v0.9.4-rc3

v0.9.4-rc4

v0.9.4-rc1

v0.9.4-rc2

v0.9.4-rc0

v0.9.3

v0.9.3-rc5

v0.9.4-citest0

v0.9.3-rc4

v0.9.3-rc3

v0.9.3-rc2

v0.9.3-rc1

v0.9.3-rc0

v0.9.2

v0.9.1

v0.9.1-rc1

v0.9.1-rc0

v0.9.1-ci1

v0.9.1-ci0

v0.9.0

v0.9.0-rc0

v0.8.0

v0.8.0-rc0

v0.7.1-rc2

v0.7.1

v0.7.1-rc1

v0.7.1-rc0

v0.7.0

v0.7.0-rc1

v0.7.0-rc0

v0.6.9-rc0

v0.6.8

v0.6.8-rc0

v0.6.7

v0.6.7-rc2

v0.6.7-rc1

v0.6.7-rc0

v0.6.6

v0.6.6-rc2

v0.6.6-rc1

v0.6.6-rc0

v0.6.5-rc1

v0.6.5

v0.6.5-rc0

v0.6.4-rc0

v0.6.4

v0.6.3-rc1

v0.6.3

v0.6.3-rc0

v0.6.2

v0.6.2-rc0

v0.6.1

v0.6.1-rc0

v0.6.0-rc0

v0.6.0

v0.5.14-rc0

v0.5.13

v0.5.13-rc6

v0.5.13-rc5

v0.5.13-rc4

v0.5.13-rc3

v0.5.13-rc2

v0.5.13-rc1

v0.5.13-rc0

v0.5.12

v0.5.12-rc1

v0.5.12-rc0

v0.5.11

v0.5.10

v0.5.9

v0.5.9-rc0

v0.5.8-rc13

v0.5.8

v0.5.8-rc12

v0.5.8-rc11

v0.5.8-rc10

v0.5.8-rc9

v0.5.8-rc8

v0.5.8-rc7

v0.5.8-rc6

v0.5.8-rc5

v0.5.8-rc4

v0.5.8-rc3

v0.5.8-rc2

v0.5.8-rc1

v0.5.8-rc0

v0.5.7

v0.5.6

v0.5.5

v0.5.5-rc0

v0.5.4

v0.5.3

v0.5.3-rc0

v0.5.2

v0.5.2-rc3

v0.5.2-rc2

v0.5.2-rc1

v0.5.2-rc0

v0.5.1

v0.5.0

v0.5.0-rc1

v0.4.8-rc0

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.3-rc0

v0.4.2

v0.4.2-rc1

v0.4.2-rc0

v0.4.1

v0.4.1-rc0

v0.4.0

v0.4.0-rc8

v0.4.0-rc7

v0.4.0-rc6

v0.4.0-rc5

v0.4.0-rc4

v0.4.0-rc3

v0.4.0-rc2

v0.4.0-rc1

v0.4.0-rc0

v0.4.0-ci3

v0.3.14

v0.3.14-rc0

v0.3.13

v0.3.12

v0.3.12-rc5

v0.3.12-rc4

v0.3.12-rc3

v0.3.12-rc2

v0.3.12-rc1

v0.3.11

v0.3.11-rc4

v0.3.11-rc3

v0.3.11-rc2

v0.3.11-rc1

v0.3.10

v0.3.10-rc1

v0.3.9

v0.3.8

v0.3.7

v0.3.7-rc6

v0.3.7-rc5

v0.3.7-rc4

v0.3.7-rc3

v0.3.7-rc2

v0.3.7-rc1

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.8

v0.2.8-rc2

v0.2.8-rc1

v0.2.7

v0.2.6

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.2-rc2

v0.2.2-rc1

v0.2.1

v0.2.0

v0.1.49-rc14

v0.1.49-rc13

v0.1.49-rc12

v0.1.49-rc11

v0.1.49-rc10

v0.1.49-rc9

v0.1.49-rc8

v0.1.49-rc7

v0.1.49-rc6

v0.1.49-rc4

v0.1.49-rc5

v0.1.49-rc3

v0.1.49-rc2

v0.1.49-rc1

v0.1.48

v0.1.47

v0.1.46

v0.1.45-rc5

v0.1.45

v0.1.45-rc4

v0.1.45-rc3

v0.1.45-rc2

v0.1.45-rc1

v0.1.44

v0.1.43

v0.1.42

v0.1.41

v0.1.40

v0.1.40-rc1

v0.1.39

v0.1.39-rc2

v0.1.39-rc1

v0.1.38

v0.1.37

v0.1.36

v0.1.35

v0.1.35-rc1

v0.1.34

v0.1.34-rc1

v0.1.33

v0.1.33-rc7

v0.1.33-rc6

v0.1.33-rc5

v0.1.33-rc4

v0.1.33-rc3

v0.1.33-rc2

v0.1.33-rc1

v0.1.32

v0.1.32-rc2

v0.1.32-rc1

v0.1.31

v0.1.30

v0.1.29

v0.1.28

v0.1.27

v0.1.26

v0.1.25

v0.1.24

v0.1.23

v0.1.22

v0.1.21

v0.1.20

v0.1.19

v0.1.18

v0.1.17

v0.1.16

v0.1.15

v0.1.14

v0.1.13

v0.1.12

v0.1.11

v0.1.10

v0.1.9

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

amd

api

app

bug

build

cli

cloud

compatibility

context-length

create

docker

documentation

embeddings

feature request

feedback wanted

good first issue

gpt-oss

gpu

harmony

help wanted

image

install

intel

js

launch

linux

macos

memory

mlx

model

needs more info

networking

nvidia

ollama.com

performance

pull-request

Mirrored from GitHub Pull Request

python

question

registry

rendering

thinking

tools

top

vulkan

windows

wsl

No Label build feature request

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#47773