github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9439] can ollma support on-chip gpu acceleration on raspberry pi series devices #6154

New Issue
Closed
opened 2026-04-12 17:30:13 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-12 17:30:13 -05:00
Owner
Copy Link

Originally created by @yuezhongtao on GitHub (Mar 1, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9439

Currently, ollama can only run in CPU mode on the Raspberry Pi. However, many edge devices similar to the Raspberry Pi do not have independent accelerator cards, but there is a need to run slightly smaller models, such as deepseek-r1:1.b. I feel that ollama can be developed based on the Raspberry Pi series of devices that sell a lot to support the arm on-chip GPU acceleration feature.

Originally created by @yuezhongtao on GitHub (Mar 1, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9439 Currently, ollama can only run in CPU mode on the Raspberry Pi. However, many edge devices similar to the Raspberry Pi do not have independent accelerator cards, but there is a need to run slightly smaller models, such as deepseek-r1:1.b. I feel that ollama can be developed based on the Raspberry Pi series of devices that sell a lot to support the arm on-chip GPU acceleration feature.
GiteaMirror added the feature request label 2026-04-12 17:30:13 -05:00
GiteaMirror commented 2026-04-12 17:30:16 -05:00
Author
Owner
Copy Link

@flywiththetide commented on GitHub (Mar 4, 2025):

Currently, Ollama runs in CPU mode only on Raspberry Pi devices because:

  1. The Raspberry Pi's onboard GPU (Broadcom VideoCore) does not support CUDA, ROCm, or Vulkan compute, which are required for acceleration.
  2. Ollama's GPU acceleration is optimized for NVIDIA (CUDA) and AMD (ROCm) GPUs, which are not present on Raspberry Pi.

Potential Workarounds:

1. Use OpenCL for GPU Acceleration

While the VideoCore VI GPU on Raspberry Pi does not natively support Ollama's backend, some ML models can run on OpenCL. You can try:

sudo apt install mesa-opencl-icd ocl-icd-libopencl1

Then, test OpenCL support with:

clinfo

However, OpenCL performance is limited on Raspberry Pi.

2. Consider Using an External USB Accelerator

For better inference speed, you can use:

  • Google Coral TPU (edgetpu)
  • Intel Movidius Neural Compute Stick (ncs2)
  • Raspberry Pi AI Kit (Hailo-8 or Myriad-X based accelerators)`

These devices support TensorFlow Lite and ONNX Runtime, which may be integrated with Ollama in the future.

3. Optimize Ollama for ARM CPU Execution

  • Use 4-bit or 8-bit quantized models to reduce CPU load: 
    ollama pull llama3-8b-q4
  • Overclock your Raspberry Pi to increase CPU performance.

Future Possibilities

If the Raspberry Pi Foundation adds Vulkan compute or OpenCL 3.0 support in future versions, GPU acceleration may become more feasible for Ollama.

Would you like help testing an OpenCL or Vulkan-based workaround for your Raspberry Pi model?

<!-- gh-comment-id:2696243010 --> @flywiththetide commented on GitHub (Mar 4, 2025): Currently, Ollama runs in **CPU mode only** on Raspberry Pi devices because: 1. The **Raspberry Pi's onboard GPU (Broadcom VideoCore)** does not support CUDA, ROCm, or Vulkan compute, which are required for acceleration. 2. Ollama's GPU acceleration is optimized for **NVIDIA (CUDA) and AMD (ROCm) GPUs**, which are not present on Raspberry Pi. ### **Potential Workarounds:** #### **1. Use OpenCL for GPU Acceleration** While the **VideoCore VI GPU** on Raspberry Pi does not natively support Ollama's backend, some ML models can run on OpenCL. You can try: ```bash sudo apt install mesa-opencl-icd ocl-icd-libopencl1 ``` Then, test OpenCL support with: ```bash clinfo ``` However, OpenCL performance is limited on Raspberry Pi. #### **2. Consider Using an External USB Accelerator** For better inference speed, you can use: - **Google Coral TPU** (`edgetpu`) - **Intel Movidius Neural Compute Stick** (`ncs2`) - **Raspberry Pi AI Kit (Hailo-8 or Myriad-X based accelerators)`** These devices support **TensorFlow Lite** and **ONNX Runtime**, which may be integrated with Ollama in the future. #### **3. Optimize Ollama for ARM CPU Execution** - Use **4-bit or 8-bit quantized models** to reduce CPU load: ```bash ollama pull llama3-8b-q4 ``` - Overclock your Raspberry Pi to **increase CPU performance**. ### **Future Possibilities** If the Raspberry Pi Foundation adds **Vulkan compute or OpenCL 3.0 support** in future versions, GPU acceleration may become more feasible for Ollama. Would you like help testing an OpenCL or Vulkan-based workaround for your Raspberry Pi model?
GiteaMirror commented 2026-04-12 17:30:16 -05:00
Author
Owner
Copy Link

@flywiththetide commented on GitHub (Mar 4, 2025):

Currently, Ollama is designed to work with auto-regressive transformer models like LLaMA, Mistral, and Phi-series. Diffusion-based LLMs (dLLMs), such as LLaDa, require a different processing pipeline.

Challenges of Supporting Diffusion LLMs in Ollama

  1. Fundamentally Different Architecture

    • Diffusion models generate sequences differently from transformer-based LLMs.
    • Ollama's inference engine is optimized for auto-regressive token generation, not the iterative denoising steps of diffusion.

  2. Performance Considerations

    • Diffusion-based LLMs require multiple sampling steps per token, leading to higher computational overhead.
    • Without optimizations like latent-space diffusion, the performance might not be ideal for local inference.

Potential Paths for Future Support

  • Evaluate integration with frameworks like Hugging Face Diffusers to test feasibility.
  • Enable model adapter layers to allow hybrid inference between auto-regressive and diffusion-based models.
  • Community-driven experiments: If there’s strong demand, it might be possible to adapt Ollama’s backend for dLLMs.

For now, running diffusion-based LLMs may require separate inference setups, but it would be interesting to see how this evolves.

Would you be interested in helping test experimental dLLM models with Ollama?

<!-- gh-comment-id:2696247452 --> @flywiththetide commented on GitHub (Mar 4, 2025): Currently, Ollama is designed to work with **auto-regressive transformer models** like LLaMA, Mistral, and Phi-series. Diffusion-based LLMs (dLLMs), such as **LLaDa**, require a different processing pipeline. ### **Challenges of Supporting Diffusion LLMs in Ollama** 1. **Fundamentally Different Architecture** - Diffusion models generate sequences differently from transformer-based LLMs. - Ollama's inference engine is optimized for auto-regressive token generation, not the iterative denoising steps of diffusion. 2. **Performance Considerations** - Diffusion-based LLMs require **multiple sampling steps per token**, leading to higher computational overhead. - Without optimizations like **latent-space diffusion**, the performance might not be ideal for local inference. ### **Potential Paths for Future Support** - **Evaluate integration with frameworks like Hugging Face Diffusers** to test feasibility. - **Enable model adapter layers** to allow hybrid inference between auto-regressive and diffusion-based models. - **Community-driven experiments**: If there’s strong demand, it might be possible to adapt Ollama’s backend for dLLMs. For now, running diffusion-based LLMs may require **separate inference setups**, but it would be interesting to see how this evolves. Would you be interested in helping test experimental dLLM models with Ollama?
GiteaMirror commented 2026-04-12 17:30:17 -05:00
Author
Owner
Copy Link

@gotyer commented on GitHub (Mar 6, 2025):

   * Diffusion-based LLMs require **multiple sampling steps per token**, leading to higher computational overhead.

Not quite. Each iteration denoises multiple tokens simultaneously.
The higher computational overhead would only apply if we generated a very small amount of tokens, which is of course, not the goal of a LLM. For bigger chunks of texts and not just few words responses, the dLLMs are extremely efficient.

From what we have seen by inceptionlabs, a 50x speed boost is the bare minimum to expect, and faster usually means more energy efficient for real world applications.

Don't forget that we will likely also see many generation parameters that already exist in image diffusion, such as choosing the number steps or different schedulers.

In the context of ollama, I'm convinced that implementing a processing pipeline for dLLMs in parallel of autoregression would greatly benefit the world of LLMs

<!-- gh-comment-id:2703724863 --> @gotyer commented on GitHub (Mar 6, 2025): > * Diffusion-based LLMs require **multiple sampling steps per token**, leading to higher computational overhead. Not quite. Each iteration denoises multiple tokens simultaneously. The higher computational overhead would only apply if we generated a very small amount of tokens, which is of course, not the goal of a LLM. For bigger chunks of texts and not just few words responses, the dLLMs are extremely efficient. From what we have seen by inceptionlabs, a 50x speed boost is the bare minimum to expect, and faster usually means more energy efficient for real world applications. Don't forget that we will likely also see many generation parameters that already exist in image diffusion, such as choosing the number steps or different schedulers. In the context of ollama, I'm convinced that implementing a processing pipeline for dLLMs in parallel of autoregression would greatly benefit the world of LLMs
GiteaMirror commented 2026-04-12 17:30:17 -05:00
Author
Owner
Copy Link

@hackdefendr commented on GitHub (Mar 11, 2025):

What about off-chip GPU/ASIC chips like the Hailo AI chips? Would that be something worth exploring? I have a Pi5 with the AI Kit so I could dump registers, test code, etc.

<!-- gh-comment-id:2715661028 --> @hackdefendr commented on GitHub (Mar 11, 2025): What about off-chip GPU/ASIC chips like the Hailo AI chips? Would that be something worth exploring? I have a Pi5 with the AI Kit so I could dump registers, test code, etc.
GiteaMirror commented 2026-04-12 17:30:17 -05:00
Author
Owner
Copy Link

@yuezhongtao commented on GitHub (Mar 13, 2025):

Currently, Ollama runs in CPU mode only on Raspberry Pi devices because:

  1. The Raspberry Pi's onboard GPU (Broadcom VideoCore) does not support CUDA, ROCm, or Vulkan compute, which are required for acceleration.
  2. Ollama's GPU acceleration is optimized for NVIDIA (CUDA) and AMD (ROCm) GPUs, which are not present on Raspberry Pi.

Potential Workarounds:

1. Use OpenCL for GPU Acceleration

While the VideoCore VI GPU on Raspberry Pi does not natively support Ollama's backend, some ML models can run on OpenCL. You can try:

sudo apt install mesa-opencl-icd ocl-icd-libopencl1
Then, test OpenCL support with:

clinfo
However, OpenCL performance is limited on Raspberry Pi.

2. Consider Using an External USB Accelerator

For better inference speed, you can use:

  • Google Coral TPU (edgetpu)
  • Intel Movidius Neural Compute Stick (ncs2)
  • Raspberry Pi AI Kit (Hailo-8 or Myriad-X based accelerators)`

These devices support TensorFlow Lite and ONNX Runtime, which may be integrated with Ollama in the future.

3. Optimize Ollama for ARM CPU Execution

  • Use 4-bit or 8-bit quantized models to reduce CPU load:
    ollama pull llama3-8b-q4
  • Overclock your Raspberry Pi to increase CPU performance.

Future Possibilities

If the Raspberry Pi Foundation adds Vulkan compute or OpenCL 3.0 support in future versions, GPU acceleration may become more feasible for Ollama.

Would you like help testing an OpenCL or Vulkan-based workaround for your Raspberry Pi model?

Raspberry Pi 5 has support Vulkan， i happen to have a Raspberry Pi 5 and can help test vulkan or opencl

<!-- gh-comment-id:2721085628 --> @yuezhongtao commented on GitHub (Mar 13, 2025): > Currently, Ollama runs in **CPU mode only** on Raspberry Pi devices because: > > 1. The **Raspberry Pi's onboard GPU (Broadcom VideoCore)** does not support CUDA, ROCm, or Vulkan compute, which are required for acceleration. > 2. Ollama's GPU acceleration is optimized for **NVIDIA (CUDA) and AMD (ROCm) GPUs**, which are not present on Raspberry Pi. > > ### **Potential Workarounds:** > #### **1. Use OpenCL for GPU Acceleration** > While the **VideoCore VI GPU** on Raspberry Pi does not natively support Ollama's backend, some ML models can run on OpenCL. You can try: > > sudo apt install mesa-opencl-icd ocl-icd-libopencl1 > Then, test OpenCL support with: > > clinfo > However, OpenCL performance is limited on Raspberry Pi. > > #### **2. Consider Using an External USB Accelerator** > For better inference speed, you can use: > > * **Google Coral TPU** (`edgetpu`) > * **Intel Movidius Neural Compute Stick** (`ncs2`) > * **Raspberry Pi AI Kit (Hailo-8 or Myriad-X based accelerators)`** > > These devices support **TensorFlow Lite** and **ONNX Runtime**, which may be integrated with Ollama in the future. > > #### **3. Optimize Ollama for ARM CPU Execution** > * Use **4-bit or 8-bit quantized models** to reduce CPU load: > ollama pull llama3-8b-q4 > * Overclock your Raspberry Pi to **increase CPU performance**. > > ### **Future Possibilities** > If the Raspberry Pi Foundation adds **Vulkan compute or OpenCL 3.0 support** in future versions, GPU acceleration may become more feasible for Ollama. > > Would you like help testing an OpenCL or Vulkan-based workaround for your Raspberry Pi model? Raspberry Pi 5 has support Vulkan， i happen to have a Raspberry Pi 5 and can help test vulkan or opencl
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6154
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?