github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #15012] flash streaming #56158

New Issue
Open
opened 2026-04-29 10:19:52 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-29 10:19:52 -05:00
Owner
Copy Link

Originally created by @sbe-arg on GitHub (Mar 22, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15012

if anyone is keen on experimenting with mlx-flash and flash-moe to bring something like this to ollama

https://www.reddit.com/r/LocalLLaMA/comments/1s0mqto/has_anyone_tried_this_flashmoe_running_a_397b/

Originally created by @sbe-arg on GitHub (Mar 22, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15012 if anyone is keen on experimenting with mlx-flash and flash-moe to bring something like this to ollama https://www.reddit.com/r/LocalLLaMA/comments/1s0mqto/has_anyone_tried_this_flashmoe_running_a_397b/
GiteaMirror added the feature request label 2026-04-29 10:19:52 -05:00
GiteaMirror commented 2026-04-29 10:19:54 -05:00
Author
Owner
Copy Link

@AndrewFarley commented on GitHub (Mar 23, 2026):

10/10 would love if someone could bring this to Ollama. Please yeah, would allow for massive model usage. I've tested this myself, and it seems to work, can run a 400B model on a 32GB mac laptop with 5-6 tok/sec. While that isn't insane performance, it's basically "free" to run a massive model on an affordable piece of hardware without requiring an huge cost to build an massive multi-GPU rig. I'd love if someone can make this possible via Ollama, so we can make MoE models more accessible on the mac. So, while it's "slow" by some consideration, the speed is irrelevant when compared to the size of the models possible and the cost (and security benefits) of being able to run massive models offline/locally.

If you read/review the example repo: https://github.com/danveloper/flash-moe there's a little bit of post-processing you need to do to make it work from the downloaded models from huggingface to make this work. But, I assume that process can be done in some post-download/pre-runtime job in Ollama to prep for running this via this Flash MoE method. If anyone tries to make this work, I'd be happy to try this out. I'm subscribed to this issue, ping me. I have a handful of different Macs at my disposal, from M1's up to M4's of various RAM sizes from 32-128GB and would happily beta test.

<!-- gh-comment-id:4113430004 --> @AndrewFarley commented on GitHub (Mar 23, 2026): 10/10 would love if someone could bring this to Ollama. Please yeah, would allow for massive model usage. I've tested this myself, and it seems to work, can run a 400B model on a 32GB mac laptop with 5-6 tok/sec. While that isn't insane performance, it's basically "free" to run a massive model on an affordable piece of hardware without requiring an huge cost to build an massive multi-GPU rig. I'd love if someone can make this possible via Ollama, so we can make MoE models more accessible on the mac. So, while it's "slow" by some consideration, the speed is irrelevant when compared to the size of the models possible and the cost (and security benefits) of being able to run massive models offline/locally. If you read/review the example repo: https://github.com/danveloper/flash-moe there's a little bit of post-processing you need to do to make it work from the downloaded models from huggingface to make this work. But, I assume that process can be done in some post-download/pre-runtime job in Ollama to prep for running this via this Flash MoE method. If anyone tries to make this work, I'd be happy to try this out. I'm subscribed to this issue, ping me. I have a handful of different Macs at my disposal, from M1's up to M4's of various RAM sizes from 32-128GB and would happily beta test.
GiteaMirror commented 2026-04-29 10:19:55 -05:00
Author
Owner
Copy Link

@pragun-ananda commented on GitHub (Mar 24, 2026):

This is super cool, I'll try working on it!

<!-- gh-comment-id:4115721708 --> @pragun-ananda commented on GitHub (Mar 24, 2026): This is super cool, I'll try working on it!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#56158
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?