github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3898] Apple Silicone Neural Engine: Core ML model package format #48925

New Issue
Open
opened 2026-04-28 10:06:14 -05:00 by GiteaMirror · 24 comments
GiteaMirror commented 2026-04-28 10:06:14 -05:00
Owner
Copy Link

Originally created by @qdrddr on GitHub (Apr 25, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3898

Please consider adding Core ML model package format support to utilize Apple Silicone Nural Engine + GPU.

List of Core ML package format models

https://github.com/likedan/Awesome-CoreML-Models

Originally created by @qdrddr on GitHub (Apr 25, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3898 Please consider adding Core ML model package format support to utilize Apple Silicone Nural Engine + GPU. List of Core ML package format models https://github.com/likedan/Awesome-CoreML-Models
GiteaMirror added the feature request label 2026-04-28 10:06:14 -05:00
GiteaMirror commented 2026-04-28 10:07:02 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Apr 25, 2024):

This is about running LLMs locally on Apple Silicone. Core ML is a framework that can redistribute workload across CPU, GPU & Nural Engine (ANE). ANE is available on all modern Apple Devices: iPhones & Macs (A14 or newer and M1 or newer). Ideally, we want to run LLMs on ANE only as it has optimizations for running ML tasks compared to GPU. Apple claims "deploying your Transformer models on Apple devices with an A14 or newer and M1 or newer chip to achieve up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations".

  1. To utilize Core ML first, you need to convert a model from TensorFlow, PyTorch to Core ML model package format using coremltools (or simply utilize existing models in Core ML package format ).
  2. Second, you must now use that converted package with an implementation designed for Apple Devices. Here is the Apple XCode reference PyTorch implementation.

https://machinelearning.apple.com/research/neural-engine-transformers

<!-- gh-comment-id:2077124195 --> @qdrddr commented on GitHub (Apr 25, 2024): This is about running LLMs locally on Apple Silicone. Core ML is a framework that can redistribute workload across CPU, GPU & Nural Engine (ANE). ANE is available on all modern Apple Devices: iPhones & Macs (A14 or newer and M1 or newer). Ideally, we want to run LLMs on ANE only as it has optimizations for running ML tasks compared to GPU. Apple claims ["deploying your Transformer models on Apple devices with an A14 or newer and M1 or newer chip to achieve up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations"](https://github.com/apple/ml-ane-transformers). 1. To utilize Core ML first, you need to convert a model from TensorFlow, PyTorch to Core ML model package format using [coremltools](https://coremltools.readme.io/docs) (or simply [utilize existing models in Core ML package format](https://github.com/likedan/Awesome-CoreML-Models) ). 2. Second, you must now use that converted package with an implementation designed for Apple Devices. Here is the [Apple XCode reference PyTorch implementation](https://github.com/apple/ml-ane-transformers). https://machinelearning.apple.com/research/neural-engine-transformers
GiteaMirror commented 2026-04-28 10:07:03 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Apr 25, 2024):

Work in progress on CoreML implementation for [whisper.cpp]. They see x3 performance improvements for some models. (https://github.com/ggerganov/whisper.cpp/discussions/548) you might be interested in.
You might also be interested in another implementation Swift Transformers.
Example of CoreML application
https://github.com/huggingface/swift-chat

<!-- gh-comment-id:2077142163 --> @qdrddr commented on GitHub (Apr 25, 2024): Work in progress on CoreML implementation for [whisper.cpp]. They see x3 performance improvements for some models. (https://github.com/ggerganov/whisper.cpp/discussions/548) you might be interested in. You might also be interested in another implementation [Swift Transformers](https://huggingface.co/blog/swift-coreml-llm). Example of CoreML application https://github.com/huggingface/swift-chat
GiteaMirror commented 2026-04-28 10:07:03 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (May 22, 2024):

https://appleinsider.com/articles/24/05/07/secret-apple-project-acdc-to-pioneer-ai-chips-for-data-centers

Under the internal name "Project ACDC," Apple is developing Apple Silicon designed specifically for server farms dedicated to AI processing. The company aims to optimize AI applications within its data centers for future versions of its platforms.

<!-- gh-comment-id:2123709683 --> @qdrddr commented on GitHub (May 22, 2024): https://appleinsider.com/articles/24/05/07/secret-apple-project-acdc-to-pioneer-ai-chips-for-data-centers > Under the internal name "Project ACDC," Apple is developing [Apple Silicon](https://appleinsider.com/inside/apple-silicon) designed specifically for server farms dedicated to AI processing. The company aims to optimize AI applications within its data centers for future versions of its platforms.
GiteaMirror commented 2026-04-28 10:07:04 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Jun 13, 2024):

2024 CoreML Updates
https://developer.apple.com/documentation/updates/coreml

WWDC 2024 sessions about CoreML:
https://developer.apple.com/wwdc24/10160
https://developer.apple.com/wwdc24/10218
https://developer.apple.com/wwdc24/10161

<!-- gh-comment-id:2166612131 --> @qdrddr commented on GitHub (Jun 13, 2024): 2024 CoreML Updates https://developer.apple.com/documentation/updates/coreml WWDC 2024 sessions about CoreML: https://developer.apple.com/wwdc24/10160 https://developer.apple.com/wwdc24/10218 https://developer.apple.com/wwdc24/10161
GiteaMirror commented 2026-04-28 10:07:04 -05:00
Author
Owner
Copy Link

@BrandonEchols04 commented on GitHub (Jul 6, 2024):

Any news on Ollama's usage of Apple Silicon's Neural Engine?

<!-- gh-comment-id:2211979678 --> @BrandonEchols04 commented on GitHub (Jul 6, 2024): Any news on Ollama's usage of Apple Silicon's Neural Engine?
GiteaMirror commented 2026-04-28 10:07:05 -05:00
Author
Owner
Copy Link

@Michael-Z-Freeman commented on GitHub (Sep 14, 2024):

Until this is fixed can anyone suggest a similar project that does utilise Apple Silicon resources ?

<!-- gh-comment-id:2351073057 --> @Michael-Z-Freeman commented on GitHub (Sep 14, 2024): Until this is fixed can anyone suggest a similar project that does utilise Apple Silicon resources ?
GiteaMirror commented 2026-04-28 10:07:06 -05:00
Author
Owner
Copy Link

@hg0428 commented on GitHub (Oct 10, 2024):

This would be a wonderful feature to have as it would allow us to do inference many times faster than what is currently possible.

<!-- gh-comment-id:2405273280 --> @hg0428 commented on GitHub (Oct 10, 2024): This would be a wonderful feature to have as it would allow us to do inference many times faster than what is currently possible.
GiteaMirror commented 2026-04-28 10:07:07 -05:00
Author
Owner
Copy Link

@joselael commented on GitHub (Oct 26, 2024):

Until this is fixed can anyone suggest a similar project that does utilise Apple Silicon resources ?

Other than DrawThings for diffusion models, I'm not aware of any other projects that use CoreML. DrawThings is partially open source, maybe y'all can look into it to see how it's done. Otherwise, llama.cpp supports MLX, I think gregor mentioned that coreml isn't worth it yet or something. Maybe I'll make a PR for coreml when I have a long vacation someday.

<!-- gh-comment-id:2439339567 --> @joselael commented on GitHub (Oct 26, 2024): > Until this is fixed can anyone suggest a similar project that does utilise Apple Silicon resources ? Other than DrawThings for diffusion models, I'm not aware of any other projects that use CoreML. DrawThings is partially open source, maybe y'all can look into it to see how it's done. Otherwise, llama.cpp supports MLX, I think gregor mentioned that coreml isn't worth it yet or something. Maybe I'll make a PR for coreml when I have a long vacation someday.
GiteaMirror commented 2026-04-28 10:07:07 -05:00
Author
Owner
Copy Link

@tmpbci commented on GitHub (Nov 14, 2024):

not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML

<!-- gh-comment-id:2475527063 --> @tmpbci commented on GitHub (Nov 14, 2024): not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML
GiteaMirror commented 2026-04-28 10:07:08 -05:00
Author
Owner
Copy Link

@johnalanwoods commented on GitHub (Nov 24, 2024):

So right now Ollama can't use the ANE right?

<!-- gh-comment-id:2496045073 --> @johnalanwoods commented on GitHub (Nov 24, 2024): So right now Ollama can't use the ANE right?
GiteaMirror commented 2026-04-28 10:07:09 -05:00
Author
Owner
Copy Link

@hg0428 commented on GitHub (Nov 24, 2024):

So right now Ollama can't use the ANE right?

Correct, Ollama cannot currently use the ANE. However, they are working on this in Llama.cpp rn.

<!-- gh-comment-id:2496046349 --> @hg0428 commented on GitHub (Nov 24, 2024): > So right now Ollama can't use the ANE right? Correct, Ollama cannot currently use the ANE. However, they are working on this in Llama.cpp rn.
GiteaMirror commented 2026-04-28 10:07:11 -05:00
Author
Owner
Copy Link

@hg0428 commented on GitHub (Nov 24, 2024):

https://github.com/ggerganov/llama.cpp/issues/10453

<!-- gh-comment-id:2496046619 --> @hg0428 commented on GitHub (Nov 24, 2024): https://github.com/ggerganov/llama.cpp/issues/10453
GiteaMirror commented 2026-04-28 10:07:12 -05:00
Author
Owner
Copy Link

@dalisoft commented on GitHub (Nov 24, 2024):

Using ANE over MLX will be more efficient, lower token/sec but uses a lot less power, less noise and lower battery usage.

There are few examples and/or repositories i'm found:

And as mentioned already above

not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML

<!-- gh-comment-id:2496064476 --> @dalisoft commented on GitHub (Nov 24, 2024): Using ANE over MLX will be more efficient, lower token/sec but uses a lot less power, less noise and lower battery usage. There are few examples and/or repositories i'm found: - https://github.com/smpanaro/coreml-llm-cli - https://github.com/apple/ml-ane-transformers And as mentioned already above > not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML
GiteaMirror commented 2026-04-28 10:07:12 -05:00
Author
Owner
Copy Link

@hg0428 commented on GitHub (Nov 24, 2024):

Using ANE over MLX will be more efficient, lower token/sec but uses a lot less power, less noise and lower battery usage.

There are few examples and/or repositories i'm found:

And as mentioned already above

not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML

If implemented properly, it can give significantly higher token/second.
Also, the implementation being considered by Llama.cpp would not use CoreML.

<!-- gh-comment-id:2496065430 --> @hg0428 commented on GitHub (Nov 24, 2024): > Using ANE over MLX will be more efficient, lower token/sec but uses a lot less power, less noise and lower battery usage. > > There are few examples and/or repositories i'm found: > > * https://github.com/smpanaro/coreml-llm-cli > * https://github.com/apple/ml-ane-transformers > > And as mentioned already above > > > not exactly like ollama but running a model with coreML https://github.com/TheMurusTeam/FreeScaler-CoreML If implemented properly, it can give significantly higher token/second. Also, the implementation being considered by Llama.cpp would not use CoreML.
GiteaMirror commented 2026-04-28 10:07:13 -05:00
Author
Owner
Copy Link

@dalisoft commented on GitHub (Nov 24, 2024):

@hg0428 I am newbie in LLM/AI techs so i'm only happy to see same or higher performance if it is possible. Even 2x lower speed with 10x lower resource usage is good for me

<!-- gh-comment-id:2496066558 --> @dalisoft commented on GitHub (Nov 24, 2024): @hg0428 I am newbie in LLM/AI techs so i'm only happy to see same or higher performance if it is possible. Even 2x lower speed with 10x lower resource usage is good for me
GiteaMirror commented 2026-04-28 10:07:14 -05:00
Author
Owner
Copy Link

@AShortcuts commented on GitHub (Feb 18, 2025):

I was wondering if there were any updates on this yet?

<!-- gh-comment-id:2665842668 --> @AShortcuts commented on GitHub (Feb 18, 2025): I was wondering if there were any updates on this yet?
GiteaMirror commented 2026-04-28 10:07:15 -05:00
Author
Owner
Copy Link

@jbaenaxd commented on GitHub (Feb 19, 2025):

I found this post: https://www.reddit.com/r/LocalLLaMA/comments/1iplsk1/you_can_now_run_models_on_the_neural_engine_if/

It is a new tool, it looks like it's a tool that converts the existing models to be ANE (Apple Neural Engine) compatible: https://github.com/Anemll/Anemll

They mention that it was not an easy task, apparently Apple is limiting the use of this tool.

Conversion Process Overview
ANE models on iOS are limited to 1GB file size. macOS will work with ~2GB
We split models during the conversion process to avoid this limit

They already have some models that you can test, like DeepSeek R1 https://huggingface.co/anemll/anemll-DeepSeekR1-8B-ctx1024_0.1.1

<!-- gh-comment-id:2668703586 --> @jbaenaxd commented on GitHub (Feb 19, 2025): I found this post: https://www.reddit.com/r/LocalLLaMA/comments/1iplsk1/you_can_now_run_models_on_the_neural_engine_if/ It is a new tool, it looks like it's a tool that converts the existing models to be ANE (Apple Neural Engine) compatible: https://github.com/Anemll/Anemll They mention that it was not an easy task, apparently Apple is limiting the use of this tool. > **Conversion Process Overview** ANE models on iOS are limited to 1GB file size. macOS will work with ~2GB We split models during the conversion process to avoid this limit They already have some models that you can test, like DeepSeek R1 https://huggingface.co/anemll/anemll-DeepSeekR1-8B-ctx1024_0.1.1
GiteaMirror commented 2026-04-28 10:07:15 -05:00
Author
Owner
Copy Link

@AShortcuts commented on GitHub (Feb 19, 2025):

Nice 👍

<!-- gh-comment-id:2668772140 --> @AShortcuts commented on GitHub (Feb 19, 2025): Nice 👍
GiteaMirror commented 2026-04-28 10:07:16 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Feb 20, 2025):

Would that be of any help the LM Studio has implemented MLX. And there's FastMLX with an Apache 2.0 license.

<!-- gh-comment-id:2671685175 --> @qdrddr commented on GitHub (Feb 20, 2025): Would that be of any help the [LM Studio has implemented MLX](https://github.com/lmstudio-ai/mlx-engine). And there's [FastMLX](https://github.com/arcee-ai/fastmlx) with an Apache 2.0 license.
GiteaMirror commented 2026-04-28 10:07:17 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Mar 5, 2025):

there's Apple MPS Metal GPU flash attention in swift by the way

<!-- gh-comment-id:2701550396 --> @qdrddr commented on GitHub (Mar 5, 2025): there's Apple MPS [Metal GPU flash attention in swift by the way](https://github.com/philipturner/metal-flash-attention)
GiteaMirror commented 2026-04-28 10:07:18 -05:00
Author
Owner
Copy Link

@Anemll commented on GitHub (Mar 8, 2025):

See https://github.com/anemll/anemll

<!-- gh-comment-id:2708298253 --> @Anemll commented on GitHub (Mar 8, 2025): See https://github.com/anemll/anemll
GiteaMirror commented 2026-04-28 10:07:21 -05:00
Author
Owner
Copy Link

@keefar commented on GitHub (May 29, 2025):

There is also https://llm.mlc.ai

<!-- gh-comment-id:2920717247 --> @keefar commented on GitHub (May 29, 2025): There is also [https://llm.mlc.ai](url)
GiteaMirror commented 2026-04-28 10:07:22 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Sep 20, 2025):

Metal Flash Attention PoC with Python, C, Rust, etc bindings

https://github.com/bghira/universal-metal-flash-attention

<!-- gh-comment-id:3315155372 --> @qdrddr commented on GitHub (Sep 20, 2025): Metal Flash Attention PoC with Python, C, Rust, etc bindings https://github.com/bghira/universal-metal-flash-attention
GiteaMirror commented 2026-04-28 10:07:23 -05:00
Author
Owner
Copy Link

@qdrddr commented on GitHub (Mar 7, 2026):

Metal flash attention in python
https://github.com/marcogva-hub/mlx-flashattention-steel/

<!-- gh-comment-id:4017520293 --> @qdrddr commented on GitHub (Mar 7, 2026): Metal flash attention in python https://github.com/marcogva-hub/mlx-flashattention-steel/
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#48925
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?