github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 17:49:24 -05:00
Code Issues 14.6k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7863] OLMo-2-1124-13B & 7B #51541

New Issue
Closed
opened 2026-04-28 20:29:41 -05:00 by GiteaMirror · 16 comments
GiteaMirror commented 2026-04-28 20:29:41 -05:00
Owner
Copy Link

Originally created by @vYLQs6 on GitHub (Nov 27, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7863

Originally assigned to: @BruceMacD on GitHub.

https://huggingface.co/collections/allenai/olmo-2-674117b93ab84e98afc72edc

Evaluation

Core model results for OLMo 2 7B and 13B models are found below.

Model Train FLOPs Average ARC/C HSwag WinoG MMLU DROP NQ AGIEval GSM8k MMLUPro TriviaQA
Open weights models:
Llama-2-13B 1.6·10²³ 54.1 67.3 83.9 74.9 55.7 45.6 38.4 41.5 28.1 23.9 81.3
Mistral-7B-v0.3 n/a 58.8 78.3 83.1 77.7 63.5 51.8 37.2 47.3 40.1 30 79.3
Llama-3.1-8B 7.2·10²³ 61.8 79.5 81.6 76.6 66.9 56.4 33.9 51.3 56.5 34.7 80.3
Mistral-Nemo-12B n/a 66.9 85.2 85.6 81.5 69.5 69.2 39.7 54.7 62.1 36.7 84.6
Qwen-2.5-7B 8.2·10²³ 67.4 89.5 89.7 74.2 74.4 55.8 29.9 63.7 81.5 45.8 69.4
Gemma-2-9B 4.4·10²³ 67.8 89.5 87.3 78.8 70.6 63 38 57.3 70.1 42 81.8
Qwen-2.5-14B 16.0·10²³ 72.2 94 94 80 79.3 51.5 37.3 71 83.4 52.8 79.1
Partially open models:
StableLM-2-12B 2.9·10²³ 62.2 81.9 84.5 77.7 62.4 55.5 37.6 50.9 62 29.3 79.9
Zamba-2-7B n/c 65.2 92.2 89.4 79.6 68.5 51.7 36.5 55.5 67.2 32.8 78.8
Fully open models:
Amber-7B 0.5·10²³ 35.2 44.9 74.5 65.5 24.7 26.1 18.7 21.8 4.8 11.7 59.3
OLMo-7B 1.0·10²³ 38.3 46.4 78.1 68.5 28.3 27.3 24.8 23.7 9.2 12.1 64.1
MAP-Neo-7B 2.1·10²³ 49.6 78.4 72.8 69.2 58 39.4 28.9 45.8 12.5 25.9 65.1
OLMo-0424-7B 0.9·10²³ 50.7 66.9 80.1 73.6 54.3 50 29.6 43.9 27.7 22.1 58.8
DCLM-7B 1.0·10²³ 56.9 79.8 82.3 77.3 64.4 39.3 28.8 47.5 46.1 31.3 72.1
OLMo-2-1124-7B 1.8·10²³ 62.9 79.8 83.8 77.2 63.7 60.8 36.9 50.4 67.5 31 78
OLMo-2-1124-13B 4.6·10²³ 68.3 83.5 86.4 81.5 67.5 70.7 46.7 54.2 75.1 35.1 81.9
Originally created by @vYLQs6 on GitHub (Nov 27, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7863 Originally assigned to: @BruceMacD on GitHub. https://huggingface.co/collections/allenai/olmo-2-674117b93ab84e98afc72edc ## Evaluation Core model results for OLMo 2 7B and 13B models are found below. | Model | Train FLOPs | Average | ARC/C | HSwag | WinoG | MMLU | DROP | NQ | AGIEval | GSM8k | MMLUPro | TriviaQA | |-------------------|------------|---------|--------|--------|--------|-------|-------|-----|----------|--------|-----------|-----------| | *Open weights models:* | | Llama-2-13B | 1.6·10²³ | 54.1 | 67.3 | 83.9 | 74.9 | 55.7 | 45.6 | 38.4 | 41.5 | 28.1 | 23.9 | 81.3 | | Mistral-7B-v0.3 | n/a | 58.8 | 78.3 | 83.1 | 77.7 | 63.5 | 51.8 | 37.2 | 47.3 | 40.1 | 30 | 79.3 | | Llama-3.1-8B | 7.2·10²³ | 61.8 | 79.5 | 81.6 | 76.6 | 66.9 | 56.4 | 33.9 | 51.3 | 56.5 | 34.7 | 80.3 | | Mistral-Nemo-12B | n/a | 66.9 | 85.2 | 85.6 | 81.5 | 69.5 | 69.2 | 39.7 | 54.7 | 62.1 | 36.7 | 84.6 | | Qwen-2.5-7B | 8.2·10²³ | 67.4 | 89.5 | 89.7 | 74.2 | 74.4 | 55.8 | 29.9 | 63.7 | 81.5 | 45.8 | 69.4 | | Gemma-2-9B | 4.4·10²³ | 67.8 | 89.5 | 87.3 | 78.8 | 70.6 | 63 | 38 | 57.3 | 70.1 | 42 | 81.8 | | Qwen-2.5-14B | 16.0·10²³ | 72.2 | 94 | 94 | 80 | 79.3 | 51.5 | 37.3 | 71 | 83.4 | 52.8 | 79.1 | | *Partially open models:* | | StableLM-2-12B | 2.9·10²³ | 62.2 | 81.9 | 84.5 | 77.7 | 62.4 | 55.5 | 37.6 | 50.9 | 62 | 29.3 | 79.9 | | Zamba-2-7B | n/c | 65.2 | 92.2 | 89.4 | 79.6 | 68.5 | 51.7 | 36.5 | 55.5 | 67.2 | 32.8 | 78.8 | | *Fully open models:* | | Amber-7B | 0.5·10²³ | 35.2 | 44.9 | 74.5 | 65.5 | 24.7 | 26.1 | 18.7 | 21.8 | 4.8 | 11.7 | 59.3 | | OLMo-7B | 1.0·10²³ | 38.3 | 46.4 | 78.1 | 68.5 | 28.3 | 27.3 | 24.8 | 23.7 | 9.2 | 12.1 | 64.1 | | MAP-Neo-7B | 2.1·10²³ | 49.6 | 78.4 | 72.8 | 69.2 | 58 | 39.4 | 28.9 | 45.8 | 12.5 | 25.9 | 65.1 | | OLMo-0424-7B | 0.9·10²³ | 50.7 | 66.9 | 80.1 | 73.6 | 54.3 | 50 | 29.6 | 43.9 | 27.7 | 22.1 | 58.8 | | DCLM-7B | 1.0·10²³ | 56.9 | 79.8 | 82.3 | 77.3 | 64.4 | 39.3 | 28.8 | 47.5 | 46.1 | 31.3 | 72.1 | | **OLMo-2-1124-7B** | 1.8·10²³ | 62.9 | 79.8 | 83.8 | 77.2 | 63.7 | 60.8 | 36.9 | 50.4 | 67.5 | 31 | 78 | | **OLMo-2-1124-13B** | 4.6·10²³ | 68.3 | 83.5 | 86.4 | 81.5 | 67.5 | 70.7 | 46.7 | 54.2 | 75.1 | 35.1 | 81.9 |
GiteaMirror added the model label 2026-04-28 20:29:41 -05:00
GiteaMirror commented 2026-04-28 20:29:43 -05:00
Author
Owner
Copy Link

@UberMetroid commented on GitHub (Nov 28, 2024):

I second this request

<!-- gh-comment-id:2506241848 --> @UberMetroid commented on GitHub (Nov 28, 2024): I second this request
GiteaMirror commented 2026-04-28 20:29:44 -05:00
Author
Owner
Copy Link

@kth8 commented on GitHub (Nov 28, 2024):

ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF
ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF
<!-- gh-comment-id:2506274827 --> @kth8 commented on GitHub (Nov 28, 2024): ``` ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF ```
GiteaMirror commented 2026-04-28 20:29:44 -05:00
Author
Owner
Copy Link

@UberMetroid commented on GitHub (Nov 28, 2024):

ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF
ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF

llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade

<!-- gh-comment-id:2506454672 --> @UberMetroid commented on GitHub (Nov 28, 2024): > ``` > ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF > ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF > ``` llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade
GiteaMirror commented 2026-04-28 20:29:45 -05:00
Author
Owner
Copy Link

@MoreFoxBeans commented on GitHub (Nov 28, 2024):

Third!

<!-- gh-comment-id:2506747577 --> @MoreFoxBeans commented on GitHub (Nov 28, 2024): Third!
GiteaMirror commented 2026-04-28 20:29:46 -05:00
Author
Owner
Copy Link

@razvanab commented on GitHub (Nov 30, 2024):

+1

<!-- gh-comment-id:2508928050 --> @razvanab commented on GitHub (Nov 30, 2024): +1
GiteaMirror commented 2026-04-28 20:29:47 -05:00
Author
Owner
Copy Link

@gad2103 commented on GitHub (Dec 3, 2024):

+1

<!-- gh-comment-id:2513535032 --> @gad2103 commented on GitHub (Dec 3, 2024): +1
GiteaMirror commented 2026-04-28 20:29:48 -05:00
Author
Owner
Copy Link

@UberMetroid commented on GitHub (Dec 6, 2024):

keeping this alive

<!-- gh-comment-id:2523491538 --> @UberMetroid commented on GitHub (Dec 6, 2024): keeping this alive
GiteaMirror commented 2026-04-28 20:29:49 -05:00
Author
Owner
Copy Link

@tcsenpai commented on GitHub (Dec 8, 2024):

i second this. Open source models for an open source project.

<!-- gh-comment-id:2525683506 --> @tcsenpai commented on GitHub (Dec 8, 2024): i second this. Open source models for an open source project.
GiteaMirror commented 2026-04-28 20:29:50 -05:00
Author
Owner
Copy Link

@liar666 commented on GitHub (Dec 9, 2024):

i second this. Open source models for an open source project.
+1

<!-- gh-comment-id:2527327676 --> @liar666 commented on GitHub (Dec 9, 2024): > i second this. Open source models for an open source project. +1
GiteaMirror commented 2026-04-28 20:29:52 -05:00
Author
Owner
Copy Link

@liar666 commented on GitHub (Dec 9, 2024):

ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF
ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF

llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade

Same problem here... Getting the weights does not seem to be enough. Is there anything in the OLMo architecture that lacks in Ollama?

<!-- gh-comment-id:2527329358 --> @liar666 commented on GitHub (Dec 9, 2024): > > ``` > > ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF > > ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF > > ``` > > llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade Same problem here... Getting the weights does not seem to be enough. Is there anything in the OLMo architecture that lacks in Ollama?
GiteaMirror commented 2026-04-28 20:29:53 -05:00
Author
Owner
Copy Link

@tcsenpai commented on GitHub (Dec 9, 2024):

ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF
ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF

llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade

Same problem here... Getting the weights does not seem to be enough. Is there anything in the OLMo architecture that lacks in Ollama?

As I see it, each time a new architecture ( "XXXForCausalLLM" styled) comes out, it needs a bit of time before it can work. I think is supported in llama.cpp tho so maybe this will be faster.

<!-- gh-comment-id:2527903861 --> @tcsenpai commented on GitHub (Dec 9, 2024): > > > ``` > > > ollama run hf.co/bartowski/OLMo-2-1124-7B-Instruct-GGUF > > > ollama run hf.co/bartowski/OLMo-2-1124-13B-Instruct-GGUF > > > ``` > > > > > > llama runner process has terminated: this model is not supported by your version of Ollama. You may need to upgrade > > Same problem here... Getting the weights does not seem to be enough. Is there anything in the OLMo architecture that lacks in Ollama? As I see it, each time a new architecture ( "XXXForCausalLLM" styled) comes out, it needs a bit of time before it can work. I think is supported in llama.cpp tho so maybe this will be faster.
GiteaMirror commented 2026-04-28 20:29:59 -05:00
Author
Owner
Copy Link

@olumolu commented on GitHub (Jan 4, 2025):

https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF
https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF

<!-- gh-comment-id:2570046153 --> @olumolu commented on GitHub (Jan 4, 2025): https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF
GiteaMirror commented 2026-04-28 20:30:01 -05:00
Author
Owner
Copy Link

@tcsenpai commented on GitHub (Jan 4, 2025):

https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF

Does this works? I think being the same architecture it doesn't?

<!-- gh-comment-id:2570934724 --> @tcsenpai commented on GitHub (Jan 4, 2025): > https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF Does this works? I think being the same architecture it doesn't?
GiteaMirror commented 2026-04-28 20:30:04 -05:00
Author
Owner
Copy Link

@vYLQs6 commented on GitHub (Jan 5, 2025):

https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF

Does this works? I think being the same architecture it doesn't?

Llama.cpp and ollama are now "very separate" projects. Ollama switched to a backend written in Go, so even if llama.cpp supports a model, it does not mean that ollama will instantly support it.

<!-- gh-comment-id:2571662444 --> @vYLQs6 commented on GitHub (Jan 5, 2025): > > https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF > > Does this works? I think being the same architecture it doesn't? Llama.cpp and ollama are now "very separate" projects. Ollama switched to a backend written in Go, so even if llama.cpp supports a model, it does not mean that ollama will instantly support it.
GiteaMirror commented 2026-04-28 20:30:10 -05:00
Author
Owner
Copy Link

@olumolu commented on GitHub (Jan 5, 2025):

https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF

Does this works? I think being the same architecture it doesn't?

Llama.cpp and ollama are now "very separate" projects. Ollama switched to a backend written in Go, so even if llama.cpp supports a model, it does not mean that ollama will instantly support it.

Ollama use the llama.cpp as a backend.

<!-- gh-comment-id:2571663985 --> @olumolu commented on GitHub (Jan 5, 2025): > > > https://huggingface.co/allenai/OLMo-2-1124-7B-GGUF https://huggingface.co/allenai/OLMo-2-1124-13B-GGUF > > > > > > Does this works? I think being the same architecture it doesn't? > > Llama.cpp and ollama are now "very separate" projects. Ollama switched to a backend written in Go, so even if llama.cpp supports a model, it does not mean that ollama will instantly support it. Ollama use the llama.cpp as a backend.
GiteaMirror commented 2026-04-28 20:30:12 -05:00
Author
Owner
Copy Link

@UberMetroid commented on GitHub (Jan 12, 2025):

You can pull Olmo now. https://ollama.com/library/olmo2:13b

<!-- gh-comment-id:2585701235 --> @UberMetroid commented on GitHub (Jan 12, 2025): You can pull Olmo now. https://ollama.com/library/olmo2:13b
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label model
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51541
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?