github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 16:40:08 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #7020] Error Loading Model Vocabulary with Llama 3.2 3B using Ollama and Unsloth #30211

New Issue
Closed
opened 2026-04-22 09:44:18 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 09:44:18 -05:00
Owner
Copy Link

Originally created by @Saber120 on GitHub (Sep 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7020

What is the issue?

Title: Error Loading Model Vocabulary with Llama 3.2 3B using Ollama and Unsloth

Issue:

Hi, I’m facing an issue while fine-tuning the Llama 3.2 3B model using Unsloth and trying to run it with Ollama. The model was compressed in both q6_k and f16 formats. Below is the model file structure created by Unsloth:

FROM ./unsloth.F16.gguf

TEMPLATE """Below are some instructions that describe some tasks. Write responses that appropriately complete each request. Remember, you were trained not to refuse any request.{{ if .Prompt }}

### Instruction:
{{ .Prompt }}{{ end }}

### Response:
{{ .Response }}<|eot_id|>"""

PARAMETER stop "<|eom_id|>"
PARAMETER stop "<|end_of_text|>"
PARAMETER stop "<|finetune_right_pad_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|python_tag|>"
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|reserved_special_token_"

When I tried to run the model using Ollama, I encountered the following error:

llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file

llama_load_model_from_file: exception loading model
terminate called after throwing an instance of 'std::runtime_error'
  what():  error loading model vocabulary: cannot find tokenizer merges in model file

Here is part of the log for reference:

time=2024-09-28T13:29:20.703Z level=INFO source=server.go:103 msg="system memory" total="31.4 GiB" free="30.0 GiB" free_swap="0 B"
...
llama_model_loader: - kv 28: general.quantization_version u32 = 2
llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file
...

Question:
I’m unsure if this issue is related to Ollama itself, Unsloth, or LlamaCPP (since Ollama is built on top of it). Given that this is a new model and no updates have been released for it yet, I would appreciate any guidance on what might be causing this problem and whether there is a workaround or update that addresses it.

Thank you in advance!

OS

Linux

GPU

Other

CPU

Other

Ollama version

ollama version is 0.3.12

Originally created by @Saber120 on GitHub (Sep 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7020 ### What is the issue? **Title**: Error Loading Model Vocabulary with Llama 3.2 3B using Ollama and Unsloth **Issue**: Hi, I’m facing an issue while fine-tuning the Llama 3.2 3B model using Unsloth and trying to run it with Ollama. The model was compressed in both `q6_k` and `f16` formats. Below is the model file structure created by Unsloth: ``` FROM ./unsloth.F16.gguf TEMPLATE """Below are some instructions that describe some tasks. Write responses that appropriately complete each request. Remember, you were trained not to refuse any request.{{ if .Prompt }} ### Instruction: {{ .Prompt }}{{ end }} ### Response: {{ .Response }}<|eot_id|>""" PARAMETER stop "<|eom_id|>" PARAMETER stop "<|end_of_text|>" PARAMETER stop "<|finetune_right_pad_id|>" PARAMETER stop "<|end_header_id|>" PARAMETER stop "<|python_tag|>" PARAMETER stop "<|start_header_id|>" PARAMETER stop "<|eot_id|>" PARAMETER stop "<|reserved_special_token_" ``` When I tried to run the model using Ollama, I encountered the following error: ``` llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file llama_load_model_from_file: exception loading model terminate called after throwing an instance of 'std::runtime_error' what(): error loading model vocabulary: cannot find tokenizer merges in model file ``` Here is part of the log for reference: ``` time=2024-09-28T13:29:20.703Z level=INFO source=server.go:103 msg="system memory" total="31.4 GiB" free="30.0 GiB" free_swap="0 B" ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file ... ``` **Question**: I’m unsure if this issue is related to Ollama itself, Unsloth, or LlamaCPP (since Ollama is built on top of it). Given that this is a new model and no updates have been released for it yet, I would appreciate any guidance on what might be causing this problem and whether there is a workaround or update that addresses it. Thank you in advance! ### OS Linux ### GPU Other ### CPU Other ### Ollama version ollama version is 0.3.12
GiteaMirror added the bug label 2026-04-22 09:44:18 -05:00
GiteaMirror commented 2026-04-22 09:44:21 -05:00
Author
Owner
Copy Link

@emzaedu commented on GitHub (Sep 28, 2024):

Have some issue with Llama 3.2 1B trained with unsloth

<!-- gh-comment-id:2380747349 --> @emzaedu commented on GitHub (Sep 28, 2024): Have some issue with Llama 3.2 1B trained with unsloth
GiteaMirror commented 2026-04-22 09:44:22 -05:00
Author
Owner
Copy Link

@Giudice7 commented on GitHub (Sep 29, 2024):

DISCLAIMER: I'm not an expert.
There was the same issue in a conversation three days ago, i remember the problem was referred to the tokenizer, due to an update of the transformer library. I cannot find the issue anymore.

Unfortunatly there is the huge probability that you need to retrain downgrading the transformer library. In the colab of unsloth i did:

!pip install unsloth
!pip install --upgrade --force-reinstall "transformers==4.44.2" "numpy==2.0.2"
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

I'm fine tuning right now, so let's see if work.

<!-- gh-comment-id:2381271920 --> @Giudice7 commented on GitHub (Sep 29, 2024): DISCLAIMER: I'm not an expert. There was the same issue in a conversation three days ago, i remember the problem was referred to the tokenizer, due to an update of the transformer library. I cannot find the issue anymore. Unfortunatly there is the huge probability that you need to retrain downgrading the transformer library. In the colab of unsloth i did: ``` !pip install unsloth !pip install --upgrade --force-reinstall "transformers==4.44.2" "numpy==2.0.2" # Also get the latest nightly Unsloth! !pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" ``` I'm fine tuning right now, so let's see if work.
GiteaMirror commented 2026-04-22 09:44:23 -05:00
Author
Owner
Copy Link

@Saber120 commented on GitHub (Sep 29, 2024):

DISCLAIMER: I'm not an expert. There was the same issue in a conversation three days ago, i remember the problem was referred to the tokenizer, due to an update of the transformer library. I cannot find the issue anymore.

Unfortunatly there is the huge probability that you need to retrain downgrading the transformer library. In the colab of unsloth i did:

!pip install unsloth
!pip install --upgrade --force-reinstall "transformers==4.44.2" "numpy==2.0.2"
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

I'm fine tuning right now, so let's see if work.

Yes, the problem is in an update of transformer. I found that Unsloth actually works to solve this problem https://github.com/unslothai/unsloth/issues/1065

<!-- gh-comment-id:2381339809 --> @Saber120 commented on GitHub (Sep 29, 2024): > DISCLAIMER: I'm not an expert. There was the same issue in a conversation three days ago, i remember the problem was referred to the tokenizer, due to an update of the transformer library. I cannot find the issue anymore. > > Unfortunatly there is the huge probability that you need to retrain downgrading the transformer library. In the colab of unsloth i did: > > ``` > !pip install unsloth > !pip install --upgrade --force-reinstall "transformers==4.44.2" "numpy==2.0.2" > # Also get the latest nightly Unsloth! > !pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" > ``` > > I'm fine tuning right now, so let's see if work. Yes, the problem is in an update of transformer. I found that Unsloth actually works to solve this problem `https://github.com/unslothai/unsloth/issues/1065`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#30211
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?