github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8132] Fine-tuned Qwen2.5-Instruct isn't supported as expectation #67247

New Issue
Closed
opened 2026-05-04 09:43:26 -05:00 by GiteaMirror · 20 comments
GiteaMirror commented 2026-05-04 09:43:26 -05:00
Owner
Copy Link

Originally created by @sunday-hao on GitHub (Dec 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/8132

Originally assigned to: @pdevine on GitHub.

What is the issue?

Hi, there!
The base model I use is Qwen2.5-Instruct. Firstly, I fine-tuned the base model on my private dataset. Then, I convert it to .gguf format by llama.cpp. Trying to make my model easily supported by ollama.
but it turns out to be:

ollama create my_model -f ./Modelfile
transferring model data 100%
converting model
Error: unsupported architecture

Also, I tried with --quantize and without --quantize argument, both failed with same error as above.
Nevertheless, the qwen2.5 series is supported by newest version of ollama based on ollama.com/library. So is there anyone can help me? thanks a lot

OS

Windows

GPU

AMD

CPU

AMD

Ollama version

0.5.1

Originally created by @sunday-hao on GitHub (Dec 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/8132 Originally assigned to: @pdevine on GitHub. ### What is the issue? Hi, there! The base model I use is Qwen2.5-Instruct. Firstly, I fine-tuned the base model on my private dataset. Then, I convert it to .gguf format by llama.cpp. Trying to make my model easily supported by ollama. but it turns out to be: > ollama create my_model -f ./Modelfile > transferring model data 100% > converting model > Error: unsupported architecture Also, I tried with `--quantize` and without `--quantize` argument, both failed with same error as above. Nevertheless, the qwen2.5 series is supported by newest version of ollama based on [ollama.com/library](url). So is there anyone can help me? thanks a lot ### OS Windows ### GPU AMD ### CPU AMD ### Ollama version 0.5.1
GiteaMirror added the bug label 2026-05-04 09:43:26 -05:00
GiteaMirror commented 2026-05-04 09:43:29 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 17, 2024):

And the command of llama.cpp used to convert the format of my fine-tuned model as below:
python convert-hf-to-gguf.py --outfile path/to/outfile path/to/safetensors

contents of Modelfile as below:
FROM /path/to/gguf_file

<!-- gh-comment-id:2547770078 --> @sunday-hao commented on GitHub (Dec 17, 2024): And the command of llama.cpp used to convert the format of my fine-tuned model as below: `python convert-hf-to-gguf.py --outfile path/to/outfile path/to/safetensors` contents of Modelfile as below: `FROM /path/to/gguf_file`
GiteaMirror commented 2026-05-04 09:43:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 17, 2024):

Which fine tuner?

<!-- gh-comment-id:2547820641 --> @rick-github commented on GitHub (Dec 17, 2024): Which fine tuner?
GiteaMirror commented 2026-05-04 09:43:29 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 17, 2024):

Fullly fine-tune the base model with the tool of LLaMA-Factory

<!-- gh-comment-id:2547831729 --> @sunday-hao commented on GitHub (Dec 17, 2024): Fullly fine-tune the base model with the tool of LLaMA-Factory
GiteaMirror commented 2026-05-04 09:43:30 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 17, 2024):

And there is total 4 safetensors, like model-00001-of-00004.safetensors, model-00002-of-00004.safetensors, under the same root path that I used in the command when I was trying to convert the format to gguf by llama.cpp. Every safetensors is a part of the fine-tuned model, all of these safetensors together is the whole model.
And the memory of .gguf file is equal to the sum of the four .safetensors files approximately.

<!-- gh-comment-id:2548107189 --> @sunday-hao commented on GitHub (Dec 17, 2024): And there is total 4 safetensors, like model-00001-of-00004.safetensors, model-00002-of-00004.safetensors, under the same root path that I used in the command when I was trying to convert the format to gguf by llama.cpp. Every safetensors is a part of the fine-tuned model, all of these safetensors together is the whole model. And the memory of .gguf file is equal to the sum of the four .safetensors files approximately.
GiteaMirror commented 2026-05-04 09:43:30 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 18, 2024):

PTAL when you have a chance. Just to make sure you are here.
And thanks for your self-assignment.

<!-- gh-comment-id:2550230709 --> @sunday-hao commented on GitHub (Dec 18, 2024): PTAL when you have a chance. Just to make sure you are here. And thanks for your self-assignment.
GiteaMirror commented 2026-05-04 09:43:30 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 18, 2024):

Sorry, I meant to comment earlier. The convert logic for Qwen hasn't been added to ollama create yet, but you can still convert the model to GGUF w/ llama.cpp's convert_hf_to_gguf.py script, and then import the resulting GGUF into ollama (again, using ollama create).

I'm not sure when I'll get time to add the convert logic directly, but I am working on some changes for ollama create.

<!-- gh-comment-id:2550325614 --> @pdevine commented on GitHub (Dec 18, 2024): Sorry, I meant to comment earlier. The convert logic for Qwen hasn't been added to `ollama create` yet, but you can still convert the model to GGUF w/ llama.cpp's `convert_hf_to_gguf.py` script, and then import the resulting GGUF into ollama (again, using `ollama create`). I'm not sure when I'll get time to add the convert logic directly, but I am working on some changes for `ollama create`.
GiteaMirror commented 2026-05-04 09:43:31 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 18, 2024):

you mean ollama create is not available at present upon Qwen2.5 series both .safetensors and GGUF format. Is it right?

<!-- gh-comment-id:2550350543 --> @sunday-hao commented on GitHub (Dec 18, 2024): you mean `ollama create` is not available at present upon Qwen2.5 series both .safetensors and GGUF format. Is it right?
GiteaMirror commented 2026-05-04 09:43:32 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 18, 2024):

@sunday-hao ollama create can pull in the GGUF model, but not the Safetensors model right now for Qwen 2.5. I'll add the logic for safetensors, but just haven't had time yet.

<!-- gh-comment-id:2550354410 --> @pdevine commented on GitHub (Dec 18, 2024): @sunday-hao `ollama create` can pull in the GGUF model, but not the Safetensors model right now for Qwen 2.5. I'll add the logic for safetensors, but just haven't had time yet.
GiteaMirror commented 2026-05-04 09:43:33 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 18, 2024):

if my Modelfile that points to the GGUF model under /home/Modelfile and my GGUF model converted from qwen2.5 under /home/downloads/.
So, the command is like ollama create example -f /home/Modelfile.
But, the error occured, unsupported arthitecture.
You mean the convert logic don't support qwen2.5 at present, but the convert logic supports other models?

<!-- gh-comment-id:2550370735 --> @sunday-hao commented on GitHub (Dec 18, 2024): if my Modelfile that points to the GGUF model under /home/Modelfile and my GGUF model converted from qwen2.5 under /home/downloads/. So, the command is like `ollama create example -f /home/Modelfile`. But, the error occured, unsupported arthitecture. You mean the convert logic don't support qwen2.5 at present, but the convert logic supports other models?
GiteaMirror commented 2026-05-04 09:43:34 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 18, 2024):

Make a Modelfile and include a line which looks like:

FROM /path/to/my/gguf/file.gguf

You can then call ollama create -f Modelfile my-model. This is assuming that the GGUF file was created correctly. The vision models are also not (yet) supported.

<!-- gh-comment-id:2550468853 --> @pdevine commented on GitHub (Dec 18, 2024): Make a Modelfile and include a line which looks like: ``` FROM /path/to/my/gguf/file.gguf ``` You can then call `ollama create -f Modelfile my-model`. This is assuming that the GGUF file was created correctly. The vision models are also not (yet) supported.
GiteaMirror commented 2026-05-04 09:43:34 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Dec 18, 2024):

yeah, I followed the pipeline strictly assumming that the GGUF file was created correctly. And the model I convert is Qwen2.5-Instruct, but the error occurred.
According to your response, the reason is qwen2.5 is not supported at present.

<!-- gh-comment-id:2550474154 --> @sunday-hao commented on GitHub (Dec 18, 2024): yeah, I followed the pipeline strictly assumming that the GGUF file was created correctly. And the model I convert is Qwen2.5-Instruct, but the error occurred. According to your response, the reason is qwen2.5 is not supported at present.
GiteaMirror commented 2026-05-04 09:43:35 -05:00
Author
Owner
Copy Link

@loodydo commented on GitHub (Jan 3, 2025):

I am having the same problem when using Unsloth to fine tune and then attempt to import into Ollama. I essentially followed this guide:

https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ

except I used Qwen2.5-Coder in place of llama-3.

After converting to GGUF and attempting to import to Ollama

Modelfile
FROM C:\path_to\fine_tuned_qwen2.5_coder.gguf

ollama create qwen2.5-coder:7b-finetune -f Modelfile

I get

converting model
Error: unsupported architecture

<!-- gh-comment-id:2569821253 --> @loodydo commented on GitHub (Jan 3, 2025): I am having the same problem when using Unsloth to fine tune and then attempt to import into Ollama. I essentially followed this guide: `https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ` except I used Qwen2.5-Coder in place of llama-3. After converting to GGUF and attempting to import to Ollama Modelfile `FROM C:\path_to\fine_tuned_qwen2.5_coder.gguf` `ollama create qwen2.5-coder:7b-finetune -f Modelfile` I get converting model Error: unsupported architecture
GiteaMirror commented 2026-05-04 09:43:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 3, 2025):

The ollama converter doesn't support qwen2 at the moment. Use llama.cpp to convert to GGUF and then import.

<!-- gh-comment-id:2569945114 --> @rick-github commented on GitHub (Jan 3, 2025): The ollama converter doesn't support qwen2 at the moment. Use [llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py) to convert to GGUF and then import.
GiteaMirror commented 2026-05-04 09:43:35 -05:00
Author
Owner
Copy Link

@loodydo commented on GitHub (Jan 4, 2025):

@rick-github I must not understand what GGUF is very well. I assumed converting to GGUF would mean it was GGUF.

I used:
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
which completes without errors and creates "fine_tuned_qwen2.5_coder.gguf" which I then specify in my Modelfile.

I must be missing something fundamental.

<!-- gh-comment-id:2569950861 --> @loodydo commented on GitHub (Jan 4, 2025): @rick-github I must not understand what GGUF is very well. I assumed converting to GGUF would mean it was GGUF. I used: `model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")` which completes without errors and creates "fine_tuned_qwen2.5_coder.gguf" which I then specify in my Modelfile. I must be missing something fundamental.
GiteaMirror commented 2026-05-04 09:43:36 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Jan 4, 2025):

@loodydo it's just a file format. My guess is whatever library you're using to convert it filled in the general.architecture field incorrectly inside of the GGUF.

You can use the convert_hf_to_gguf.py script which you can find in llm/llama.cpp/convert_hf_to_gguf.py.

<!-- gh-comment-id:2569953487 --> @pdevine commented on GitHub (Jan 4, 2025): @loodydo it's just a file format. My guess is whatever library you're using to convert it filled in the `general.architecture` field incorrectly inside of the GGUF. You can use the `convert_hf_to_gguf.py` script which you can find in `llm/llama.cpp/convert_hf_to_gguf.py`.
GiteaMirror commented 2026-05-04 09:43:37 -05:00
Author
Owner
Copy Link

@loodydo commented on GitHub (Jan 4, 2025):

@pdevine The problem may very well be in the gguf creation process. I repeated the exact same fine tuning process were the only change was using a llama3.2 model instead of qwen2 and it imported into Ollama just fine. I will take a closer look at the gguf creation process and see what I find.

<!-- gh-comment-id:2569965462 --> @loodydo commented on GitHub (Jan 4, 2025): @pdevine The problem may very well be in the gguf creation process. I repeated the exact same fine tuning process were the only change was using a llama3.2 model instead of qwen2 and it imported into Ollama just fine. I will take a closer look at the gguf creation process and see what I find.
GiteaMirror commented 2026-05-04 09:43:38 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Jan 4, 2025):

@loodydo hi, I solved my problem in the same way with you. I change my base model from Qwen2.5 to Llama3.1, and I repeated the same fine tune process, converting it to .gguf file. No problem occurred when I run the fine-tuned Llama3.1 model. So I think maybe the root of the problem is Ollama doesn't support Qwen2.5 right now.

<!-- gh-comment-id:2569981282 --> @sunday-hao commented on GitHub (Jan 4, 2025): @loodydo hi, I solved my problem in the same way with you. I change my base model from Qwen2.5 to Llama3.1, and I repeated the same fine tune process, converting it to .gguf file. No problem occurred when I run the fine-tuned Llama3.1 model. So I think maybe the root of the problem is Ollama doesn't support Qwen2.5 right now.
GiteaMirror commented 2026-05-04 09:43:39 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Jan 5, 2025):

@sunday-hao ollama create can't yet directly convert qwen 2.5 safetensors models right now, but the workaround is to use the convert_hf_to_gguf.py script that I mentioned.

<!-- gh-comment-id:2571487970 --> @pdevine commented on GitHub (Jan 5, 2025): @sunday-hao `ollama create` can't yet directly convert qwen 2.5 safetensors models right now, but the workaround is to use the `convert_hf_to_gguf.py` script that I mentioned.
GiteaMirror commented 2026-05-04 09:43:39 -05:00
Author
Owner
Copy Link

@loodydo commented on GitHub (Jan 6, 2025):

I must have had a corrupt GGUF file. I cleaned out my working directory and ran my script again for the qwen 2.5 fine tune. This time I was able to import the generated qwen 2.5 GGUF file into Ollama without errors.

<!-- gh-comment-id:2573406988 --> @loodydo commented on GitHub (Jan 6, 2025): I must have had a corrupt GGUF file. I cleaned out my working directory and ran my script again for the qwen 2.5 fine tune. This time I was able to import the generated qwen 2.5 GGUF file into Ollama without errors.
GiteaMirror commented 2026-05-04 09:43:40 -05:00
Author
Owner
Copy Link

@sunday-hao commented on GitHub (Jan 7, 2025):

@loodydo Thanks for your try. I will convert it again.

<!-- gh-comment-id:2574232917 --> @sunday-hao commented on GitHub (Jan 7, 2025): @loodydo Thanks for your try. I will convert it again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#67247
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?