github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6823] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出 #66348

New Issue
Closed
opened 2026-05-04 02:49:35 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-05-04 02:49:35 -05:00
Owner
Copy Link

Originally created by @czhcc on GitHub (Sep 16, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6823

What is the issue?

而同样的gguf，用其它框架直接加载使用是正常的。
原始的qwen2模型转为gguf加载也不需要TEMPLATE参数。只有微调训练后生成的gguf要加TEMPLATE才不会胡乱输出。为什么？

OS

Linux, Docker

GPU

Nvidia

CPU

Intel

Ollama version

0.3.10

Originally created by @czhcc on GitHub (Sep 16, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6823 ### What is the issue? 而同样的gguf，用其它框架直接加载使用是正常的。 原始的qwen2模型转为gguf加载也不需要TEMPLATE参数。只有微调训练后生成的gguf要加TEMPLATE才不会胡乱输出。为什么？ ### OS Linux, Docker ### GPU Nvidia ### CPU Intel ### Ollama version 0.3.10
GiteaMirror added the bug label 2026-05-04 02:49:35 -05:00
GiteaMirror commented 2026-05-04 02:49:56 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2024):

The TEMPLATE is used to parse the query into a format suitable for the model. Without a properly formatted query, the output from the model will be inconsistent. Other frameworks are probably using the chat_template embedded in the GGUF file. When creating a new model, ollama tries to create a TEMPLATE based on known chat_templates but if it's not recognized, the default TEMPLATE may not be suitable. If you are fine-tuning a qwen2 model then the chat_template may be changed or removed by the fine-tuning process, in which case the template from the ollama library may work.

<!-- gh-comment-id:2352103595 --> @rick-github commented on GitHub (Sep 16, 2024): The TEMPLATE is used to parse the query into a format suitable for the model. Without a properly formatted query, the output from the model will be inconsistent. Other frameworks are probably using the chat_template embedded in the GGUF file. When creating a new model, ollama tries to create a TEMPLATE based on known chat_templates but if it's not recognized, the default TEMPLATE may not be suitable. If you are fine-tuning a qwen2 model then the chat_template may be changed or removed by the fine-tuning process, in which case the [template](https://ollama.com/library/qwen2/blobs/77c91b422cc9) from the ollama library may work.
GiteaMirror commented 2026-05-04 02:50:06 -05:00
Author
Owner
Copy Link

@qscwdv65 commented on GitHub (Sep 21, 2024):

I also encountered a similar issue where the model's generation quality in Ollama is extremely poor, producing nonsensical outputs or repeating the same text. I am fine-tuning this model: shenzhi-wang/Llama3.1-8B-Chinese-Chat. How can I determine which TEMPLATE to use in Ollama?

I tried using the TEMPLATE for Llama 3.1, but I still encountered the same issue (poor generation quality, producing nonsensical outputs or repeating the same text).

https://ollama.com/library/llama3.1:8b/blobs/948af2743fc7

<!-- gh-comment-id:2365101246 --> @qscwdv65 commented on GitHub (Sep 21, 2024): I also encountered a similar issue where the model's generation quality in Ollama is extremely poor, producing nonsensical outputs or repeating the same text. I am fine-tuning this model: shenzhi-wang/Llama3.1-8B-Chinese-Chat. How can I determine which TEMPLATE to use in Ollama? I tried using the TEMPLATE for Llama 3.1, but I still encountered the same issue (poor generation quality, producing nonsensical outputs or repeating the same text). https://ollama.com/library/llama3.1:8b/blobs/948af2743fc7
GiteaMirror commented 2026-05-04 02:50:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 21, 2024):

It depends on how you are creating the model you are importing to ollama. Are you creating a new GGUF file or just an adapter? What framework are you using to create the GGUF/adapter? What's the contents of the Modelfile?

<!-- gh-comment-id:2365144836 --> @rick-github commented on GitHub (Sep 21, 2024): It depends on how you are creating the model you are importing to ollama. Are you creating a new GGUF file or just an adapter? What framework are you using to create the GGUF/adapter? What's the contents of the Modelfile?
GiteaMirror commented 2026-05-04 02:50:23 -05:00
Author
Owner
Copy Link

@qscwdv65 commented on GitHub (Sep 21, 2024):

Hi @rick-github, thank you for your reply.

I am currently trying to fine-tune a model. The model I am working on is shenzhi-wang/Llama3.1-8B-Chinese-Chat. I am fine-tuning it using Llama Factory and exporting it in safetensors format.

I have already tried two methods to import the model into Ollama:

  1. Using llama.cpp to convert safetensors to gguf.
  2. Following the instructions on this page: Importing a model from safetensors weights to directly import the safetensors file into Ollama.

For both methods, I used the modelfile and ollama create commands to import the model into Ollama. I have tried including the following two types in the modelfile, but the quality of the generated responses is still not good:

  1. SYSTEM """<some of my system message...>"""
  2. Ollama's Llama 3.1 TEMPLATE: https://ollama.com/library/llama3.1:8b/blobs/948af2743fc7

From my testing, it seems that the issue is with the TEMPLATE, but I am unsure of which TEMPLATE to use to ensure good quality responses.
I would really appreciate any guidance or suggestions you can provide. Thank you so much!

<!-- gh-comment-id:2365223678 --> @qscwdv65 commented on GitHub (Sep 21, 2024): Hi @rick-github, thank you for your reply. I am currently trying to fine-tune a model. The model I am working on is shenzhi-wang/Llama3.1-8B-Chinese-Chat. I am fine-tuning it using Llama Factory and exporting it in safetensors format. I have already tried two methods to import the model into Ollama: 1. Using llama.cpp to convert safetensors to gguf. 2. Following the instructions on this page: [Importing a model from safetensors weights](https://github.com/ollama/ollama/blob/main/docs/import.md#importing-a-model-from-safetensors-weights) to directly import the safetensors file into Ollama. For both methods, I used the modelfile and ollama create commands to import the model into Ollama. I have tried including the following two types in the modelfile, but the quality of the generated responses is still not good: 1. `SYSTEM """<some of my system message...>"""` 2. Ollama's Llama 3.1 TEMPLATE: https://ollama.com/library/llama3.1:8b/blobs/948af2743fc7 From my testing, it seems that the issue is with the TEMPLATE, but I am unsure of which TEMPLATE to use to ensure good quality responses. I would really appreciate any guidance or suggestions you can provide. Thank you so much!
GiteaMirror commented 2026-05-04 02:50:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 22, 2024):

Are you using the webui or the cli to fine-tune the model? If the latter, what's the contents of the yaml file?

<!-- gh-comment-id:2366846284 --> @rick-github commented on GitHub (Sep 22, 2024): Are you using the webui or the cli to fine-tune the model? If the latter, what's the contents of the yaml file?
GiteaMirror commented 2026-05-04 02:50:34 -05:00
Author
Owner
Copy Link

@qscwdv65 commented on GitHub (Sep 23, 2024):

Are you using the webui or the cli to fine-tune the model? If the latter, what's the contents of the yaml file?

I used the Llama Factory web UI to fine-tune the model. After re-importing the model in safetensors format into Ollama earlier, it worked successfully this time. I used the second TEMPLATE mentioned above (‘s Llama 3.1 TEMPLATE). The only difference was that I removed the License paragraph (although I’m not sure if the initial issue was due to an incomplete copy). In any case, after the import, the model no longer outputs nonsensical responses.

However, I still have a question. If I want to fine-tune models that have already been fine-tuned by others, such as shenzhi-wang/Llama3.1-8B-Chinese-Chat, which is based on meta-llama/Meta-Llama-3.1-8B, I encounter an issue. When I choose the Llama-3.1-8B TEMPLATE in Llama Factory, the model outputs random responses. But when I use LLaMA3.1-8B-Chat, the model behaves normally. So, I’m not sure how to determine which TEMPLATE to use. Could you please provide some guidance? Thank you!

<!-- gh-comment-id:2368575777 --> @qscwdv65 commented on GitHub (Sep 23, 2024): > Are you using the webui or the cli to fine-tune the model? If the latter, what's the contents of the yaml file? I used the Llama Factory web UI to fine-tune the model. After re-importing the model in safetensors format into Ollama earlier, it worked successfully this time. I used the second TEMPLATE mentioned above (‘s Llama 3.1 TEMPLATE). The only difference was that I removed the License paragraph (although I’m not sure if the initial issue was due to an incomplete copy). In any case, after the import, the model no longer outputs nonsensical responses. However, I still have a question. If I want to fine-tune models that have already been fine-tuned by others, such as shenzhi-wang/Llama3.1-8B-Chinese-Chat, which is based on meta-llama/Meta-Llama-3.1-8B, I encounter an issue. When I choose the Llama-3.1-8B TEMPLATE in Llama Factory, the model outputs random responses. But when I use LLaMA3.1-8B-Chat, the model behaves normally. So, I’m not sure how to determine which TEMPLATE to use. Could you please provide some guidance? Thank you!
GiteaMirror commented 2026-05-04 02:50:37 -05:00
Author
Owner
Copy Link

@qq1273834091 commented on GitHub (Dec 11, 2024):

如果是后者，yaml文件的内容是什么您是否使用webui或cli来微调模型？

我使用了Llama Factory web UI来微调模型。在将安全传感器格式的模型重新导入Ollama之后，这次成功了。我使用上面提到的第二个模板 (的Llama 3.1模板)。唯一的区别是我删除了许可证段落 (尽管我不确定最初的问题是否是由于不完整的副本)。无论如何，在导入之后，模型不再输出无意义的响应。

然而，我仍然有一个问题。如果我想微调别人已经微调过的模型，比如神智网/Llama3.1-8B-Chinese-Chat，它是基于美洲驼/Meta-Llama-3.1-8B的，我遇到一个问题。当我在Llama工厂中选择Llama-3.1-8B模板时，模型输出随机响应。但是当我使用LLaMA3.1-8B-聊天时，模型行为正常。所以，我不知道如何确定使用哪个模板。你能提供一些指导吗？谢谢!

遇到同样的问题，在使用Llama Factory微调时导出的模型再使用llama.cpp转换成gguf格式 导入到ollama中会存在胡言乱语。
猜测是我再 使用 ollama create model -f Modelfile 中的配置不正确 我的配置是 FROM ./model.gguf

<!-- gh-comment-id:2533367205 --> @qq1273834091 commented on GitHub (Dec 11, 2024): > > 如果是后者，yaml文件的内容是什么您是否使用webui或cli来微调模型？ > > 我使用了Llama Factory web UI来微调模型。在将安全传感器格式的模型重新导入Ollama之后，这次成功了。我使用上面提到的第二个模板 (的Llama 3.1模板)。唯一的区别是我删除了许可证段落 (尽管我不确定最初的问题是否是由于不完整的副本)。无论如何，在导入之后，模型不再输出无意义的响应。 > > 然而，我仍然有一个问题。如果我想微调别人已经微调过的模型，比如神智网/Llama3.1-8B-Chinese-Chat，它是基于美洲驼/Meta-Llama-3.1-8B的，我遇到一个问题。当我在Llama工厂中选择Llama-3.1-8B模板时，模型输出随机响应。但是当我使用LLaMA3.1-8B-聊天时，模型行为正常。所以，我不知道如何确定使用哪个模板。你能提供一些指导吗？谢谢! 遇到同样的问题，在使用Llama Factory微调时导出的模型再使用llama.cpp转换成gguf格式 导入到ollama中会存在胡言乱语。 猜测是我再 使用 ollama create model -f Modelfile 中的配置不正确 我的配置是 FROM ./model.gguf
GiteaMirror commented 2026-05-04 02:50:38 -05:00
Author
Owner
Copy Link

@breadxiexie commented on GitHub (Dec 16, 2024):

我目前正在尝试微调模型。我正在研究的模型是 shenzhi-wang/Llama3.1-8B-Chinese-Chat。我正在使用 Llama Factory 对其进行微调，并将其导出为 safetensors 格式。
2. 按照此页面上的说明进行操作：从 safetensors 权重导入模型，以将 safetensors 文件直接导入 Ollama。

请问你是如何成功将微调后合并的safetensors格式的模型导入的？按照上面提到的流程来的话在运行ollama create Modelfile这一步会报错：ollama create Modelfile
transferring model data 100%
converting model
Error: json: cannot unmarshal array into Go struct field .model.merges of type string

<!-- gh-comment-id:2544298277 --> @breadxiexie commented on GitHub (Dec 16, 2024): > 我目前正在尝试微调模型。我正在研究的模型是 shenzhi-wang/Llama3.1-8B-Chinese-Chat。我正在使用 Llama Factory 对其进行微调，并将其导出为 safetensors 格式。 > 2. 按照此页面上的说明进行操作：[从 safetensors 权重导入模型](https://github.com/ollama/ollama/blob/main/docs/import.md#importing-a-model-from-safetensors-weights)，以将 safetensors 文件直接导入 Ollama。 请问你是如何成功将微调后合并的safetensors格式的模型导入的？按照上面提到的流程来的话在运行ollama create Modelfile这一步会报错：ollama create Modelfile transferring model data 100% converting model Error: json: cannot unmarshal array into Go struct field .model.merges of type string
GiteaMirror commented 2026-05-04 02:50:43 -05:00
Author
Owner
Copy Link

@qq1273834091 commented on GitHub (Dec 16, 2024):

我并没有使用ollama接口safetensors导入。我使用的是MODEFILE 导入gguf的文件
。我先使用llama.cpp把safetensors文件模型生成gguf文件后再导入的。暂时还没有尝试过ollama提供的safetensors接口。

---原始邮件---
发件人: @.>
发送时间: 2024年12月16日(周一) 上午9:36
收件人: @.>;
抄送: @.@.>;
主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出 (Issue #6823)

我目前正在尝试微调模型。我正在研究的模型是 shenzhi-wang/Llama3.1-8B-Chinese-Chat。我正在使用 Llama Factory 对其进行微调，并将其导出为 safetensors 格式。
2. 按照此页面上的说明进行操作：从 safetensors 权重导入模型，以将 safetensors 文件直接导入 Ollama。

请问你是如何成功将微调后合并的safetensors格式的模型导入的？按照上面提到的流程来的话在运行ollama create Modelfile这一步会报错：ollama create Modelfile
transferring model data 100%
converting model
Error: json: cannot unmarshal array into Go struct field .model.merges of type string

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

<!-- gh-comment-id:2544308125 --> @qq1273834091 commented on GitHub (Dec 16, 2024): 我并没有使用ollama接口safetensors导入。我使用的是MODEFILE 导入gguf的文件 。我先使用llama.cpp把safetensors文件模型生成gguf文件后再导入的。暂时还没有尝试过ollama提供的safetensors接口。 ---原始邮件--- 发件人: ***@***.***&gt; 发送时间: 2024年12月16日(周一) 上午9:36 收件人: ***@***.***&gt;; 抄送: ***@***.******@***.***&gt;; 主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出 (Issue #6823) 我目前正在尝试微调模型。我正在研究的模型是 shenzhi-wang/Llama3.1-8B-Chinese-Chat。我正在使用 Llama Factory 对其进行微调，并将其导出为 safetensors 格式。 2. 按照此页面上的说明进行操作：从 safetensors 权重导入模型，以将 safetensors 文件直接导入 Ollama。 请问你是如何成功将微调后合并的safetensors格式的模型导入的？按照上面提到的流程来的话在运行ollama create Modelfile这一步会报错：ollama create Modelfile transferring model data 100% converting model Error: json: cannot unmarshal array into Go struct field .model.merges of type string — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;
GiteaMirror commented 2026-05-04 02:50:48 -05:00
Author
Owner
Copy Link

@Elijah0405 commented on GitHub (Feb 14, 2025):

主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出，请问，怎么指定TEMPLATE呢，在ollama creat model -f Modelfile这个步骤指定吗，还是在Modelfile文件里面，还是在转换格式的那一步？我也遇到了同样的问题，转换成gguf格式以后就开始胡言乱语

<!-- gh-comment-id:2658176778 --> @Elijah0405 commented on GitHub (Feb 14, 2025): 主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出，请问，怎么指定TEMPLATE呢，在ollama creat model -f Modelfile这个步骤指定吗，还是在Modelfile文件里面，还是在转换格式的那一步？我也遇到了同样的问题，转换成gguf格式以后就开始胡言乱语
GiteaMirror commented 2026-05-04 02:50:54 -05:00
Author
Owner
Copy Link

@Elijah0405 commented on GitHub (Feb 14, 2025):

主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出，请问，怎么指定TEMPLATE呢，在ollama creat model -f Modelfile这个步骤指定吗，还是在Modelfile文件里面，还是在转换格式的那一步？我也遇到了同样的问题，转换成gguf格式以后就开始胡言乱语

已解决，模板（Template）是大模型与用户对话的“协议框架”，它确保模型在推理阶段接收和生成的数据格式与训练时完全一致。即使模型训练时掌握了知识，格式错位会导致模型“学以致用”的能力失效。复制ollama官方模型下面的TEMPLATE，粘贴到Modelfile中，再利用ollama部署，模型就可以正常回答了。

<!-- gh-comment-id:2658526503 --> @Elijah0405 commented on GitHub (Feb 14, 2025): > 主题: Re: [ollama/ollama] 用qwen2微调训练出的hf转为gguf后，用ollama加载，要指定TEMPLATE才不会胡乱输出，请问，怎么指定TEMPLATE呢，在ollama creat model -f Modelfile这个步骤指定吗，还是在Modelfile文件里面，还是在转换格式的那一步？我也遇到了同样的问题，转换成gguf格式以后就开始胡言乱语 已解决，模板（Template）是大模型与用户对话的“协议框架”，它确保模型在推理阶段接收和生成的数据格式与训练时完全一致。即使模型训练时掌握了知识，格式错位会导致模型“学以致用”的能力失效。复制ollama官方模型下面的TEMPLATE，粘贴到Modelfile中，再利用ollama部署，模型就可以正常回答了。
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#66348
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?