github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8363] After fine-tuning llama3.1 and deploying it to ollama, the inference may not stop, or the output may be repeated, or the output may be gibberish. #51875

New Issue
Closed
opened 2026-04-28 21:06:33 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-28 21:06:33 -05:00
Owner
Copy Link

Originally created by @xjtupy on GitHub (Jan 9, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8363

What is the issue?

After fine-tuning llama3.1, I exported a .gguf file and deployed it using ollama create. However, problems such as being unable to stop and repeated output occurred during the inference process.
image

How to solve this problem？

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.3.2

Originally created by @xjtupy on GitHub (Jan 9, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8363 ### What is the issue? After fine-tuning llama3.1, I exported a .gguf file and deployed it using ollama create. However, problems such as being unable to stop and repeated output occurred during the inference process. ![image](https://github.com/user-attachments/assets/9ffe8bd7-40f5-4a5b-b4d2-abd054911657) How to solve this problem？ ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.3.2
GiteaMirror added the bug label 2026-04-28 21:06:33 -05:00
GiteaMirror commented 2026-04-28 21:06:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 9, 2025):

The Modelfile you used to import the GGUF file is missing PARAMETER stop <|end_of_text|> and possibly other stop tokens.

<!-- gh-comment-id:2580038275 --> @rick-github commented on GitHub (Jan 9, 2025): The Modelfile you used to import the GGUF file is missing `PARAMETER stop <|end_of_text|>` and possibly other stop tokens.
GiteaMirror commented 2026-04-28 21:06:44 -05:00
Author
Owner
Copy Link

@xjtupy commented on GitHub (Jan 10, 2025):

@rick-github Thanks
I converted the model file to .gguf format through the convert-hf-to-gguf.py file under the llama.cpp project, and then created a Modelfile file with the following content:

FROM /home/xxxx/megred-model-path.gguf

Finally, deploy it to ollama using the following command:

ollama create llama3-chat-merged -f Modelfile

Could you please give me a sample Modelfile?

<!-- gh-comment-id:2581634998 --> @xjtupy commented on GitHub (Jan 10, 2025): @rick-github Thanks I converted the model file to .gguf format through the convert-hf-to-gguf.py file under the llama.cpp project, and then created a Modelfile file with the following content: ```shell FROM /home/xxxx/megred-model-path.gguf ``` Finally, deploy it to ollama using the following command: ```shell ollama create llama3-chat-merged -f Modelfile ``` Could you please give me a sample Modelfile?
GiteaMirror commented 2026-04-28 21:06:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 10, 2025):

FROM /home/xxxx/megred-model-path.gguf
PARAMETER stop <|end_of_text|>
<!-- gh-comment-id:2581638894 --> @rick-github commented on GitHub (Jan 10, 2025): ```modelfile FROM /home/xxxx/megred-model-path.gguf PARAMETER stop <|end_of_text|> ```
GiteaMirror commented 2026-04-28 21:06:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 10, 2025):

Also note that you probably want a TEMPLATE as well. If you look at the model settings for llama3.1 that's in the library that will give you a starting point. Because you have fine-tuned the model you probably can't just drop in the values from the ollama library, some experimentation may be necessary.

<!-- gh-comment-id:2581642801 --> @rick-github commented on GitHub (Jan 10, 2025): Also note that you probably want a `TEMPLATE` as well. If you look at the model settings for llama3.1 that's [in the library](https://ollama.com/library/llama3.1) that will give you a starting point. Because you have fine-tuned the model you probably can't just drop in the values from the ollama library, some experimentation may be necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51875
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?