github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1110] Converting mistral7b finetuned model produces garbage output #78233

New Issue
Closed
opened 2026-05-08 22:23:49 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-08 22:23:49 -05:00
Owner
Copy Link

Originally created by @amritap-ef on GitHub (Nov 13, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1110

Hi,

I've finetuned a Mistral-7B-v0.1 model from hugging face using QLoRa as per the typical SFT scripts released by the hugging face team. However, after converting it to gguf format using the docker method, I found that my model produces garbage.

For reference, this is the format of the prompts I trained on:

### User:
{{user prompt}}

### Assistant:
{{assistant response}}
</s>

The inference results from hugging face look ok, I struggle with getting the model to stop generating but setting max_tokens generally solves this (although an unideal workaround).

But when converting to gguf and running via Ollama, the model often goes off-task and does its own thing. Furthermore, unless I put '### Assistant:' and '' as stop tokens, it often produces input interleaving with lots of '### Assistant''s e.g.:

We've noticed that some people love to go sightseeing in Paris and we have a special offer for this year!

### Assistant:
I love Paris

### Assistant:
I love Paris.

So I have a few questions/paths to investigate:

  1. Has anyone been able to successfully fine-tune a model and convert it to gguf and get it to work well with Ollama?
  2. Could it be that I'm doing something wrong in converting the mistral finetuned model? I save the merged lora model and run the docker converting to gguf via llama.cpp on that merged model as per normal.
  3. Could it be that I need to produce a better 'base' fine-tuned model before converting to gguf for it to work well in Ollama?

Note, I also tried converting a base mistral7b model downloaded from hugging face as as sanity check and that worked great.

Originally created by @amritap-ef on GitHub (Nov 13, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1110 Hi, I've finetuned a Mistral-7B-v0.1 model from hugging face using QLoRa as per the typical SFT scripts released by the hugging face team. However, after converting it to gguf format using the docker method, I found that my model produces garbage. For reference, this is the format of the prompts I trained on: ```<s> ### User: {{user prompt}} ### Assistant: {{assistant response}} </s> ``` The inference results from hugging face look ok, I struggle with getting the model to stop generating but setting max_tokens generally solves this (although an unideal workaround). But when converting to gguf and running via Ollama, the model often goes off-task and does its own thing. Furthermore, unless I put '### Assistant:' and '</s>' as stop tokens, it often produces input interleaving with lots of '### Assistant''s e.g.: ``` We've noticed that some people love to go sightseeing in Paris and we have a special offer for this year! ### Assistant: I love Paris ### Assistant: I love Paris. ``` So I have a few questions/paths to investigate: 1. Has anyone been able to successfully fine-tune a model and convert it to gguf and get it to work well with Ollama? 2. Could it be that I'm doing something wrong in converting the mistral finetuned model? I save the merged lora model and run the docker converting to gguf via llama.cpp on that merged model as per normal. 3. Could it be that I need to produce a better 'base' fine-tuned model before converting to gguf for it to work well in Ollama? Note, I also tried converting a base mistral7b model downloaded from hugging face as as sanity check and that worked great.
GiteaMirror added the question label 2026-05-08 22:23:49 -05:00
GiteaMirror commented 2026-05-08 22:23:56 -05:00
Author
Owner
Copy Link

@65a commented on GitHub (Nov 17, 2023):

I've used fine-tuned models converted from safetensors with ollama successfully, but I usually check out the latest ggerganov/llama.cpp and run the convert.py and quantize.py from the repo, because I was burned by some gguf bugs in the past. You might want to try this, but your description of needing to use stop tokens sounds normal to me, most models will continue for some time without explicit stop tokens.

<!-- gh-comment-id:1815648813 --> @65a commented on GitHub (Nov 17, 2023): I've used fine-tuned models converted from safetensors with ollama successfully, but I usually check out the latest ggerganov/llama.cpp and run the convert.py and quantize.py from the repo, because I was burned by some gguf bugs in the past. You might want to try this, but your description of needing to use stop tokens sounds normal to me, most models will continue for some time without explicit stop tokens.
GiteaMirror commented 2026-05-08 22:23:56 -05:00
Author
Owner
Copy Link

@PhilipAmadasun commented on GitHub (Jan 16, 2024):

@65a @amritap-ef I'm having a hard time understanding what I'm doing wrong to upload fine-tuned models to ollama. I had recently created fine-tuned layers on google-collab here. I run:

curl http://server.local:11434/api/create -d '{
  "name": "test_mistral",
  "modelfile": "FROM mistral\nADAPTER /home/robot/adapter_model.bin"
}'

which creates a fine-tuned model that doesn't work. I get an error about the fine-tuned layers. I don't know what to do and scowering the internet hasn't gotten me closer to figuring that out. Please any help would be greatly appreciated.

<!-- gh-comment-id:1893038606 --> @PhilipAmadasun commented on GitHub (Jan 16, 2024): @65a @amritap-ef I'm having a hard time understanding what I'm doing wrong to upload fine-tuned models to ollama. I had recently created fine-tuned layers on google-collab [here](https://huggingface.co/uyiosa/test_mistral_7b/tree/main). I run: ``` curl http://server.local:11434/api/create -d '{ "name": "test_mistral", "modelfile": "FROM mistral\nADAPTER /home/robot/adapter_model.bin" }' ``` which creates a fine-tuned model that doesn't work. I get an error about the fine-tuned layers. I don't know what to do and scowering the internet hasn't gotten me closer to figuring that out. Please any help would be greatly appreciated.
GiteaMirror commented 2026-05-08 22:23:57 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Oct 16, 2024):

Just found this issue again. There are now docs which explain how to do this here and a lot better tutorials (on the web/youtube) on how to do this properly. I'm going to go ahead and close out the issue since it's now a year old 😅

<!-- gh-comment-id:2417928659 --> @pdevine commented on GitHub (Oct 16, 2024): Just found this issue again. There are now docs which explain how to do this [here](https://github.com/ollama/ollama/blob/main/docs/import.md) and a lot better tutorials (on the web/youtube) on how to do this properly. I'm going to go ahead and close out the issue since it's now a year old 😅
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#78233
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?