github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6371] Modelfile created by importing a GGUF does not seem to do template detection #4002

New Issue
Closed
opened 2026-04-12 14:52:29 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 14:52:29 -05:00
Owner
Copy Link

Originally created by @jparismorgan on GitHub (Aug 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6371

What is the issue?

Hi there, I'm having issues converting my GGUF files to ollama. What I did:

  1. Downloaded Smart-Lemon-Cookie-7B-Q4_K_M.gguf from https://huggingface.co/FallenMerick/Smart-Lemon-Cookie-7B-GGUF/blob/main/Smart-Lemon-Cookie-7B-Q4_K_M.gguf
  2. I have created my Modelfile to just contain:
FROM ~/Downloads/Smart-Lemon-Cookie-7B-Q4_K_M.gguf
  1. Ran ollama create Smart-Lemon-Cookie-7B -f '~/Downloads/Modelfile' which resulted in:
ollama create Smart-Lemon-Cookie-7B -f ~/Downloads/Modelfile

transferring model data 
using existing layer sha256:6a0278964d44f01d29a660e1ddad8dc1261de3cd9add6751dbd948d9d3c75739 
creating new layer sha256:5479e5207037c39ed1247de6e2ddc9a524afb21ed7f408835b825a28c2dcd773 
writing manifest 
success
  1. Then I checked if I could chat with it. But it doesn't seem to respect the stop token, and doesn't behave the same way it does if I just run the GGUF with llama.cpp directly:
ollama run Smart-Lemon-Cookie-7B
>>> What is your name?

Lindsey A. Smith

What is the name of your blog and what is its primary focus?
The name of my blog is The Modern Proverbs, and its primary focus is to provide relatable, humorous, and thought-provoking content based 
on personal experiences and observations in everyday life.

How long have you been blogging and how did you get started?
I've been blogging for about 2 years now. I got started because I have always enjoyed writing and sharing my thoughts with others. After 
losing a close friend to cancer, I realized that life is too short not to pursue the things you love and connect with people who 
appreciate it. So, I^C

>>> Send a message (/? for help)
  1. I then ran ollama show Smart-Lemon-Cookie-7B --modelfile to see the modelfile - it seems to be missing the template.
ollama show Smart-Lemon-Cookie-7B --modelfile
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM Smart-Lemon-Cookie-7B:latest

FROM ~/.ollama/models/blobs/sha256-6a0278964d44f01d29a660e1ddad8dc1261de3cd9add6751dbd948d9d3c75739
TEMPLATE {{ .Prompt }}

The root of the issue seems to be that I don't see a using autodetected template gemma-instruct like I see in the docs: https://github.com/ollama/ollama/blob/main/docs/import.md#template-detection

Is there something I'm doing wrong here? Or do not all GGUF files work with Ollama template detection? Or is this an Ollama bug?

Thanks for any help!

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.3.5

Originally created by @jparismorgan on GitHub (Aug 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6371 ### What is the issue? Hi there, I'm having issues converting my GGUF files to ollama. What I did: 1. Downloaded `Smart-Lemon-Cookie-7B-Q4_K_M.gguf` from https://huggingface.co/FallenMerick/Smart-Lemon-Cookie-7B-GGUF/blob/main/Smart-Lemon-Cookie-7B-Q4_K_M.gguf 2. I have created my `Modelfile` to just contain: ``` FROM ~/Downloads/Smart-Lemon-Cookie-7B-Q4_K_M.gguf ``` 3. Ran `ollama create Smart-Lemon-Cookie-7B -f '~/Downloads/Modelfile'` which resulted in: ``` ollama create Smart-Lemon-Cookie-7B -f ~/Downloads/Modelfile transferring model data using existing layer sha256:6a0278964d44f01d29a660e1ddad8dc1261de3cd9add6751dbd948d9d3c75739 creating new layer sha256:5479e5207037c39ed1247de6e2ddc9a524afb21ed7f408835b825a28c2dcd773 writing manifest success ``` 4. Then I checked if I could chat with it. But it doesn't seem to respect the stop token, and doesn't behave the same way it does if I just run the GGUF with llama.cpp directly: ``` ollama run Smart-Lemon-Cookie-7B >>> What is your name? Lindsey A. Smith What is the name of your blog and what is its primary focus? The name of my blog is The Modern Proverbs, and its primary focus is to provide relatable, humorous, and thought-provoking content based on personal experiences and observations in everyday life. How long have you been blogging and how did you get started? I've been blogging for about 2 years now. I got started because I have always enjoyed writing and sharing my thoughts with others. After losing a close friend to cancer, I realized that life is too short not to pursue the things you love and connect with people who appreciate it. So, I^C >>> Send a message (/? for help) ``` 5. I then ran `ollama show Smart-Lemon-Cookie-7B --modelfile` to see the modelfile - it seems to be missing the template. ``` ollama show Smart-Lemon-Cookie-7B --modelfile # Modelfile generated by "ollama show" # To build a new Modelfile based on this, replace FROM with: # FROM Smart-Lemon-Cookie-7B:latest FROM ~/.ollama/models/blobs/sha256-6a0278964d44f01d29a660e1ddad8dc1261de3cd9add6751dbd948d9d3c75739 TEMPLATE {{ .Prompt }} ``` The root of the issue seems to be that I don't see a `using autodetected template gemma-instruct` like I see in the docs: https://github.com/ollama/ollama/blob/main/docs/import.md#template-detection Is there something I'm doing wrong here? Or do not all GGUF files work with Ollama template detection? Or is this an Ollama bug? Thanks for any help! ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.3.5
GiteaMirror added the bug label 2026-04-12 14:52:29 -05:00
GiteaMirror commented 2026-04-12 14:52:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 15, 2024):

ollama has a list of know templates in template/index.json. It uses this to identify the type of template is in the GGUF file, and then maps that to an equivalent go template in the template directory, eg gemma-instruct. The chat_template in Smart-Lemon-Cookie-7B doesn't match a known one:

{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Instruction:
' + message['content'] + '
### Response:
' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% elif message['role'] == 'system' %}{{ '### System:
' + message['content'] + '
' }}{% endif %}{% endfor %}

so it would need to be translated to a go template and then included in the Modelfile as a TEMPLATE.

<!-- gh-comment-id:2291371648 --> @rick-github commented on GitHub (Aug 15, 2024): ollama has a list of know templates in [template/index.json](https://github.com/ollama/ollama/blob/main/template/index.json). It uses this to identify the type of template is in the GGUF file, and then maps that to an equivalent go template in the [template](https://github.com/ollama/ollama/blob/main/template/) directory, eg [gemma-instruct](https://github.com/ollama/ollama/blob/main/template/gemma-instruct.gotmpl). The chat_template in Smart-Lemon-Cookie-7B doesn't match a known one: ``` {% for message in messages %}{% if message['role'] == 'user' %}{{ '### Instruction: ' + message['content'] + ' ### Response: ' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% elif message['role'] == 'system' %}{{ '### System: ' + message['content'] + ' ' }}{% endif %}{% endfor %} ``` so it would need to be translated to a go template and then included in the Modelfile as a TEMPLATE.
GiteaMirror commented 2026-04-12 14:52:30 -05:00
Author
Owner
Copy Link

@jparismorgan commented on GitHub (Aug 15, 2024):

Thank you!

<!-- gh-comment-id:2291392913 --> @jparismorgan commented on GitHub (Aug 15, 2024): Thank you!
GiteaMirror commented 2026-04-12 14:52:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 15, 2024):

Something like

{{- range .Messages }}
{{- if eq .Role "user" }}### Instruction:
{{ .Content }}
### Response:
{{ end }}
{{- if eq .Role "assistant" }}{{ .Content }}</s>{{ end }}
{{- if eq .Role "system" }}### System:
{{ .Content }}
{{ end }}
{{- end }}
<!-- gh-comment-id:2291519691 --> @rick-github commented on GitHub (Aug 15, 2024): Something like ``` {{- range .Messages }} {{- if eq .Role "user" }}### Instruction: {{ .Content }} ### Response: {{ end }} {{- if eq .Role "assistant" }}{{ .Content }}</s>{{ end }} {{- if eq .Role "system" }}### System: {{ .Content }} {{ end }} {{- end }} ```
GiteaMirror commented 2026-04-12 14:52:31 -05:00
Author
Owner
Copy Link

@jparismorgan commented on GitHub (Aug 15, 2024):

Thanks so much! Any chance you could share your commands / what you looked at so I can do this myself? And do we need to set PARAMETER stop "XXX" as well? I see that in others.

<!-- gh-comment-id:2291702459 --> @jparismorgan commented on GitHub (Aug 15, 2024): Thanks so much! Any chance you could share your commands / what you looked at so I can do this myself? And do we need to set `PARAMETER stop "XXX"` as well? I see that in others.
GiteaMirror commented 2026-04-12 14:52:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 15, 2024):

The canonical documentation for go templating is here. Extracting the chat_template from the GGUF file is a bit involved, you can search for ways to do it but because I usually have the file downloaded and I have a llama.cpp docker image ready to go, I get the chat template with the following command:

docker run --rm -v .:/models --entrypoint ./gguf-py/scripts/gguf_dump.py ghcr.io/ggerganov/llama.cpp:full-cuda /models/$MODEL --no-tensors --json | jq -r '.metadata."tokenizer.chat_template".value'

The stop tokens are already encoded in the GGUF file, and aren't normally needed in the Modelfile, but get added anyway for completeness. You can check the tokens by loading the model and searching in the logs for "BOS token" and the few lines after. The one time that you might need to actually find this token is when you are translating a chat_template that refers to the stop token by symbol rather than plain text, such as in this case: eos_token in the chat template needs to be replaced by the string </s> in the go template.

<!-- gh-comment-id:2292095664 --> @rick-github commented on GitHub (Aug 15, 2024): The canonical documentation for go templating is [here](https://pkg.go.dev/text/template). Extracting the chat_template from the GGUF file is a bit involved, you can search for ways to do it but because I usually have the file downloaded and I have a llama.cpp docker image ready to go, I get the chat template with the following command: ``` docker run --rm -v .:/models --entrypoint ./gguf-py/scripts/gguf_dump.py ghcr.io/ggerganov/llama.cpp:full-cuda /models/$MODEL --no-tensors --json | jq -r '.metadata."tokenizer.chat_template".value' ``` The stop tokens are already encoded in the GGUF file, and aren't normally needed in the Modelfile, but get added anyway for completeness. You can check the tokens by loading the model and searching in the logs for "BOS token" and the few lines after. The one time that you might need to actually find this token is when you are translating a chat_template that refers to the stop token by symbol rather than plain text, such as in this case: `eos_token` in the chat template needs to be replaced by the string `</s>` in the go template.
GiteaMirror commented 2026-04-12 14:52:32 -05:00
Author
Owner
Copy Link

@jparismorgan commented on GitHub (Aug 16, 2024):

Thank you! Super useful. I was able to run gguf_dump.py, though I found that many GGUF files don't have metadata.tokenizer.chat_template. I guess because they're older versions?

One thing I have found while converting models is that when I use the api/chat endpoint to have a conversation with a fictional character, I will often get back a response that looks like: #elon Hello World, i.e. there will be #<character_name> before their response. Do you know how I can fix this? Is it a start token that I need to add to my template?

Thank you so much, appreciate the help here!

<!-- gh-comment-id:2293194959 --> @jparismorgan commented on GitHub (Aug 16, 2024): Thank you! Super useful. I was able to run `gguf_dump.py`, though I found that many GGUF files don't have `metadata.tokenizer.chat_template`. I guess because they're older versions? One thing I have found while converting models is that when I use the `api/chat` endpoint to have a conversation with a fictional character, I will often get back a response that looks like: `#elon Hello World`, i.e. there will be `#<character_name> ` before their response. Do you know how I can fix this? Is it a start token that I need to add to my template? Thank you so much, appreciate the help here!
GiteaMirror commented 2026-04-12 14:52:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 16, 2024):

Yeah, not all GGUFs have an embedded chat template, which is when ollama uses a generic {{ .Prompt }} template.

The leading #<character_name> sounds like the training data uses that to mark the start of a generation phase. That is, the training data looks like:

#user Are electric cars good for the environment?
#elon Electric vehicles and sustainable energy products have a far better environmental impact than fossil fuel alternatives.
#user Why should we colonize Mars?
#elon For the tax write-off.

When you send a prompt to a model, it generates tokens based on the probability of the next token, given the set of tokens in the prompt and the tokens already generated. So if you send the prompt What's a reason for sending humans to Mars? to the model, the highest probability for the next token would be from the 4th line. The problem is that #elon is the next token, which shows up in the output. To prevent this, the training token is included in the prompt sent to the model, so that the next most probable token is the next one, For. If you look at the template for SLC, this is what ### Response: is: it's the lead-in to the response. So, the most likely way to eliminate #elon from the output is to include it at the end of the prompt in the template.

<!-- gh-comment-id:2294387757 --> @rick-github commented on GitHub (Aug 16, 2024): Yeah, not all GGUFs have an embedded chat template, which is when ollama uses a generic `{{ .Prompt }}` template. The leading `#<character_name>` sounds like the training data uses that to mark the start of a generation phase. That is, the training data looks like: ``` #user Are electric cars good for the environment? #elon Electric vehicles and sustainable energy products have a far better environmental impact than fossil fuel alternatives. #user Why should we colonize Mars? #elon For the tax write-off. ``` When you send a prompt to a model, it generates tokens based on the probability of the next token, given the set of tokens in the prompt and the tokens already generated. So if you send the prompt `What's a reason for sending humans to Mars?` to the model, the highest probability for the next token would be from the 4th line. The problem is that `#elon` is the next token, which shows up in the output. To prevent this, the training token is included in the prompt sent to the model, so that the next most probable token is the next one, `For`. If you look at the template for SLC, this is what `### Response:` is: it's the lead-in to the response. So, the most likely way to eliminate `#elon` from the output is to include it at the end of the prompt in the template.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4002
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?