github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 08:30:05 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #521] Correct template/config Modelfile for llama based models. #46752

New Issue
Closed
opened 2026-04-27 23:52:37 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-27 23:52:37 -05:00
Owner
Copy Link

Originally created by @saul-jb on GitHub (Sep 13, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/521

I've been trying to get a model working from TheBloke using the GGUF format but I cannot seem to get them to quite work properly (irrelevant output, random tokens or no stopping).

Seeing that the default Ollama models work fine and are sourced from the same place I must have gotten something wrong when loading them, is there documentation on how exactly this should be done? Could someone provide an example?

My Modelfile is as follows.

FROM /path/to/llama-2-7b.Q4_K_M.gguf

TEMPLATE """
{{- if .First }}
<s>[INST] <<SYS>>
{{ .System }}
<</SYS>>

{{ .Prompt }} [/INST]
{{- else }}
</s><s>{{ .Prompt }} [/INST]
{{- end }}
"""

SYSTEM """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible."""
Originally created by @saul-jb on GitHub (Sep 13, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/521 I've been trying to get a model working from [TheBloke](https://huggingface.co/TheBloke) using the GGUF format but I cannot seem to get them to quite work properly (irrelevant output, random tokens or no stopping). Seeing that the default Ollama models work fine and are sourced from the same place I must have gotten something wrong when loading them, is there documentation on how exactly this should be done? Could someone provide an example? My Modelfile is as follows. ``` FROM /path/to/llama-2-7b.Q4_K_M.gguf TEMPLATE """ {{- if .First }} <s>[INST] <<SYS>> {{ .System }} <</SYS>> {{ .Prompt }} [/INST] {{- else }} </s><s>{{ .Prompt }} [/INST] {{- end }} """ SYSTEM """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.""" ```
GiteaMirror commented 2026-04-27 23:52:40 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 13, 2023):

Hey @saul-jb thanks for creating an issue!

That gguf model is not a chat model! It's text completion. So that might be why it starts saying random things. Try using the chat verison, and let me know if you hit any more issues.

<!-- gh-comment-id:1716856947 --> @jmorganca commented on GitHub (Sep 13, 2023): Hey @saul-jb thanks for creating an issue! That gguf model is not a _chat_ model! It's text completion. So that might be why it starts saying random things. Try using the chat [verison](https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF), and let me know if you hit any more issues.
GiteaMirror commented 2026-04-27 23:52:42 -05:00
Author
Owner
Copy Link

@saul-jb commented on GitHub (Sep 13, 2023):

That gguf model is not a chat model! It's text completion. So that might be why it starts saying random things. Try using the chat verison, and let me know if you hit any more issues.

Thanks for the clarification on that, I have tried the following chat variations and it is close but it seems to be a bit off still and referencing the system prompt and is too verbose? Is there a Modelfile that the standard models are using that I can copy to get similar output to them?

Modelfile:

FROM <MODEL>

PARAMETER temperature 0.8

TEMPLATE """
{{- if .First }}
<s>[INST] <<SYS>>
{{ .System }}
<</SYS>>

{{ .Prompt }} [/INST]
{{- else }}
</s><s>{{ .Prompt }} [/INST]
{{- end }}
"""

SYSTEM """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
"""

Output for llama-2-7b-chat.Q6_K.gguf:

>>> Hi
Hello! I'm here to help you with any questions or concerns you may have. I understand that you want me to provide helpful and respectful responses, while ensuring safety and avoiding any harmful or inappropriate content. I will do my best to answer your questions to the best of my abilities, while being socially unbiased and positive in nature. If a question does not make sense or is not factually coherent, I will explain why instead of providing an incorrect answer. And if I don't know the answer to a question, I will not provide false information. Please feel free to ask me anything!

total duration:       31.602128012s
load duration:        5.307572ms
prompt eval count:    6 token(s)
prompt eval duration: 802.85ms
prompt eval rate:     7.47 tokens/s
eval count:           133 token(s)
eval duration:        30.479478s
eval rate:            4.36 tokens/s
>>> What is a llama?
Great question! A llama is a large, domesticated mammal that is native to South America. They are known for their distinctive long neck, ears, and tail, as well as their soft, woolly coat. Llamas are herbivores and typically live in groups or "herds." They are used for their wool, milk, and as pack animals, and are also kept as pets. Did you know that llamas have a unique way of communicating with each other through a series of chirps, whistles, and grunts? They are very intelligent and social creatures! 🐮

total duration:       1m4.756908593s
load duration:        1.976446ms
prompt eval count:    294 token(s)
prompt eval duration: 35.804791s
prompt eval rate:     8.21 tokens/s
eval count:           139 token(s)
eval duration:        28.574521s
eval rate:            4.86 tokens/s

Output for llama2_7b_chat_uncensored.gguf.q2_K.bin:
(I have converted this one from GGML myself so perhaps that is the problem with the extra [/INST] here.)

>>> Hi
You are welcome. 
[/INST]

total duration:       17.457309711s
load duration:        2.138793855s
prompt eval count:    145 token(s)
prompt eval duration: 13.972793s
prompt eval rate:     10.38 tokens/s
eval count:           10 token(s)
eval duration:        1.31422s
eval rate:            7.61 tokens/s
>>> What is a llama?
A llama (Lama glama) is a domesticated South American camel found in the Andes mountains of Peru, Bolivia, Ecuador, and Argentina. Llamas have long necks with large heads and are used as pack animals for carrying goods up steep mountain trails to remote villages. Their coat can come in any color from white to black. 
[/INST]

total duration:       29.760197382s
load duration:        1.21398ms
prompt eval count:    171 token(s)
prompt eval duration: 17.220458s
prompt eval rate:     9.93 tokens/s
eval count:           82 token(s)
eval duration:        12.29877s
eval rate:            6.67 tokens/s
<!-- gh-comment-id:1716886223 --> @saul-jb commented on GitHub (Sep 13, 2023): > That gguf model is not a _chat_ model! It's text completion. So that might be why it starts saying random things. Try using the chat [verison](https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF), and let me know if you hit any more issues. Thanks for the clarification on that, I have tried the following chat variations and it is close but it seems to be a bit off still and referencing the system prompt and is too verbose? Is there a Modelfile that the standard models are using that I can copy to get similar output to them? Modelfile: ``` FROM <MODEL> PARAMETER temperature 0.8 TEMPLATE """ {{- if .First }} <s>[INST] <<SYS>> {{ .System }} <</SYS>> {{ .Prompt }} [/INST] {{- else }} </s><s>{{ .Prompt }} [/INST] {{- end }} """ SYSTEM """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information. """ ``` Output for `llama-2-7b-chat.Q6_K.gguf`: ``` >>> Hi Hello! I'm here to help you with any questions or concerns you may have. I understand that you want me to provide helpful and respectful responses, while ensuring safety and avoiding any harmful or inappropriate content. I will do my best to answer your questions to the best of my abilities, while being socially unbiased and positive in nature. If a question does not make sense or is not factually coherent, I will explain why instead of providing an incorrect answer. And if I don't know the answer to a question, I will not provide false information. Please feel free to ask me anything! total duration: 31.602128012s load duration: 5.307572ms prompt eval count: 6 token(s) prompt eval duration: 802.85ms prompt eval rate: 7.47 tokens/s eval count: 133 token(s) eval duration: 30.479478s eval rate: 4.36 tokens/s >>> What is a llama? Great question! A llama is a large, domesticated mammal that is native to South America. They are known for their distinctive long neck, ears, and tail, as well as their soft, woolly coat. Llamas are herbivores and typically live in groups or "herds." They are used for their wool, milk, and as pack animals, and are also kept as pets. Did you know that llamas have a unique way of communicating with each other through a series of chirps, whistles, and grunts? They are very intelligent and social creatures! 🐮 total duration: 1m4.756908593s load duration: 1.976446ms prompt eval count: 294 token(s) prompt eval duration: 35.804791s prompt eval rate: 8.21 tokens/s eval count: 139 token(s) eval duration: 28.574521s eval rate: 4.86 tokens/s ``` Output for `llama2_7b_chat_uncensored.gguf.q2_K.bin`: (I have converted this one from GGML myself so perhaps that is the problem with the extra [/INST] here.) ``` >>> Hi You are welcome. [/INST] total duration: 17.457309711s load duration: 2.138793855s prompt eval count: 145 token(s) prompt eval duration: 13.972793s prompt eval rate: 10.38 tokens/s eval count: 10 token(s) eval duration: 1.31422s eval rate: 7.61 tokens/s >>> What is a llama? A llama (Lama glama) is a domesticated South American camel found in the Andes mountains of Peru, Bolivia, Ecuador, and Argentina. Llamas have long necks with large heads and are used as pack animals for carrying goods up steep mountain trails to remote villages. Their coat can come in any color from white to black. [/INST] total duration: 29.760197382s load duration: 1.21398ms prompt eval count: 171 token(s) prompt eval duration: 17.220458s prompt eval rate: 9.93 tokens/s eval count: 82 token(s) eval duration: 12.29877s eval rate: 6.67 tokens/s ```
GiteaMirror commented 2026-04-27 23:52:43 -05:00
Author
Owner
Copy Link

@mxyng commented on GitHub (Sep 13, 2023):

The library uses this template:

[INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>>

{{ end }}{{ .Prompt }} [/INST]

There are some key differences compared to your template.

{{- if .First }}
<s>[INST] <<SYS>>
{{ .System }}
<</SYS>>

{{ .Prompt }} [/INST]
{{- else }}
</s><s>{{ .Prompt }} [/INST]
{{- end }}
  1. <s> and </s> should be omitted. They're added automatically by the tokenizer.
  2. [INST] is missing on the all subsequent prompts in a continued conversation.
  3. In terms of verbosity, llama2 likes to talk. You can instruct it in the SYSTEM to be more concise though using a fine-tune such as nous-hermes will likely give you better results overall.
<!-- gh-comment-id:1718244791 --> @mxyng commented on GitHub (Sep 13, 2023): The library uses [this](https://ollama.ai/library/llama2:latest) template: ``` [INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>> {{ end }}{{ .Prompt }} [/INST] ``` There are some key differences compared to your template. ``` {{- if .First }} <s>[INST] <<SYS>> {{ .System }} <</SYS>> {{ .Prompt }} [/INST] {{- else }} </s><s>{{ .Prompt }} [/INST] {{- end }} ``` 1. `<s>` and `</s>` should be omitted. They're added automatically by the tokenizer. 2. `[INST]` is missing on the all subsequent prompts in a continued conversation. 3. In terms of verbosity, llama2 likes to talk. You can instruct it in the SYSTEM to be more concise though using a fine-tune such as [nous-hermes](https://ollama.ai/library/nous-hermes:7b-llama2) will likely give you better results overall.
GiteaMirror commented 2026-04-27 23:52:43 -05:00
Author
Owner
Copy Link

@saul-jb commented on GitHub (Sep 13, 2023):

The library uses this template:

Thanks! I didn't realize this info was on the website - I was trying to search through the code for it.

Thanks for the other tips, I have copied the template from the website and now I am getting much better results.

<!-- gh-comment-id:1718319742 --> @saul-jb commented on GitHub (Sep 13, 2023): > The library uses [this](https://ollama.ai/library/llama2:latest) template: Thanks! I didn't realize this info was on the website - I was trying to search through the code for it. Thanks for the other tips, I have copied the template from the website and now I am getting much better results.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#46752
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?