github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8982] Missing default TEMPLATE and PARAMETER when importing from GGUF #5830

New Issue
Closed
opened 2026-04-12 17:10:27 -05:00 by GiteaMirror · 12 comments
GiteaMirror commented 2026-04-12 17:10:27 -05:00
Owner
Copy Link

Originally created by @mozophe on GitHub (Feb 10, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8982

What is the issue?

I imported a GGUF of DeepSeek-R1-Qwen2.5-14B in Ollama but it did not set the default template and parameters properly. I have noticed the same problem for Mistral based GGUFs.

It would be easier to explain with example:

Modelfile:
FROM "C:\Users\singh.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF\DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf"

PS C:\Users\singh\.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF> ollama create DeepSeek-R1-Distill-Qwen-14B

gathering model components
copying file sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c 100%
parsing GGUF
using existing layer sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c
writing manifest
success
PS C:\Users\singh\.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF> ollama show --modelfile DeepSeek-R1-Distill-Qwen-14B
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM DeepSeek-R1-Distill-Qwen-14B:latest

FROM C:\Users\singh\.ollama\models\blobs\sha256-67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c
TEMPLATE {{ .Prompt }}

The modelfile is incomplete.

The template and parameters should be same as here: https://ollama.com/library/deepseek-r1:14b (it's the exact same quant Q4_K_M)

Template:

{{- if .System }}{{ .System }}{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1}}
{{- if eq .Role "user" }}<｜User｜>{{ .Content }}
{{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }}
{{- end }}
{{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }}
{{- end }}

Parameters: (not in the same format, but the information should be present in the modelfile using multiple PARAMETER stop)

{
    "stop": [
        "<｜begin▁of▁sentence｜>",
        "<｜end▁of▁sentence｜>",
        "<｜User｜>",
        "<｜Assistant｜>"
    ]
}

I would be happy to provide any information required to resolve this issue.

BTW, I understand, a temporary fix for this would be to create a modelfile with this info but I don't know how to format the modelfile properly to represent these template and parameters. I would be grateful if someone could guide me on how to use the above information to properly make a modelfile. Alternatively, I can download this model using ollama, but the problem remains for other GGUFs that needs to be downloaded from huggingface, because they are not available in the list of ollama models.

Relevant log output


app.log
time=2025-02-10T06:20:53.773+01:00 level=INFO source=logging.go:50 msg="ollama app started"
time=2025-02-10T06:20:53.774+01:00 level=INFO source=lifecycle.go:19 msg="app config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\singh\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-02-10T06:20:53.789+01:00 level=INFO source=server.go:182 msg="unable to connect to server"
time=2025-02-10T06:20:53.789+01:00 level=INFO source=server.go:141 msg="starting server..."
time=2025-02-10T06:20:53.791+01:00 level=INFO source=server.go:127 msg="started ollama server with pid 5304"
time=2025-02-10T06:20:53.791+01:00 level=INFO source=server.go:129 msg="ollama server logs C:\\Users\\singh\\AppData\\Local\\Ollama\\server.log"
time=2025-02-10T06:21:26.505+01:00 level=INFO source=lifecycle.go:89 msg="Waiting for ollama server to shutdown..."
time=2025-02-10T06:21:26.523+01:00 level=INFO source=server.go:158 msg="server shutdown with exit code 0"
time=2025-02-10T06:21:26.523+01:00 level=INFO source=lifecycle.go:93 msg="Ollama app exiting"

server.log
2025/02/10 06:20:53 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\singh\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-02-10T06:20:53.830+01:00 level=INFO source=images.go:432 msg="total blobs: 10"
time=2025-02-10T06:20:53.831+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-10T06:20:53.831+01:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)"
time=2025-02-10T06:20:53.831+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2]"
time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:183 msg="efficiency cores detected" maxEfficiencyClass=1
time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=24 efficiency=16 threads=32
time=2025-02-10T06:20:54.492+01:00 level=INFO source=gpu.go:334 msg="detected OS VRAM overhead" id=GPU-c12cc271-6678-7402-41e4-9022071ed3cd library=cuda compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 Laptop GPU" overhead="1000.7 MiB"
time=2025-02-10T06:20:54.493+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-c12cc271-6678-7402-41e4-9022071ed3cd library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 Laptop GPU" total="16.0 GiB" available="14.7 GiB"
[GIN] 2025/02/10 - 06:20:54 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/10 - 06:21:11 | 201 |   11.2701975s |       127.0.0.1 | POST     "/api/blobs/sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c"
[GIN] 2025/02/10 - 06:21:12 | 200 |    117.5335ms |       127.0.0.1 | POST     "/api/create"
[GIN] 2025/02/10 - 06:21:18 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/10 - 06:21:18 | 200 |     21.1376ms |       127.0.0.1 | POST     "/api/show"

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.5.7

Originally created by @mozophe on GitHub (Feb 10, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8982 ### What is the issue? I imported a GGUF of DeepSeek-R1-Qwen2.5-14B in Ollama but it did not set the default template and parameters properly. I have noticed the same problem for Mistral based GGUFs. It would be easier to explain with example: Modelfile: FROM "C:\Users\singh\.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF\DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf" ```powershell PS C:\Users\singh\.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF> ollama create DeepSeek-R1-Distill-Qwen-14B gathering model components copying file sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c 100% parsing GGUF using existing layer sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c writing manifest success PS C:\Users\singh\.lmstudio\models\unsloth\DeepSeek-R1-Distill-Qwen-14B-GGUF> ollama show --modelfile DeepSeek-R1-Distill-Qwen-14B # Modelfile generated by "ollama show" # To build a new Modelfile based on this, replace FROM with: # FROM DeepSeek-R1-Distill-Qwen-14B:latest FROM C:\Users\singh\.ollama\models\blobs\sha256-67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c TEMPLATE {{ .Prompt }} ``` The modelfile is incomplete. The template and parameters should be same as here: https://ollama.com/library/deepseek-r1:14b (it's the exact same quant Q4_K_M) Template: ```powershell {{- if .System }}{{ .System }}{{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1}} {{- if eq .Role "user" }}<｜User｜>{{ .Content }} {{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }} {{- end }} {{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }} {{- end }} ``` Parameters: (not in the same format, but the information should be present in the modelfile using multiple PARAMETER stop) ```powershell { "stop": [ "<｜begin▁of▁sentence｜>", "<｜end▁of▁sentence｜>", "<｜User｜>", "<｜Assistant｜>" ] } ``` I would be happy to provide any information required to resolve this issue. BTW, I understand, a temporary fix for this would be to create a modelfile with this info but I don't know how to format the modelfile properly to represent these template and parameters. I would be grateful if someone could guide me on how to use the above information to properly make a modelfile. Alternatively, I can download this model using ollama, but the problem remains for other GGUFs that needs to be downloaded from huggingface, because they are not available in the list of ollama models. ### Relevant log output ```shell app.log time=2025-02-10T06:20:53.773+01:00 level=INFO source=logging.go:50 msg="ollama app started" time=2025-02-10T06:20:53.774+01:00 level=INFO source=lifecycle.go:19 msg="app config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\singh\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-02-10T06:20:53.789+01:00 level=INFO source=server.go:182 msg="unable to connect to server" time=2025-02-10T06:20:53.789+01:00 level=INFO source=server.go:141 msg="starting server..." time=2025-02-10T06:20:53.791+01:00 level=INFO source=server.go:127 msg="started ollama server with pid 5304" time=2025-02-10T06:20:53.791+01:00 level=INFO source=server.go:129 msg="ollama server logs C:\\Users\\singh\\AppData\\Local\\Ollama\\server.log" time=2025-02-10T06:21:26.505+01:00 level=INFO source=lifecycle.go:89 msg="Waiting for ollama server to shutdown..." time=2025-02-10T06:21:26.523+01:00 level=INFO source=server.go:158 msg="server shutdown with exit code 0" time=2025-02-10T06:21:26.523+01:00 level=INFO source=lifecycle.go:93 msg="Ollama app exiting" server.log 2025/02/10 06:20:53 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\singh\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-02-10T06:20:53.830+01:00 level=INFO source=images.go:432 msg="total blobs: 10" time=2025-02-10T06:20:53.831+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-10T06:20:53.831+01:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)" time=2025-02-10T06:20:53.831+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2]" time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:183 msg="efficiency cores detected" maxEfficiencyClass=1 time=2025-02-10T06:20:53.831+01:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=24 efficiency=16 threads=32 time=2025-02-10T06:20:54.492+01:00 level=INFO source=gpu.go:334 msg="detected OS VRAM overhead" id=GPU-c12cc271-6678-7402-41e4-9022071ed3cd library=cuda compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 Laptop GPU" overhead="1000.7 MiB" time=2025-02-10T06:20:54.493+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-c12cc271-6678-7402-41e4-9022071ed3cd library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 Laptop GPU" total="16.0 GiB" available="14.7 GiB" [GIN] 2025/02/10 - 06:20:54 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/02/10 - 06:21:11 | 201 | 11.2701975s | 127.0.0.1 | POST "/api/blobs/sha256:67a7933cf2ad596a393c8e13b30bc4da2d50b283e250b78554aed18817eca31c" [GIN] 2025/02/10 - 06:21:12 | 200 | 117.5335ms | 127.0.0.1 | POST "/api/create" [GIN] 2025/02/10 - 06:21:18 | 200 | 0s | 127.0.0.1 | HEAD "/" [GIN] 2025/02/10 - 06:21:18 | 200 | 21.1376ms | 127.0.0.1 | POST "/api/show" ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.7
GiteaMirror added the bug label 2026-04-12 17:10:27 -05:00
GiteaMirror commented 2026-04-12 17:10:28 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

https://github.com/ollama/ollama/blob/main/docs/modelfile.md

Or pull from HF using the Use this model feature in the upper right of the model page.

ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M
<!-- gh-comment-id:2647391470 --> @rick-github commented on GitHub (Feb 10, 2025): https://github.com/ollama/ollama/blob/main/docs/modelfile.md Or pull from HF using the `Use this model` feature in the upper right of the model page. ``` ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M ```
GiteaMirror commented 2026-04-12 17:10:28 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 10, 2025):

https://github.com/ollama/ollama/blob/main/docs/modelfile.md

Or pull from HF using the Use this model feature in the upper right of the model page.

ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M

Thank you for the resources. As I mentioned earlier, this temporary fix would work for the DeepSeek model that I am trying to run. But the issue is that this method is only limited models that can be found on ollama model repo.

A better temporary fix would be to learn to make the appropriate modelfile (as you mentioned: https://github.com/ollama/ollama/blob/main/docs/modelfile.md).

There are multiple models, from huggingface, that provide a system prompt such as

<|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|>

or

<|im_start|>system
You're a masterful storyteller and gamemaster. Write in second person present tense (You are), crafting vivid, engaging narratives with authority and confidence.<|im_end|>
<|im_start|>user
> You peer into the darkness.<|im_end|>
<|im_start|>assistant
You have been eaten by a grue.

GAME OVER<|im_end|>

I would like to learn how to turn this into a format (Go syntax) that I could put in the modelfile along with the respective stop parameters. The example mentioned in the link is not sufficient for me to properly grasp how to modify the above to proper format.

Further, there are several other models that are neither present on ollama model repo, nor do they have information about the template on their model page. One way to temporarily fix that would be to identify the base model from which the finetune was created, find the base model on the ollama model repo, get template from there, put it in the modelfile (in proper syntax, including pararmeter stop).

I think it's quite clear that it requires some work. Same GGUFs work totally fine on LM Studio, but I have a soft spot for Ollama as it is opensource. I would prefer to have all my models on ollama.

<!-- gh-comment-id:2648578377 --> @mozophe commented on GitHub (Feb 10, 2025): > https://github.com/ollama/ollama/blob/main/docs/modelfile.md > > Or pull from HF using the `Use this model` feature in the upper right of the model page. > > ``` > ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M > ``` Thank you for the resources. As I mentioned earlier, this temporary fix would work for the DeepSeek model that I am trying to run. But the issue is that this method is only limited models that can be found on ollama model repo. A better temporary fix would be to learn to make the appropriate modelfile (as you mentioned: https://github.com/ollama/ollama/blob/main/docs/modelfile.md). There are multiple models, from huggingface, that provide a system prompt such as ```powershell <|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|> ``` or ```powershell <|im_start|>system You're a masterful storyteller and gamemaster. Write in second person present tense (You are), crafting vivid, engaging narratives with authority and confidence.<|im_end|> <|im_start|>user > You peer into the darkness.<|im_end|> <|im_start|>assistant You have been eaten by a grue. GAME OVER<|im_end|> ``` I would like to learn how to turn this into a format (Go syntax) that I could put in the modelfile along with the respective stop parameters. The example mentioned in the link is not sufficient for me to properly grasp how to modify the above to proper format. Further, there are several other models that are neither present on ollama model repo, nor do they have information about the template on their model page. One way to temporarily fix that would be to identify the base model from which the finetune was created, find the base model on the ollama model repo, get template from there, put it in the modelfile (in proper syntax, including pararmeter stop). I think it's quite clear that it requires some work. Same GGUFs work totally fine on LM Studio, but I have a soft spot for Ollama as it is opensource. I would prefer to have all my models on ollama.
GiteaMirror commented 2026-04-12 17:10:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

The source of all templating for models is the Jinja template stored in the chat_template in the GGUF file of a model. If you can find the safetensors versions of the model, the chat_template can be found in the tokenizer_config.json file, eg DeepSeek-R1. Otherwise you can dump the KV table in the GGUF file using llama.cpp and extract the template:

gguf-py/gguf/scripts/gguf_dump.py --json --no-tensors /path/to/model.gguf | jq -r  '.metadata."tokenizer.chat_template".value'

This template is then converted to a Go template used in the TEMPLATE in the Modelfile. The output you show above is the result of the prompt from the client being processed through the Modelfile template.

See index.json for examples of Jinja templates converted to Go templates - the name in the file points at the relevant Go template in the template directory. See Go templates for documentation on writing go templates.

As you mention, the easiest way to add a template is identify a model in the ollama library that has similar architecture and fine tuning to the one you want to import, and use that, perhaps with some modification.

<!-- gh-comment-id:2648644043 --> @rick-github commented on GitHub (Feb 10, 2025): The source of all templating for models is the Jinja template stored in the `chat_template` in the GGUF file of a model. If you can find the safetensors versions of the model, the `chat_template` can be found in the `tokenizer_config.json` file, eg [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/tokenizer_config.json#L34). Otherwise you can dump the KV table in the GGUF file using [llama.cpp](https://github.com/ggerganov/llama.cpp) and extract the template: ``` gguf-py/gguf/scripts/gguf_dump.py --json --no-tensors /path/to/model.gguf | jq -r '.metadata."tokenizer.chat_template".value' ``` This template is then converted to a Go template used in the TEMPLATE in the Modelfile. The output you show above is the result of the prompt from the client being processed through the Modelfile template. See [`index.json`](https://github.com/ollama/ollama/blob/main/template/index.json) for examples of Jinja templates converted to Go templates - the `name` in the file points at the relevant Go template in the [`template`](https://github.com/ollama/ollama/tree/main/template) directory. See [Go templates](https://pkg.go.dev/text/template) for documentation on writing go templates. As you mention, the easiest way to add a template is identify a model in the ollama library that has similar architecture and fine tuning to the one you want to import, and use that, perhaps with some modification.
GiteaMirror commented 2026-04-12 17:10:29 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 10, 2025):

The source of all templating for models is the Jinja template stored in the chat_template in the GGUF file of a model. If you can find the safetensors versions of the model, the chat_template can be found in the tokenizer_config.json file, eg DeepSeek-R1. Otherwise you can dump the KV table in the GGUF file using llama.cpp and extract the template:

gguf-py/gguf/scripts/gguf_dump.py --json --no-tensors /path/to/model.gguf | jq -r  '.metadata."tokenizer.chat_template".value'

This template is then converted to a Go template used in the TEMPLATE in the Modelfile. The output you show above is the result of the prompt from the client being processed through the Modelfile template.

See index.json for examples of Jinja templates converted to Go templates - the name in the file points at the relevant Go template in the template directory. See Go templates for documentation on writing go templates.

As you mention, the easiest way to add a template is identify a model in the ollama library that has similar architecture and fine tuning to the one you want to import, and use that, perhaps with some modification.

Thank you for such a detailed response. I think I have a better understanding of the underlying process behind the template/parameter generation done by ollama. I will try to make it work.

In the meantime, do you have any idea what could be going wrong with the create command when importing from GGUF?

<!-- gh-comment-id:2648855092 --> @mozophe commented on GitHub (Feb 10, 2025): > The source of all templating for models is the Jinja template stored in the `chat_template` in the GGUF file of a model. If you can find the safetensors versions of the model, the `chat_template` can be found in the `tokenizer_config.json` file, eg [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/tokenizer_config.json#L34). Otherwise you can dump the KV table in the GGUF file using [llama.cpp](https://github.com/ggerganov/llama.cpp) and extract the template: > > ``` > gguf-py/gguf/scripts/gguf_dump.py --json --no-tensors /path/to/model.gguf | jq -r '.metadata."tokenizer.chat_template".value' > ``` > > This template is then converted to a Go template used in the TEMPLATE in the Modelfile. The output you show above is the result of the prompt from the client being processed through the Modelfile template. > > See [`index.json`](https://github.com/ollama/ollama/blob/main/template/index.json) for examples of Jinja templates converted to Go templates - the `name` in the file points at the relevant Go template in the [`template`](https://github.com/ollama/ollama/tree/main/template) directory. See [Go templates](https://pkg.go.dev/text/template) for documentation on writing go templates. > > As you mention, the easiest way to add a template is identify a model in the ollama library that has similar architecture and fine tuning to the one you want to import, and use that, perhaps with some modification. Thank you for such a detailed response. I think I have a better understanding of the underlying process behind the template/parameter generation done by ollama. I will try to make it work. In the meantime, do you have any idea what could be going wrong with the create command when importing from GGUF?
GiteaMirror commented 2026-04-12 17:10:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

There's nothing wrong with the create command. If you don't supply a template then ollama will use a default. In some cases, ollama will recognize the embedded chat_template and substitute the appropriate Go template from the template directory, but if the chat_template is unrecognized or non-existent (can happen with Lora fusions), then ollama will use a default of {{ .Prompt }}. For stop parameters, these are sometimes found in the KV block of the GGUF and ollama will use those if no overrides are supplied via PARAMETER lines in the Modelfile. Anything else (system message, other parameters, license, message history) needs to be supplied in a Modelfile.

<!-- gh-comment-id:2648875909 --> @rick-github commented on GitHub (Feb 10, 2025): There's nothing wrong with the create command. If you don't supply a template then ollama will use a default. In some cases, ollama will recognize the embedded `chat_template` and substitute the appropriate Go template from the `template` directory, but if the `chat_template` is unrecognized or non-existent (can happen with Lora fusions), then ollama will use a default of `{{ .Prompt }}`. For `stop` parameters, these are sometimes found in the KV block of the GGUF and ollama will use those if no overrides are supplied via PARAMETER lines in the Modelfile. Anything else (system message, other parameters, license, message history) needs to be supplied in a Modelfile.
GiteaMirror commented 2026-04-12 17:10:30 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 10, 2025):

Here is the template extracted (decode("utf-8")) from the GGUF file (GGUF released by unsloth DeepSeek-R1-Qwen2.5-14B-Q4_K_M.gguf):

"{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}"

It is indeed not available in the list of templates for ollama. Is there a script available, which converts this to the format used by ollama?

<!-- gh-comment-id:2649267060 --> @mozophe commented on GitHub (Feb 10, 2025): Here is the template extracted (decode("utf-8")) from the GGUF file (GGUF released by unsloth DeepSeek-R1-Qwen2.5-14B-Q4_K_M.gguf): ```powershell "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}" ``` It is indeed not available in the list of templates for ollama. Is there a script available, which converts this to the format used by ollama?
GiteaMirror commented 2026-04-12 17:10:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

Probably, but I haven't used one. Some of the constructs used in the Jinja templates (namespaces, external arguments) don't easily translate into the Go templating language used in ollama, so I find it easier to just examine the logic of the chat_template and implement the equivalent in Go template logic. The model templates are a smaller problem domain than that addressed by the full Jinja language so it's generally straightforward: process the system message, loop over the remaining messages and output the appropriate special tokens, end with an assistant prompt. It gets more complicated if you need to address tool calls and tool responses, and the reasoning models now have their own particular takes, so in those cases it can be simpler to derive from an existing template.

<!-- gh-comment-id:2649301173 --> @rick-github commented on GitHub (Feb 10, 2025): Probably, but I haven't used one. Some of the constructs used in the Jinja templates (namespaces, external arguments) don't easily translate into the Go templating language used in ollama, so I find it easier to just examine the logic of the `chat_template` and implement the equivalent in Go template logic. The model templates are a smaller problem domain than that addressed by the full Jinja language so it's generally straightforward: process the system message, loop over the remaining messages and output the appropriate special tokens, end with an assistant prompt. It gets more complicated if you need to address tool calls and tool responses, and the reasoning models now have their own particular takes, so in those cases it can be simpler to derive from an existing template.
GiteaMirror commented 2026-04-12 17:10:31 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 10, 2025):

Makes sense. I just noticed that the chat template used by this quant in the ollama library has a different chat template than the one from the gguf. There is no reference to tools in the ollama hosted chat template (mentioned below). It also means that this chat template limits what the model could do, meaning it can't call tools. Is my understanding correct?

{{- if .System }}{{ .System }}{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1}}
{{- if eq .Role "user" }}<｜User｜>{{ .Content }}
{{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }}
{{- end }}
{{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }}
{{- end }}
<!-- gh-comment-id:2649318958 --> @mozophe commented on GitHub (Feb 10, 2025): Makes sense. I just noticed that the chat template used by this quant in the ollama library has a different chat template than the one from the gguf. There is no reference to tools in the ollama hosted chat template (mentioned below). It also means that this chat template limits what the model could do, meaning it can't call tools. Is my understanding correct? ```powershell {{- if .System }}{{ .System }}{{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1}} {{- if eq .Role "user" }}<｜User｜>{{ .Content }} {{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }} {{- end }} {{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }} {{- end }} ```
GiteaMirror commented 2026-04-12 17:10:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

Correct. Other users have made available modified deepseek templates that support tools, see https://ollama.com/search?c=tools&q=deepseek .

<!-- gh-comment-id:2649328231 --> @rick-github commented on GitHub (Feb 10, 2025): Correct. Other users have made available modified deepseek templates that support tools, see https://ollama.com/search?c=tools&q=deepseek .
GiteaMirror commented 2026-04-12 17:10:31 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 11, 2025):

Thank you again for the clear explanations.

This limitation makes it quite challenging for an average ollama user to use any non-ollama library model (and some ollama library models) to its full potential. I am not even sure if most of them are even aware of it. For example, the most popular model on ollma library at the moment is https://ollama.com/library/deepseek-r1, which is limited by its template (not allowing tool use), while the original GGUF does provide a template that allows for tool use.

Also, I found an article on construction of ollama templates: https://github.com/ollama/ollama/blob/main/docs/template.md

I think all the info @rick-github mentioned above, should be added to this article, especially the information on how to find the right template for a model. I couldn't find this information anywhere else (it's very likely that the knowledge is available somewhere on the internet and would be found by someone who looks deep enough). In the meantime, the key points mentioned above could at least bridge the knowledge gap.

Further, I am no longer sure if this should be classified as a bug/feature request? or if it's just a skill issue. This, however, makes me think about how this is managed in other local LLM implementations.

<!-- gh-comment-id:2649591376 --> @mozophe commented on GitHub (Feb 11, 2025): Thank you again for the clear explanations. This limitation makes it quite challenging for an average ollama user to use any non-ollama library model (and some ollama library models) to its full potential. I am not even sure if most of them are even aware of it. For example, the most popular model on ollma library at the moment is https://ollama.com/library/deepseek-r1, which is limited by its template (not allowing tool use), while the original GGUF does provide a template that allows for tool use. Also, I found an article on construction of ollama templates: https://github.com/ollama/ollama/blob/main/docs/template.md I think all the info @rick-github mentioned above, should be added to this article, especially the information on how to find the right template for a model. I couldn't find this information anywhere else (it's very likely that the knowledge is available somewhere on the internet and would be found by someone who looks deep enough). In the meantime, the key points mentioned above could at least bridge the knowledge gap. Further, I am no longer sure if this should be classified as a bug/feature request? or if it's just a skill issue. This, however, makes me think about how this is managed in other local LLM implementations.
GiteaMirror commented 2026-04-12 17:10:32 -05:00
Author
Owner
Copy Link

@mozophe commented on GitHub (Feb 15, 2025):

Here is a quick summary of how different local LLM implementations support chat templates:

Please feel free to correct me if I have misinterpreted something.

<!-- gh-comment-id:2660612785 --> @mozophe commented on GitHub (Feb 15, 2025): Here is a quick summary of how different local LLM implementations support chat templates: - LM Studio accepts prompt template in Jinja format: https://lmstudio.ai/docs/advanced/prompt-template - vLLM accepts jinja format: https://github.com/vllm-project/vllm/pull/1493, https://docs.vllm.ai/en/v0.5.3/serving/openai_compatible_server.html#chat-template - Koboldccp uses a similar method as Ollama (heuristically infer the correct instruct template to be used for the chat completions endpoint, based on the detected Jinja template from the model.): https://github.com/LostRuins/koboldcpp/wiki#what-is---chatcompletionsadapter - llama.cpp uses a similar method as Ollama (does not include a jinja parser in llama.cpp due to its complexity. The implementation works by matching the supplied template with a list of pre-defined templates hard coded inside the function; doesn't support custom template): https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template Please feel free to correct me if I have misinterpreted something.
GiteaMirror commented 2026-04-12 17:10:32 -05:00
Author
Owner
Copy Link

@sen9to1-design commented on GitHub (Jan 16, 2026):

Thank you again for the clear explanations.

This limitation makes it quite challenging for an average ollama user to use any non-ollama library model (and some ollama library models) to its full potential. I am not even sure if most of them are even aware of it. For example, the most popular model on ollma library at the moment is https://ollama.com/library/deepseek-r1, which is limited by its template (not allowing tool use), while the original GGUF does provide a template that allows for tool use.

Also, I found an article on construction of ollama templates: https://github.com/ollama/ollama/blob/main/docs/template.md

I think all the info @rick-github mentioned above, should be added to this article, especially the information on how to find the right template for a model. I couldn't find this information anywhere else (it's very likely that the knowledge is available somewhere on the internet and would be found by someone who looks deep enough). In the meantime, the key points mentioned above could at least bridge the knowledge gap.

Further, I am no longer sure if this should be classified as a bug/feature request? or if it's just a skill issue. This, however, makes me think about how this is managed in other local LLM implementations.

Appreciating for these information. Does the lack of tool using in default modelfile influence that i use tool feature in langchain structure with ollama

<!-- gh-comment-id:3757909229 --> @sen9to1-design commented on GitHub (Jan 16, 2026): # > > Thank you again for the clear explanations. > > This limitation makes it quite challenging for an average ollama user to use any non-ollama library model (and some ollama library models) to its full potential. I am not even sure if most of them are even aware of it. For example, the most popular model on ollma library at the moment is https://ollama.com/library/deepseek-r1, which is limited by its template (not allowing tool use), while the original GGUF does provide a template that allows for tool use. > > Also, I found an article on construction of ollama templates: https://github.com/ollama/ollama/blob/main/docs/template.md > > I think all the info [@rick-github](https://github.com/rick-github) mentioned above, should be added to this article, especially the information on how to find the right template for a model. I couldn't find this information anywhere else (it's very likely that the knowledge is available somewhere on the internet and would be found by someone who looks deep enough). In the meantime, the key points mentioned above could at least bridge the knowledge gap. > > Further, I am no longer sure if this should be classified as a bug/feature request? or if it's just a skill issue. This, however, makes me think about how this is managed in other local LLM implementations. Appreciating for these information. Does the lack of tool using in default modelfile influence that i use tool feature in langchain structure with ollama
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5830
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?