github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5544] OpenAI v1/completion inserts prompt template #29224

New Issue
Closed
opened 2026-04-22 07:56:12 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-22 07:56:12 -05:00
Owner
Copy Link

Originally created by @chigkim on GitHub (Jul 8, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5544

What is the issue?

If you use the new completion feature (not chat.completion), it should be completely free form including prompt format.
Ollama inserts prompt template even if I use OpenAI completion (client.completions.create).
For example, if I use phi3, debug output shows:
time=2024-07-06T12:23:40.094-04:00 level=DEBUG source=routes.go:208 msg="generate handler" prompt="<|user|>\n{my prompt}<|end|>\n<|assistant|>\n"

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.1.49

Originally created by @chigkim on GitHub (Jul 8, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5544 ### What is the issue? If you use the new completion feature (not chat.completion), it should be completely free form including prompt format. Ollama inserts prompt template even if I use OpenAI completion (client.completions.create). For example, if I use phi3, debug output shows: time=2024-07-06T12:23:40.094-04:00 level=DEBUG source=routes.go:208 msg="generate handler" prompt="<|user|>\n{my prompt}<|end|>\n<|assistant|>\n" ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.1.49
GiteaMirror added the bug label 2026-04-22 07:56:12 -05:00
GiteaMirror commented 2026-04-22 07:56:15 -05:00
Author
Owner
Copy Link

@royjhan commented on GitHub (Jul 8, 2024):

This behavior is expected - the prompt construction is necessary to return a response for each specific model.

<!-- gh-comment-id:2215430502 --> @royjhan commented on GitHub (Jul 8, 2024): This behavior is expected - the prompt construction is necessary to return a response for each specific model.
GiteaMirror commented 2026-04-22 07:56:16 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jul 8, 2024):

To add, Ollama also has text completion versions of the models you can use with the /v1/completions API. For example llama3:text will run the Llama 3 foundation model which does not have a template. Hope this helps!

<!-- gh-comment-id:2215442501 --> @jmorganca commented on GitHub (Jul 8, 2024): To add, Ollama also has text completion versions of the models you can use with the `/v1/completions` API. For example `llama3:text` will run the Llama 3 foundation model which does not have a template. Hope this helps!
GiteaMirror commented 2026-04-22 07:56:17 -05:00
Author
Owner
Copy Link

@chigkim commented on GitHub (Jul 9, 2024):

@jmorganca, thanks for pointing out the Ollama api. However, having the OpenAI compatible API that behaves the same way helps people to run things that's not specific coded for Ollama with Ollama.

@royjhan if you use llama.cpp v1/completion, it does not insert prompt template. This is the log from llama.cpp after running the same script above with the same model.

INFO [            update_slots] all slots are idle | tid="11640" timestamp=1720511130
VERB [          kv_cache_clear] clearing KV cache | tid="11640" timestamp=1720511130
VERB [              start_loop] wait for new task | tid="11640" timestamp=1720511130
VERB [              get_new_id] new task id | tid="3352" timestamp=1720511145 new_id=0
VERB [     add_waiting_task_id] waiting for task id | tid="3352" timestamp=1720511145 id_task=0
VERB [              start_loop] new task may arrive | tid="11640" timestamp=1720511145
VERB [              start_loop] callback_new_task | tid="11640" timestamp=1720511145 id_task=0
VERB [      get_available_slot] selected slot by lru | tid="11640" timestamp=1720511145 id_slot=0 t_last=-1
INFO [   launch_slot_with_task] slot is processing task | tid="11640" timestamp=1720511145 id_slot=0 id_task=0
VERB [              start_loop] update_multitasks | tid="11640" timestamp=1720511145
VERB [              start_loop] callback_update_slots | tid="11640" timestamp=1720511145
VERB [            update_slots] posting NEXT_RESPONSE | tid="11640" timestamp=1720511145
VERB [                    post] new task id | tid="11640" timestamp=1720511145 new_id=1
VERB [            update_slots] tokenizing prompt | tid="11640" timestamp=1720511145 id_slot=0 id_task=0
VERB [            update_slots] prompt tokenized | tid="11640" timestamp=1720511145 id_slot=0 id_task=0 n_ctx=4096 n_keep=0 n_prompt_tokens=30 prompt_tokens="<s> User: Hello,\nAssistant: Hi, how can I help you?\nUser: How's it going?\nAssistant:"

As you can see, tokenized prompt only has BOS <s> inserted.

Hope you consider to make the OpenAI API consistent with others.

<!-- gh-comment-id:2216875472 --> @chigkim commented on GitHub (Jul 9, 2024): @jmorganca, thanks for pointing out the Ollama api. However, having the OpenAI compatible API that behaves the same way helps people to run things that's not specific coded for Ollama with Ollama. @royjhan if you use llama.cpp v1/completion, it does not insert prompt template. This is the log from llama.cpp after running the same script above with the same model. ```bash INFO [ update_slots] all slots are idle | tid="11640" timestamp=1720511130 VERB [ kv_cache_clear] clearing KV cache | tid="11640" timestamp=1720511130 VERB [ start_loop] wait for new task | tid="11640" timestamp=1720511130 VERB [ get_new_id] new task id | tid="3352" timestamp=1720511145 new_id=0 VERB [ add_waiting_task_id] waiting for task id | tid="3352" timestamp=1720511145 id_task=0 VERB [ start_loop] new task may arrive | tid="11640" timestamp=1720511145 VERB [ start_loop] callback_new_task | tid="11640" timestamp=1720511145 id_task=0 VERB [ get_available_slot] selected slot by lru | tid="11640" timestamp=1720511145 id_slot=0 t_last=-1 INFO [ launch_slot_with_task] slot is processing task | tid="11640" timestamp=1720511145 id_slot=0 id_task=0 VERB [ start_loop] update_multitasks | tid="11640" timestamp=1720511145 VERB [ start_loop] callback_update_slots | tid="11640" timestamp=1720511145 VERB [ update_slots] posting NEXT_RESPONSE | tid="11640" timestamp=1720511145 VERB [ post] new task id | tid="11640" timestamp=1720511145 new_id=1 VERB [ update_slots] tokenizing prompt | tid="11640" timestamp=1720511145 id_slot=0 id_task=0 VERB [ update_slots] prompt tokenized | tid="11640" timestamp=1720511145 id_slot=0 id_task=0 n_ctx=4096 n_keep=0 n_prompt_tokens=30 prompt_tokens="<s> User: Hello,\nAssistant: Hi, how can I help you?\nUser: How's it going?\nAssistant:" ``` As you can see, tokenized prompt only has BOS `<s>` inserted. Hope you consider to make the OpenAI API consistent with others.
GiteaMirror commented 2026-04-22 07:56:18 -05:00
Author
Owner
Copy Link

@chigkim commented on GitHub (Jul 9, 2024):

Sorry, probably the log didn't come through correctly via email. I reformatted the output correctly. Please refer to the correctly formatted log on the web.

<!-- gh-comment-id:2216888552 --> @chigkim commented on GitHub (Jul 9, 2024): Sorry, probably the log didn't come through correctly via email. I reformatted the output correctly. Please refer to the correctly formatted log on the web.
GiteaMirror commented 2026-04-22 07:56:18 -05:00
Author
Owner
Copy Link

@mistrjirka commented on GitHub (Sep 25, 2024):

To add, Ollama also has text completion versions of the models you can use with the /v1/completions API. For example llama3:text will run the Llama 3 foundation model which does not have a template. Hope this helps!

I do not understand, which models are supported for endpoint /v1/completions, because from what I see you seem to indicate that if the model is the text version it then work with the endpoint. But I tried "llama3.1:8b-text-q6_K", and I get llama3.1:8b-text-q6_K does not support insert

<!-- gh-comment-id:2373854910 --> @mistrjirka commented on GitHub (Sep 25, 2024): > To add, Ollama also has text completion versions of the models you can use with the `/v1/completions` API. For example `llama3:text` will run the Llama 3 foundation model which does not have a template. Hope this helps! I do not understand, which models are supported for endpoint /v1/completions, because from what I see you seem to indicate that if the model is the text version it then work with the endpoint. But I tried "llama3.1:8b-text-q6_K", and I get llama3.1:8b-text-q6_K does not support insert
GiteaMirror commented 2026-04-22 07:56:18 -05:00
Author
Owner
Copy Link

@SkutteOleg commented on GitHub (Apr 14, 2025):

To add, Ollama also has text completion versions of the models you can use with the /v1/completions API. For example llama3:text will run the Llama 3 foundation model which does not have a template. Hope this helps!

@jmorganca what should I do if there are no text completion version of a model I want to use? Why can I use any model I want for text completion with llama.cpp and all of its forks, but not with Ollama?

This is giving me a bit of headache because my software relies on text completions and requires /v1/completions endpoint in order to function properly. My software allows users to use custom OpenAI endpoints. It works well with llama.cpp server, LM Studio, koboldcpp, etc., but when my users use Ollama it doesn't work as expected and produces chat responses instead. In turn, my users think that there is something wrong with my software.

<!-- gh-comment-id:2801985154 --> @SkutteOleg commented on GitHub (Apr 14, 2025): > To add, Ollama also has text completion versions of the models you can use with the `/v1/completions` API. For example `llama3:text` will run the Llama 3 foundation model which does not have a template. Hope this helps! @jmorganca what should I do if there are no text completion version of a model I want to use? Why can I use any model I want for text completion with llama.cpp and all of its forks, but not with Ollama? This is giving me a bit of headache because my software relies on text completions and requires `/v1/completions` endpoint in order to function properly. My software allows users to use custom OpenAI endpoints. It works well with llama.cpp server, LM Studio, koboldcpp, etc., but when my users use Ollama it doesn't work as expected and produces chat responses instead. In turn, my users think that there is something wrong with my software.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#29224
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?