github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2673] Stop tokens appear in the model output. #1589

New Issue
Closed
opened 2026-04-12 11:31:15 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-12 11:31:15 -05:00
Owner
Copy Link

Originally created by @olafgeibig on GitHub (Feb 22, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2673

I created my own Ollama model of https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO-GGUF
Here is my modelfile:

FROM ./nous-hermes-2-mistral-7b-dpo.Q5_K_M.gguf
PARAMETER num_ctx 8192
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

When running the model with crewAI with a coding agent crew then sometimes stop tokens appear in the output. That doesn't happen with the same model hosted at together.ai. What am doing wrong? I think my modelfile is correct, since it is mostly a copy of the official openhermes modelfile.

Example output:

Use Tool: Pygame for game development and graphics renderingHere is a valid schema for Pygame tool:

{
    "tool_name": "Pygame",
    "arguments": {
        "window_size": (int, int), # tuple with width and height of window
        "frame_rate": float, # frame rate of the game loop
        "colors": dict, # dictionary of colors used in the game or graphics
        "fonts": dict, # dictionary of fonts used in the game or graphics
        "sprites": list, # list of sprite objects used in the game
        "sound_effects": dict, # dictionary of sound effects used in the game
        "music": str, # path to music file for background music
        "additional_features": list, # list of additional features used in the game
    }
}
```<|im_end|>{
    "tool_name": "Pygame",
    "arguments": {
        "window_size": (800, 600),
        "fps": 60,
        "colors": ["red", "blue"],
        "sounds": ["sound1.wav", "sound2.mp3"]
    }
}
Originally created by @olafgeibig on GitHub (Feb 22, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2673 I created my own Ollama model of https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO-GGUF Here is my modelfile: ``` FROM ./nous-hermes-2-mistral-7b-dpo.Q5_K_M.gguf PARAMETER num_ctx 8192 TEMPLATE """<|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" ``` When running the model with crewAI with a coding agent crew then sometimes stop tokens appear in the output. That doesn't happen with the same model hosted at together.ai. What am doing wrong? I think my modelfile is correct, since it is mostly a copy of the official openhermes modelfile. Example output: ``` Use Tool: Pygame for game development and graphics renderingHere is a valid schema for Pygame tool: { "tool_name": "Pygame", "arguments": { "window_size": (int, int), # tuple with width and height of window "frame_rate": float, # frame rate of the game loop "colors": dict, # dictionary of colors used in the game or graphics "fonts": dict, # dictionary of fonts used in the game or graphics "sprites": list, # list of sprite objects used in the game "sound_effects": dict, # dictionary of sound effects used in the game "music": str, # path to music file for background music "additional_features": list, # list of additional features used in the game } } ```<|im_end|>{ "tool_name": "Pygame", "arguments": { "window_size": (800, 600), "fps": 60, "colors": ["red", "blue"], "sounds": ["sound1.wav", "sound2.mp3"] } } ```
GiteaMirror added the bug label 2026-04-12 11:31:15 -05:00
GiteaMirror commented 2026-04-12 11:31:17 -05:00
Author
Owner
Copy Link

@Ath0ms38 commented on GitHub (Feb 23, 2024):

I have the same issue. Every output ends with : <|im_end|>

<!-- gh-comment-id:1961292380 --> @Ath0ms38 commented on GitHub (Feb 23, 2024): I have the same issue. Every output ends with : <|im_end|>
GiteaMirror commented 2026-04-12 11:31:17 -05:00
Author
Owner
Copy Link

@maxtheman commented on GitHub (Mar 13, 2024):

Confirmed this is also happening for the new Hermes Pro model with many different variations of this template.

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""

(Tried adding {{ .Response }}<|im_end|>, just response, just end, ect.)

<!-- gh-comment-id:1995961751 --> @maxtheman commented on GitHub (Mar 13, 2024): Confirmed this is also happening for the new Hermes Pro model with many different variations of this template. TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant """ (Tried adding {{ .Response }}<|im_end|>, just response, just end, ect.)
GiteaMirror commented 2026-04-12 11:31:17 -05:00
Author
Owner
Copy Link

@maxtheman commented on GitHub (Mar 13, 2024):

Update from @mchiang0610 — all of our files need these for ChatML

PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

@olafgeibig try yours without the quotation marks?

<!-- gh-comment-id:1996079741 --> @maxtheman commented on GitHub (Mar 13, 2024): Update from @mchiang0610 — all of our files need these for ChatML PARAMETER stop <|im_start|> PARAMETER stop <|im_end|> @olafgeibig try yours without the quotation marks?
GiteaMirror commented 2026-04-12 11:31:18 -05:00
Author
Owner
Copy Link

@olafgeibig commented on GitHub (Mar 15, 2024):

@maxtheman this actually helped, now my modelfile looks like this and the stop token from the end is gone.

FROM ./Hermes-2-Pro-Mistral-7B.Q5_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
PARAMETER num_ctx 8192
<!-- gh-comment-id:2000099099 --> @olafgeibig commented on GitHub (Mar 15, 2024): @maxtheman this actually helped, now my modelfile looks like this and the stop token from the end is gone. ``` FROM ./Hermes-2-Pro-Mistral-7B.Q5_K_M.gguf TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant """ PARAMETER stop <|im_start|> PARAMETER stop <|im_end|> PARAMETER num_ctx 8192 ```
GiteaMirror commented 2026-04-12 11:31:20 -05:00
Author
Owner
Copy Link

@honeykjoule commented on GitHub (Mar 19, 2024):

PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

adding this to the Modelfile worked for me too

<!-- gh-comment-id:2005705310 --> @honeykjoule commented on GitHub (Mar 19, 2024): > PARAMETER stop <|im_start|> > PARAMETER stop <|im_end|> adding this to the `Modelfile` worked for me too
GiteaMirror commented 2026-04-12 11:31:20 -05:00
Author
Owner
Copy Link

@telehan commented on GitHub (Apr 17, 2024):

issue still there, here is my test

$ ollama -v
ollama version is 0.1.32
$ ollama show --modelfile mistral:7b-hermes2pro-q8
FROM ~/.ollama/models/blobs/sha256-b6d95d7ec9a395b7568cc94b0447fd4f90b6f69d6e44794b1fbb84e3f732baca
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER num_ctx 8192
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

test case in macOS

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOllama(model="mistral:7b-hermes2pro-q8")
prompt = ChatPromptTemplate.from_template("Tell me a short joke about {topic}")
chain = prompt | llm | StrOutputParser()
print(chain.invoke({"topic": "Space travel"}))

Why don't we ever tell secrets on Mars? Because Curiosity Rover! 🚀😉<|im_end|>

<!-- gh-comment-id:2061043888 --> @telehan commented on GitHub (Apr 17, 2024): issue still there, here is my test ```bash $ ollama -v ollama version is 0.1.32 $ ollama show --modelfile mistral:7b-hermes2pro-q8 FROM ~/.ollama/models/blobs/sha256-b6d95d7ec9a395b7568cc94b0447fd4f90b6f69d6e44794b1fbb84e3f732baca TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant """ PARAMETER num_ctx 8192 PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" ``` test case in macOS ```python from langchain_community.chat_models import ChatOllama from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate llm = ChatOllama(model="mistral:7b-hermes2pro-q8") prompt = ChatPromptTemplate.from_template("Tell me a short joke about {topic}") chain = prompt | llm | StrOutputParser() print(chain.invoke({"topic": "Space travel"})) ``` Why don't we ever tell secrets on Mars? Because Curiosity Rover! 🚀😉<|im_end|>
GiteaMirror commented 2026-04-12 11:31:21 -05:00
Author
Owner
Copy Link

@Ath0ms38 commented on GitHub (Apr 17, 2024):

@telehan

Try removing the "" in the model file for the the stop token

so change:
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
to :
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>

<!-- gh-comment-id:2061293876 --> @Ath0ms38 commented on GitHub (Apr 17, 2024): @telehan Try removing the "" in the model file for the the stop token so change: PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" to : PARAMETER stop <|im_start|> PARAMETER stop <|im_end|>
GiteaMirror commented 2026-04-12 11:31:21 -05:00
Author
Owner
Copy Link

@telehan commented on GitHub (May 6, 2024):

the issue should be close, re-test with ollama 0.1.33, no suffix '<|im_end|>' appending to response

$ ollama -v
ollama version is 0.1.33

$ pip3.11 show langchain
Name: langchain
Version: 0.1.17
Summary: Building applications with LLMs through composability
Home-page: https://github.com/langchain-ai/langchain
Author:
Author-email:
License: MIT
Location: /usr/local/lib/python3.11/site-packages
Requires: aiohttp, dataclasses-json, jsonpatch, langchain-community, langchain-core, langchain-text-splitters, langsmith, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity
Required-by: langchain-experimental, rag-ollama-multi-query, ragas

$ ollama show --modelfile mistral:7b-hermes2pro-q8
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this one, replace the FROM line with:
# FROM mistral:7b-hermes2pro-q8

FROM ~/.ollama/models/blobs/sha256-b6d95d7ec9a395b7568cc94b0447fd4f90b6f69d6e44794b1fbb84e3f732baca
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
SYSTEM """You are 'Hermes 2', a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia."""
PARAMETER num_ctx 8192
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

and thanks @Ath0ms38 , the quotation marks were added by ollama show, modelfile configuration is clean, the marks seem no affect to this issue.

<!-- gh-comment-id:2095075100 --> @telehan commented on GitHub (May 6, 2024): the issue should be close, re-test with ollama 0.1.33, no suffix '<|im_end|>' appending to response ```bash $ ollama -v ollama version is 0.1.33 $ pip3.11 show langchain Name: langchain Version: 0.1.17 Summary: Building applications with LLMs through composability Home-page: https://github.com/langchain-ai/langchain Author: Author-email: License: MIT Location: /usr/local/lib/python3.11/site-packages Requires: aiohttp, dataclasses-json, jsonpatch, langchain-community, langchain-core, langchain-text-splitters, langsmith, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity Required-by: langchain-experimental, rag-ollama-multi-query, ragas $ ollama show --modelfile mistral:7b-hermes2pro-q8 # Modelfile generated by "ollama show" # To build a new Modelfile based on this one, replace the FROM line with: # FROM mistral:7b-hermes2pro-q8 FROM ~/.ollama/models/blobs/sha256-b6d95d7ec9a395b7568cc94b0447fd4f90b6f69d6e44794b1fbb84e3f732baca TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant {{ .Response }}<|im_end|>""" SYSTEM """You are 'Hermes 2', a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.""" PARAMETER num_ctx 8192 PARAMETER stop "<|im_end|>" PARAMETER stop "<|im_start|>" ``` and thanks @Ath0ms38 , the quotation marks were added by `ollama show`, modelfile configuration is clean, the marks seem no affect to this issue.
GiteaMirror commented 2026-04-12 11:31:21 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 17, 2024):

This was an issue w/ the parser incorrectly adding in the quotes and was fixed a few weeks ago. I'm going to go ahead and close it as being fixed.

<!-- gh-comment-id:2118463752 --> @pdevine commented on GitHub (May 17, 2024): This was an issue w/ the parser incorrectly adding in the quotes and was fixed a few weeks ago. I'm going to go ahead and close it as being fixed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1589
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?