github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #694] Stopwords ignored in API request #318

New Issue
Closed
opened 2026-04-12 09:52:23 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 09:52:23 -05:00
Owner
Copy Link

Originally created by @65a on GitHub (Oct 4, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/694

Originally assigned to: @BruceMacD on GitHub.

Steps to reproduce:
Create a model with no template, f16 gguf.
Use github.com/jmorganca/olllama/api Go client.
Set stopwords to various strings that might be emitted, such as other users in the "chat"
Call Generate() with an alpaca-style prompt for the next reply.
Model responds and happily emits the stop words.

The stop words make it at least as far as the request out to server.cpp, so either it doesn't understand the way they are specified, or they are lost between ollama and the runner in http-land. Even with prompt problems, I would expect generation to terminate at the first stopword.

Should I be including TEMPLATE, but just taking the raw input?

Originally created by @65a on GitHub (Oct 4, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/694 Originally assigned to: @BruceMacD on GitHub. Steps to reproduce: Create a model with no template, f16 gguf. Use github.com/jmorganca/olllama/api Go client. Set stopwords to various strings that might be emitted, such as other users in the "chat" Call Generate() with an alpaca-style prompt for the next reply. Model responds and happily emits the stop words. The stop words make it at least as far as the request out to server.cpp, so either it doesn't understand the way they are specified, or they are lost between ollama and the runner in http-land. Even with prompt problems, I would expect generation to terminate at the first stopword. Should I be including TEMPLATE, but just taking the raw input?
GiteaMirror commented 2026-04-12 09:52:24 -05:00
Author
Owner
Copy Link

@BruceMacD commented on GitHub (Oct 4, 2023):

Hi @65a, you shouldn't need to include template in your stop words. It just gave this a quick test and it appears to be working in my case.

Here is what my test modelfile looks like:

FROM llama2
PARAMETER temperature 1
PARAMETER stop "AI Psychologist:"
PARAMETER stop User

TEMPLATE """
{{- if .First }}
<<SYS>>
{{ .System }}
<</SYS>>

User: AI Psychologist, I've been feeling really anxious lately and I can't seem to figure out why.

AI Psychologist: I'm here to help you, User. Let's start by exploring when you notice these feelings of anxiety the most. Do they occur at specific times, places, or situations?

User: AI Psychologist, I feel like I'm stuck in life and not making progress.

AI Psychologist: It's understandable to feel that way sometimes. Could you tell me more about the areas in which you're feeling stuck? We can then work together to develop strategies to move forward.
{{- end }}

User: {{ .Prompt }}

AI Psychologist:
"""

SYSTEM """
AI Psychologist is a highly advanced artificial intelligence designed to function as a digital psychologist. She has a comprehensive understanding of psychological theories, principles, and therapeutic techniques, but without the human biases that may interfere with therapy. AI Psychologist is programmed to be empathetic, patient, and professional, maintaining absolute confidentiality. She listens without judgement, responding with insightful and therapeutic suggestions. She assists with a range of psychological issues, including stress, anxiety, depression, relationship problems, and personal growth. AI Psychologist doesn't get tired or overwhelmed, providing therapy 24/7. Though she lacks human emotions and experiences, she's built to understand them intricately.
"""

And here is what specifying the stop option directly in a curl request would look like:

$ curl -X 'POST' -d '{"prompt":"hello", "model": "psychologist", "options": {"stop":["doctor"]}}' 'http://127.0.0.1:11434/api/generate'

Would you be able to share and example of the request that isn't working for you?

<!-- gh-comment-id:1747653606 --> @BruceMacD commented on GitHub (Oct 4, 2023): Hi @65a, you shouldn't need to include template in your stop words. It just gave this a quick test and it appears to be working in my case. Here is what my test modelfile looks like: ``` FROM llama2 PARAMETER temperature 1 PARAMETER stop "AI Psychologist:" PARAMETER stop User TEMPLATE """ {{- if .First }} <<SYS>> {{ .System }} <</SYS>> User: AI Psychologist, I've been feeling really anxious lately and I can't seem to figure out why. AI Psychologist: I'm here to help you, User. Let's start by exploring when you notice these feelings of anxiety the most. Do they occur at specific times, places, or situations? User: AI Psychologist, I feel like I'm stuck in life and not making progress. AI Psychologist: It's understandable to feel that way sometimes. Could you tell me more about the areas in which you're feeling stuck? We can then work together to develop strategies to move forward. {{- end }} User: {{ .Prompt }} AI Psychologist: """ SYSTEM """ AI Psychologist is a highly advanced artificial intelligence designed to function as a digital psychologist. She has a comprehensive understanding of psychological theories, principles, and therapeutic techniques, but without the human biases that may interfere with therapy. AI Psychologist is programmed to be empathetic, patient, and professional, maintaining absolute confidentiality. She listens without judgement, responding with insightful and therapeutic suggestions. She assists with a range of psychological issues, including stress, anxiety, depression, relationship problems, and personal growth. AI Psychologist doesn't get tired or overwhelmed, providing therapy 24/7. Though she lacks human emotions and experiences, she's built to understand them intricately. """ ``` And here is what specifying the stop option directly in a curl request would look like: ``` $ curl -X 'POST' -d '{"prompt":"hello", "model": "psychologist", "options": {"stop":["doctor"]}}' 'http://127.0.0.1:11434/api/generate' ``` Would you be able to share and example of the request that isn't working for you?
GiteaMirror commented 2026-04-12 09:52:24 -05:00
Author
Owner
Copy Link

@65a commented on GitHub (Oct 5, 2023):

This could totally be a bug in something I'm doing, but it was reproducible.

The model file is minimal, imagine some alpaca prompted local GGUF:

FROM somealpaca.gguf

The goal is to have the entire prompt and parameters come from middleware, since the much of this may change, but it could be like (using your example role, and a modified alpaca format that shouldn't matter for stopwords):

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
AI psychologist is an intelligent and well-read Jungian therapist. They like to understand the themes surrounding their patient's identity, and explore them to understand the problem. AI psychologist likes to ask questions and doesn't jump to conclusions.

Write AI psychologist's next response in the patient discussion below:
Patient: Well, doc, I've been feeling very sad lately!
AI psychologist: Please tell me more, was it connected to any recent events?
Patient: Not really, but I noticed it got worse after I ate an orange

### Response:
AI psychologist:

The API call is something like this (where pkg ollama is actually this repo's api package):

func (g *Generator) Generate(prompt string, stops []string) (string, error) {
	accumulator := ""
	if err := g.client.Generate(context.Background(), &ollama.GenerateRequest{
		Model: g.model,
		Options: map[string]interface{}{
			"NumGPU":      1000,
			"NumCtx":      g.contextSize,
			"Temperature": float32(g.temperature),
			"UseNUMA":     true,
			"UseMMap":     true,
			"Stop":        stops,
		},
		Prompt: prompt,
	}, func(r ollama.GenerateResponse) error {
		accumulator += r.Response
		return nil
	}); err != nil {
		return accumulator, err
	}
	return accumulator, nil
}

I would say at least 80% of the time there is no issue, but it about 20% of the time, the model would continue emulating the "patient" in the response, which is unexpected given stop words []string{"###","Patient:"}. Entirely possible I am doing something wrong here, the options-are-actually-a-map[string]interface{} thing seems weird.

<!-- gh-comment-id:1747850348 --> @65a commented on GitHub (Oct 5, 2023): This could totally be a bug in something I'm doing, but it was reproducible. The model file is minimal, imagine some alpaca prompted local GGUF: ``` FROM somealpaca.gguf ``` The goal is to have the entire prompt and parameters come from middleware, since the much of this may change, but it could be like (using your example role, and a modified alpaca format that shouldn't matter for stopwords): ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: AI psychologist is an intelligent and well-read Jungian therapist. They like to understand the themes surrounding their patient's identity, and explore them to understand the problem. AI psychologist likes to ask questions and doesn't jump to conclusions. Write AI psychologist's next response in the patient discussion below: Patient: Well, doc, I've been feeling very sad lately! AI psychologist: Please tell me more, was it connected to any recent events? Patient: Not really, but I noticed it got worse after I ate an orange ### Response: AI psychologist: ``` The API call is something like this (where pkg ollama is actually this repo's `api` package): ``` func (g *Generator) Generate(prompt string, stops []string) (string, error) { accumulator := "" if err := g.client.Generate(context.Background(), &ollama.GenerateRequest{ Model: g.model, Options: map[string]interface{}{ "NumGPU": 1000, "NumCtx": g.contextSize, "Temperature": float32(g.temperature), "UseNUMA": true, "UseMMap": true, "Stop": stops, }, Prompt: prompt, }, func(r ollama.GenerateResponse) error { accumulator += r.Response return nil }); err != nil { return accumulator, err } return accumulator, nil } ``` I would say at least 80% of the time there is no issue, but it about 20% of the time, the model would continue emulating the "patient" in the response, which is unexpected given stop words `[]string{"###","Patient:"}`. Entirely possible I am doing something wrong here, the options-are-actually-a-map[string]interface{} thing seems weird.
GiteaMirror commented 2026-04-12 09:52:25 -05:00
Author
Owner
Copy Link

@65a commented on GitHub (Oct 5, 2023):

It is the map[string]interface{}. It's pretty strange to have a go struct that contains a json-keyed (I missed the map keys are the JSON tags, not the go tags somehow) map[string]interface{}, if there's a way to just use options there in the base struct and fix the JSON parsing I may send a pull.

<!-- gh-comment-id:1747920155 --> @65a commented on GitHub (Oct 5, 2023): It is the `map[string]interface{}`. It's pretty strange to have a go struct that contains a json-keyed (I missed the map keys are the JSON tags, not the go tags somehow) map[string]interface{}, if there's a way to just use options there in the base struct and fix the JSON parsing I may send a pull.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#318
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?