github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2240] How to limit output token generated: Phi model #27048

New Issue
Closed
opened 2026-04-22 03:57:09 -05:00 by GiteaMirror · 10 comments
GiteaMirror commented 2026-04-22 03:57:09 -05:00
Owner
Copy Link

Originally created by @bm777 on GitHub (Jan 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2240

Originally assigned to: @bmizerany on GitHub.

From a given context + query, the model generates well the answer, but very long -> around 2000 chars.
Is there any way to do max_output_tokens=200 like pplx or openAI API?

This is my prompt template:

_template = "You are an assistant that delivers short answers to the user inquiry from the provided context.\n\n
            context: {conditioned_passages}\n\n
            query: {query}
            answer:"
Originally created by @bm777 on GitHub (Jan 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2240 Originally assigned to: @bmizerany on GitHub. From a given context + query, the model generates well the answer, but very long -> around `2000 chars`. Is there any way to do `max_output_tokens=200` like pplx or openAI API? This is my prompt template: ```js _template = "You are an assistant that delivers short answers to the user inquiry from the provided context.\n\n context: {conditioned_passages}\n\n query: {query} answer:" ```
GiteaMirror added the feature request label 2026-04-22 03:57:09 -05:00
GiteaMirror commented 2026-04-22 03:57:10 -05:00
Author
Owner
Copy Link

@jukofyork commented on GitHub (Jan 28, 2024):

num_predict:

https://github.com/ollama/ollama/blob/main/docs/modelfile.md

I think the default is actually -1 even though the API docs say it's 128 (I had the Llemma model run all night once by accident!).

<!-- gh-comment-id:1913708832 --> @jukofyork commented on GitHub (Jan 28, 2024): `num_predict`: https://github.com/ollama/ollama/blob/main/docs/modelfile.md I think the default is actually `-1` even though the API docs say it's 128 (I had the Llemma model run all night once by accident!).
GiteaMirror commented 2026-04-22 03:57:10 -05:00
Author
Owner
Copy Link

@bm777 commented on GitHub (Jan 28, 2024):

Thanks :), i'm goint to put num_predict: 40

I noticed a kinda infinite token generation and stopped generating at some point. Maybe to prevent the early stop, could be to use stop System:

From

ollama show phi --parameters

Output

stop                           "User:"
stop                           "Assistant:"
stop                           "System:"
<!-- gh-comment-id:1913743125 --> @bm777 commented on GitHub (Jan 28, 2024): Thanks :), i'm goint to put `num_predict: 40` I noticed a kinda infinite token generation and stopped generating at some point. Maybe to prevent the early stop, could be to use stop `System:` From ```bash ollama show phi --parameters ``` Output ```bash stop "User:" stop "Assistant:" stop "System:" ```
GiteaMirror commented 2026-04-22 03:57:11 -05:00
Author
Owner
Copy Link

@bmizerany commented on GitHub (Mar 11, 2024):

@bm777 Did num_predict fix your issue? If so, do you mind closing (or me closing) the issue?

<!-- gh-comment-id:1989172977 --> @bmizerany commented on GitHub (Mar 11, 2024): @bm777 Did `num_predict` fix your issue? If so, do you mind closing (or me closing) the issue?
GiteaMirror commented 2026-04-22 03:57:11 -05:00
Author
Owner
Copy Link

@bm777 commented on GitHub (Mar 11, 2024):

@bmizerany somehow yes.

<!-- gh-comment-id:1989277857 --> @bm777 commented on GitHub (Mar 11, 2024): @bmizerany somehow yes.
GiteaMirror commented 2026-04-22 03:57:13 -05:00
Author
Owner
Copy Link

@metamec commented on GitHub (May 28, 2024):

The problem with the num_predict 'solution' is that you end up with a truncated final sentence.

<!-- gh-comment-id:2136061684 --> @metamec commented on GitHub (May 28, 2024): The problem with the num_predict 'solution' is that you end up with a truncated final sentence.
GiteaMirror commented 2026-04-22 03:57:13 -05:00
Author
Owner
Copy Link

@bm777 commented on GitHub (May 29, 2024):

in somehow, it doesn't cut properly. @bmizerany

<!-- gh-comment-id:2136497301 --> @bm777 commented on GitHub (May 29, 2024): in somehow, it doesn't cut properly. @bmizerany
GiteaMirror commented 2026-04-22 03:57:14 -05:00
Author
Owner
Copy Link

@bm777 commented on GitHub (Jun 7, 2024):

@metamec did you find a solution?

<!-- gh-comment-id:2154634712 --> @bm777 commented on GitHub (Jun 7, 2024): @metamec did you find a solution?
GiteaMirror commented 2026-04-22 03:57:14 -05:00
Author
Owner
Copy Link

@Anish-Reddy-K commented on GitHub (Jul 17, 2024):

The problem with the num_predict 'solution' is that you end up with a truncated final sentence.

did you find a solution to this?

<!-- gh-comment-id:2232390613 --> @Anish-Reddy-K commented on GitHub (Jul 17, 2024): > The problem with the num_predict 'solution' is that you end up with a truncated final sentence. did you find a solution to this?
GiteaMirror commented 2026-04-22 03:57:17 -05:00
Author
Owner
Copy Link

@metamec commented on GitHub (Jul 22, 2024):

Unfortunately not, sorry.

<!-- gh-comment-id:2242765580 --> @metamec commented on GitHub (Jul 22, 2024): Unfortunately not, sorry.
GiteaMirror commented 2026-04-22 03:57:19 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 20, 2024):

I'm going to close this since it's pretty stale a this point. I think the problem was the template wasn't correct, although it does work reasonably well with phi3, although phi3 seems to have problems when it hits the context size.

<!-- gh-comment-id:2557885915 --> @pdevine commented on GitHub (Dec 20, 2024): I'm going to close this since it's pretty stale a this point. I think the problem was the template wasn't correct, although it does work reasonably well with phi3, although phi3 seems to have problems when it hits the context size.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#27048
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?