github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2677] Questions about querying Ollama via the OpenAI API #27353

New Issue
Closed
opened 2026-04-22 04:38:47 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-22 04:38:47 -05:00
Owner
Copy Link

Originally created by @dictoon on GitHub (Feb 22, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2677

(This is a follow-up to #2595.)

  1. I'm invoking Ollama through OpenAI's API in Python. Is there documentation on passing additional options such as context size?

    I've tried this, but it doesn't work:

    options = dict(num_ctx=4096)
response = self.client.chat.completions.create(
    model=Plugin.LLM_MODEL,
    messages=conversation,
    extra_body={"options": options}
)

    However, using Ollama's native Python API, this seems to work:

    response = ollama.chat(
    model=Plugin.OLLAMA_MODEL,
    messages=conversation,
    options={ "num_ctx": 4096 }
)

  2. I thought the context window was defined by the model and couldn't be changed. Do I understand correctly that, in the case of querying Ollama via the OpenAI API, somehow the context window is shrunk? For performance perhaps?

Originally created by @dictoon on GitHub (Feb 22, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2677 (This is a follow-up to #2595.) 1. I'm invoking Ollama through OpenAI's API in Python. Is there documentation on passing additional options such as context size? I've tried this, but it doesn't work: ``` options = dict(num_ctx=4096) response = self.client.chat.completions.create( model=Plugin.LLM_MODEL, messages=conversation, extra_body={"options": options} ) ``` However, using Ollama's native Python API, this seems to work: ``` response = ollama.chat( model=Plugin.OLLAMA_MODEL, messages=conversation, options={ "num_ctx": 4096 } ) ``` 2. I thought the context window was defined by the model and couldn't be changed. Do I understand correctly that, in the case of querying Ollama via the OpenAI API, somehow the context window is shrunk? For performance perhaps?
GiteaMirror added the question label 2026-04-22 04:38:47 -05:00
GiteaMirror commented 2026-04-22 04:38:52 -05:00
Author
Owner
Copy Link

@Nurgo commented on GitHub (Mar 20, 2024):

I'm wondering the same thing, have you found an answer?

<!-- gh-comment-id:2009313710 --> @Nurgo commented on GitHub (Mar 20, 2024): I'm wondering the same thing, have you found an answer?
GiteaMirror commented 2026-04-22 04:38:52 -05:00
Author
Owner
Copy Link

@dictoon commented on GitHub (Mar 20, 2024):

I'm wondering the same thing, have you found an answer?

I haven't.

<!-- gh-comment-id:2009372619 --> @dictoon commented on GitHub (Mar 20, 2024): > I'm wondering the same thing, have you found an answer? I haven't.
GiteaMirror commented 2026-04-22 04:38:53 -05:00
Author
Owner
Copy Link

@chigkim commented on GitHub (Jun 26, 2024):

Did you find an answer?
I thought I might get luck with max_tokens parameter, but it doesn't seem to do anything according to log.
It doesn't set --num-predict for llama.cpp either.

<!-- gh-comment-id:2191581080 --> @chigkim commented on GitHub (Jun 26, 2024): Did you find an answer? I thought I might get luck with max_tokens parameter, but it doesn't seem to do anything according to log. It doesn't set --num-predict for llama.cpp either.
GiteaMirror commented 2026-04-22 04:38:54 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Jul 18, 2024):

  1. The Ollama Open AI API doc does mention the fields which are supported, but you can also use Open AIs own docs. For the context size, use the max_tokens field.
  2. It can be changed, but some models don't necessarily work well if you change it. It depends on the model.
<!-- gh-comment-id:2237744556 --> @pdevine commented on GitHub (Jul 18, 2024): 1. The [Ollama Open AI API doc](https://github.com/ollama/ollama/blob/main/docs/openai.md) does mention the fields which are supported, but you can also use Open AIs own [docs](https://platform.openai.com/docs/api-reference/chat). For the context size, use the `max_tokens` field. 2. It can be changed, but some models don't necessarily work well if you change it. It depends on the model.
GiteaMirror commented 2026-04-22 04:38:55 -05:00
Author
Owner
Copy Link

@chigkim commented on GitHub (Jul 19, 2024):

max_tokens: "The maximum number of tokens that can be generated in the chat completion.
The total length of input tokens and generated tokens is limited by the model's context length."

Isn't max_tokens on OpenAI API specifies how many tokens to generate (num_predict on Ollama, --predict in llama.cpp), not the context size (num_ctx on Ollama, --ctx-size in llama.cpp)?

<!-- gh-comment-id:2238253269 --> @chigkim commented on GitHub (Jul 19, 2024): max_tokens: "The maximum number of tokens that can be generated in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length." Isn't max_tokens on OpenAI API specifies how many tokens to generate (num_predict on Ollama, --predict in llama.cpp), not the context size (num_ctx on Ollama, --ctx-size in llama.cpp)?
GiteaMirror commented 2026-04-22 04:38:56 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Aug 9, 2024):

I don't think this is completed / should be closed as there's currently no way I can see to make Ollama set num_ctx via the OpenAI compatible API.

@pdevine if you provide max_tokens in a request, it doesn't set the num_ctx:

curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
     "messages": [{"role": "user", "content": "tell me a joke about llamas"}],
     "model": "llama3.1:8b-instruct-q6_K",
     "max_tokens": 512
   }'

results in Ollama using the context size from the Modelfile of the model (32K in this example) rather than the request (512):

DEBUG [update_slots] slot released | n_cache_tokens=40 n_ctx=32768 n_past=39 n_system_tokens=0 slot_id=0 task_id=216 tid="140073938796544" timestamp=1723176672 truncated=false
<!-- gh-comment-id:2277121530 --> @sammcj commented on GitHub (Aug 9, 2024): I don't think this is completed / should be closed as there's currently no way I can see to make Ollama set num_ctx via the OpenAI compatible API. @pdevine if you provide max_tokens in a request, it doesn't set the num_ctx: ``` curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{"role": "user", "content": "tell me a joke about llamas"}], "model": "llama3.1:8b-instruct-q6_K", "max_tokens": 512 }' ``` results in Ollama using the context size from the Modelfile of the model (32K in this example) rather than the request (512): ``` DEBUG [update_slots] slot released | n_cache_tokens=40 n_ctx=32768 n_past=39 n_system_tokens=0 slot_id=0 task_id=216 tid="140073938796544" timestamp=1723176672 truncated=false ```
GiteaMirror commented 2026-04-22 04:38:57 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Aug 28, 2024):

@sammcj max_tokens as OpenAI defines it sets the maximum number of tokens which the model will respond with. The equivalent in Ollama is the num_predict setting, not num_ctx. If we change it to also set num_ctx this would break if you wanted to feed in a large prompt which only outputted a limited amount of tokens.

It is possible to set the context size though, although not directly through the OpenAI API. You can create a Modelfile which looks like:

FROM llama3.1:8b-instruct-q6_K
PARAMETER num_ctx 512

Then use ollama create mymodel to create a new model, and then use mymodel as the model inside of the API request.

<!-- gh-comment-id:2316445809 --> @pdevine commented on GitHub (Aug 28, 2024): @sammcj `max_tokens` [as OpenAI defines it](https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens) sets the maximum number of tokens which the model will respond with. The equivalent in Ollama is the `num_predict` setting, not `num_ctx`. If we change it to also set `num_ctx` this would break if you wanted to feed in a large prompt which only outputted a limited amount of tokens. It is possible to set the context size though, although not directly through the OpenAI API. You can create a `Modelfile` which looks like: ``` FROM llama3.1:8b-instruct-q6_K PARAMETER num_ctx 512 ``` Then use `ollama create mymodel` to create a new model, and then use `mymodel` as the model inside of the API request.
GiteaMirror commented 2026-04-22 04:38:57 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Aug 29, 2024):

I've updated our docs to explain how to do this as well.

<!-- gh-comment-id:2316466555 --> @pdevine commented on GitHub (Aug 29, 2024): I've updated our [docs](https://github.com/ollama/ollama/blob/main/docs/openai.md#setting-the-context-size) to explain how to do this as well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#27353
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?