github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #4386] Support tools in OpenAI-compatible API #64777

New Issue
Closed
opened 2026-05-03 18:46:08 -05:00 by GiteaMirror · 22 comments
GiteaMirror commented 2026-05-03 18:46:08 -05:00
Owner
Copy Link

Originally created by @jackmpcollins on GitHub (May 12, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4386

Support the tools and tool_choice parameters in the OpenAI-compatible API. Currently these are not supported 4ec7445a6f/docs/openai.md (supported-request-fields)

I believe llama.cpp is used internally by Ollama, and this has support for tools and tool_choice https://github.com/abetlen/llama-cpp-python#function-calling so the implementation in Ollama can hopefully leverage this.

Having the tools parameter implemented in Ollama (or llama.cpp and surfacing this) would standardize all downstream packages (e.g. https://github.com/BerriAI/litellm , https://github.com/jackmpcollins/magentic) on a single prompt and implementation, which will make it more robust.

I would use this to improve support for local LLMs in https://github.com/jackmpcollins/magentic

Related issues

Originally created by @jackmpcollins on GitHub (May 12, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4386 Support the `tools` and `tool_choice` parameters in the OpenAI-compatible API. Currently these are not supported https://github.com/ollama/ollama/blob/4ec7445a6f678b6efc773bb9fa886d7c9b075577/docs/openai.md#supported-request-fields I believe `llama.cpp` is used internally by Ollama, and this has support for tools and tool_choice https://github.com/abetlen/llama-cpp-python#function-calling so the implementation in Ollama can hopefully leverage this. Having the tools parameter implemented in Ollama (or llama.cpp and surfacing this) would standardize all downstream packages (e.g. https://github.com/BerriAI/litellm , https://github.com/jackmpcollins/magentic) on a single prompt and implementation, which will make it more robust. I would use this to improve support for local LLMs in https://github.com/jackmpcollins/magentic Related issues - https://github.com/ollama/ollama/issues/305 - https://github.com/ollama/ollama/issues/1729 - https://github.com/ollama/ollama/issues/3165
GiteaMirror added the feature request label 2026-05-03 18:46:08 -05:00
GiteaMirror commented 2026-05-03 18:46:11 -05:00
Author
Owner
Copy Link

@zyxcambridge commented on GitHub (May 13, 2024):

OpenAI compatibility for curl https://api.openai.com/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer $OPENAI_API_KEY"
-d '{
"model": "gpt-4-turbo",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What’s in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
}
}
]
}
],
"max_tokens": 300
}'

<!-- gh-comment-id:2106601440 --> @zyxcambridge commented on GitHub (May 13, 2024): OpenAI compatibility for curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4-turbo", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What’s in this image?" }, { "type": "image_url", "image_url": { "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" } } ] } ], "max_tokens": 300 }'
GiteaMirror commented 2026-05-03 18:46:14 -05:00
Author
Owner
Copy Link

@zhangweiwei0326 commented on GitHub (May 13, 2024):

+1

<!-- gh-comment-id:2106933079 --> @zhangweiwei0326 commented on GitHub (May 13, 2024): +1
GiteaMirror commented 2026-05-03 18:46:16 -05:00
Author
Owner
Copy Link

@nikhil-swamix commented on GitHub (May 13, 2024):

+1

<!-- gh-comment-id:2108437451 --> @nikhil-swamix commented on GitHub (May 13, 2024): +1
GiteaMirror commented 2026-05-03 18:46:16 -05:00
Author
Owner
Copy Link

@kyjus25 commented on GitHub (May 14, 2024):

+1 Looks like I'm too early to the party yet again 😅

<!-- gh-comment-id:2111125493 --> @kyjus25 commented on GitHub (May 14, 2024): +1 Looks like I'm too early to the party yet again 😅
GiteaMirror commented 2026-05-03 18:46:17 -05:00
Author
Owner
Copy Link

@muliyul commented on GitHub (May 18, 2024):

I'd like to contribute to this, however I'm not familiar with the repo. If someone can lay out what needs to be done I'd be happy to help.

<!-- gh-comment-id:2118880602 --> @muliyul commented on GitHub (May 18, 2024): I'd like to contribute to this, however I'm not familiar with the repo. If someone can lay out what needs to be done I'd be happy to help.
GiteaMirror commented 2026-05-03 18:46:18 -05:00
Author
Owner
Copy Link

@gaurishmehra commented on GitHub (Jun 11, 2024):

+1

<!-- gh-comment-id:2160835295 --> @gaurishmehra commented on GitHub (Jun 11, 2024): +1
GiteaMirror commented 2026-05-03 18:46:19 -05:00
Author
Owner
Copy Link

@kkolago commented on GitHub (Jun 11, 2024):

+1

<!-- gh-comment-id:2160877581 --> @kkolago commented on GitHub (Jun 11, 2024): +1
GiteaMirror commented 2026-05-03 18:46:20 -05:00
Author
Owner
Copy Link

@slavonnet commented on GitHub (Jun 16, 2024):

+1

<!-- gh-comment-id:2171770439 --> @slavonnet commented on GitHub (Jun 16, 2024): +1
GiteaMirror commented 2026-05-03 18:46:20 -05:00
Author
Owner
Copy Link

@dariush-bahrami commented on GitHub (Jun 21, 2024):

+1

<!-- gh-comment-id:2182892591 --> @dariush-bahrami commented on GitHub (Jun 21, 2024): +1
GiteaMirror commented 2026-05-03 18:46:21 -05:00
Author
Owner
Copy Link

@nikhil-swamix commented on GitHub (Jun 22, 2024):

image

<!-- gh-comment-id:2183864197 --> @nikhil-swamix commented on GitHub (Jun 22, 2024): ![image](https://github.com/ollama/ollama/assets/54004431/f7387d17-32ce-40fe-a101-538db9b43288)
GiteaMirror commented 2026-05-03 18:46:22 -05:00
Author
Owner
Copy Link

@humcqc commented on GitHub (Jul 12, 2024):

Hello @jmorganca , any timeline for this one ?

<!-- gh-comment-id:2225882006 --> @humcqc commented on GitHub (Jul 12, 2024): Hello @jmorganca , any timeline for this one ?
GiteaMirror commented 2026-05-03 18:46:23 -05:00
Author
Owner
Copy Link

@langchain4j commented on GitHub (Jul 17, 2024):

@jmorganca even a rough estimate would be highly appreciated, thank you!

<!-- gh-comment-id:2233551898 --> @langchain4j commented on GitHub (Jul 17, 2024): @jmorganca even a rough estimate would be highly appreciated, thank you!
GiteaMirror commented 2026-05-03 18:46:23 -05:00
Author
Owner
Copy Link

@kyjus25 commented on GitHub (Jul 17, 2024):

@langchain4j uhh... I think maybe it was just merged, actually? I can't really tell but the PR name and description lines up

https://github.com/ollama/ollama/pull/5614

<!-- gh-comment-id:2233578010 --> @kyjus25 commented on GitHub (Jul 17, 2024): @langchain4j uhh... I think maybe it was just merged, actually? I can't really tell but the PR name and description lines up https://github.com/ollama/ollama/pull/5614
GiteaMirror commented 2026-05-03 18:46:24 -05:00
Author
Owner
Copy Link

@meletis commented on GitHub (Jul 19, 2024):

@jmorganca, is this going to be part of the next release?

<!-- gh-comment-id:2238720029 --> @meletis commented on GitHub (Jul 19, 2024): @jmorganca, is this going to be part of the next release?
GiteaMirror commented 2026-05-03 18:46:25 -05:00
Author
Owner
Copy Link

@humcqc commented on GitHub (Jul 24, 2024):

@langchain4j uhh... I think maybe it was just merged, actually? I can't really tell but the PR name and description lines up

#5614

I think this is just a part of what needs to be done to have tools working in ollama ?

<!-- gh-comment-id:2247372250 --> @humcqc commented on GitHub (Jul 24, 2024): > @langchain4j uhh... I think maybe it was just merged, actually? I can't really tell but the PR name and description lines up > > #5614 I think this is just a part of what needs to be done to have tools working in ollama ?
GiteaMirror commented 2026-05-03 18:46:25 -05:00
Author
Owner
Copy Link

@kyjus25 commented on GitHub (Jul 24, 2024):

@humcqc yes, agreed. I've seen lots of commits go thru now about tools

Here's an X post I saw about tools with a demo if anyone is curious; it was mentioned in their keynote.

https://x.com/AlexReibman/status/1814142347367817443

We are close!!!

<!-- gh-comment-id:2247976004 --> @kyjus25 commented on GitHub (Jul 24, 2024): @humcqc yes, agreed. I've seen lots of commits go thru now about tools Here's an X post I saw about tools with a demo if anyone is curious; it was mentioned in their keynote. https://x.com/AlexReibman/status/1814142347367817443 We are close!!!
GiteaMirror commented 2026-05-03 18:46:26 -05:00
Author
Owner
Copy Link

@kyjus25 commented on GitHub (Jul 24, 2024):

Might I also add - llama3.1's template already has tools baked in (as I'm sure others do as well) 👀
https://ollama.com/library/llama3.1/blobs/11ce4ee3e170

<!-- gh-comment-id:2248223000 --> @kyjus25 commented on GitHub (Jul 24, 2024): Might I also add - llama3.1's template already has tools baked in (as I'm sure others do as well) 👀 https://ollama.com/library/llama3.1/blobs/11ce4ee3e170 <img src="https://github.com/user-attachments/assets/7689a2f3-d544-4617-9c55-88e213f1156d" width="300" />
GiteaMirror commented 2026-05-03 18:46:27 -05:00
Author
Owner
Copy Link

@kyjus25 commented on GitHub (Jul 24, 2024):

I'll believe it when the dang "Function calling" checkbox is checked 😂 I check every day and it's getting unhealthy
https://github.com/ollama/ollama/blob/main/docs/openai.md#endpoints

<!-- gh-comment-id:2248232550 --> @kyjus25 commented on GitHub (Jul 24, 2024): I'll believe it when the dang "Function calling" checkbox is checked 😂 I check every day and it's getting unhealthy https://github.com/ollama/ollama/blob/main/docs/openai.md#endpoints
GiteaMirror commented 2026-05-03 18:46:27 -05:00
Author
Owner
Copy Link

@humcqc commented on GitHub (Jul 24, 2024):

I'll believe it when the dang "Function calling" checkbox is checked 😂 I check every day and it's getting unhealthy https://github.com/ollama/ollama/blob/main/docs/openai.md#endpoints

Yes seems we are close!!

<!-- gh-comment-id:2248237108 --> @humcqc commented on GitHub (Jul 24, 2024): > I'll believe it when the dang "Function calling" checkbox is checked 😂 I check every day and it's getting unhealthy https://github.com/ollama/ollama/blob/main/docs/openai.md#endpoints Yes seems we are close!!
GiteaMirror commented 2026-05-03 18:46:27 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jul 26, 2024):

Hi there, this is now supported as of 0.3.0 https://ollama.com/blog/tool-support

Note: OpenAI streaming tool calling isn't yet implemented, but this is something being worked on

<!-- gh-comment-id:2251738923 --> @jmorganca commented on GitHub (Jul 26, 2024): Hi there, this is now supported as of 0.3.0 https://ollama.com/blog/tool-support Note: OpenAI _streaming_ tool calling isn't yet implemented, but this is something being worked on
GiteaMirror commented 2026-05-03 18:46:28 -05:00
Author
Owner
Copy Link

@pamelafox commented on GitHub (Aug 9, 2024):

Awesome! I tested this for a RAG app and found that I had to use few shot examples to get good results, at least from llama3.1.
Blog with more details: https://blog.pamelafox.org/2024/08/making-ollama-compatible-rag-app.html

<!-- gh-comment-id:2278826383 --> @pamelafox commented on GitHub (Aug 9, 2024): Awesome! I tested this for a RAG app and found that I had to use few shot examples to get good results, at least from llama3.1. Blog with more details: https://blog.pamelafox.org/2024/08/making-ollama-compatible-rag-app.html
GiteaMirror commented 2026-05-03 18:46:28 -05:00
Author
Owner
Copy Link

@mdrxy commented on GitHub (Jun 28, 2025):

For anyone who stumbles upon this, streaming tool calling is now supported. tool_choice is not.

<!-- gh-comment-id:3014791820 --> @mdrxy commented on GitHub (Jun 28, 2025): For anyone who stumbles upon this, streaming tool calling [is now supported](https://ollama.com/blog/streaming-tool). `tool_choice` is not.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#64777
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?