github-starred/ollama-ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-04-28 00:40:20 -05:00
Code Issues 2.2k Packages Projects Releases 100 Wiki Activity

[PR #4946] [CLOSED] Support for tools requests in ollama API #10363

New Issue
Closed
opened 2025-11-12 15:26:40 -06:00 by GiteaMirror · 0 comments
GiteaMirror commented 2025-11-12 15:26:40 -06:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/4946
Author: @infinity0n3
Created: 6/9/2024
Status: ❌ Closed

Base: main ← Head: tool_calls_support

📝 Commits (2)

  • a1b2663 Support for tools requests
  • 2be8223 Added support for tool_calls response parsing

📊 Changes

4 files changed (+210 additions, -71 deletions)

View changed files

📝 api/types.go (+19 -25)
📝 server/prompt.go (+27 -9)
📝 server/prompt_test.go (+5 -2)
📝 server/routes.go (+159 -35)

📄 Description

Ollama API tools/tool_calls support

The tool support would consist of three addons to the ollama core.

  1. Adding "tools" to the chat request and "tool_calls" to the chat response
  2. Extending the model template to support .Tools and .Results variables.
  3. To support detection of the model responding with a tool call request.
  4. Further work towards OpenAI function calling support

The examples will be based on the Mistral-7b-v0.3 model it now supports tool calls.

(1) Ollama API extension

The target handler would be the chat endpoint.

First the request would be extended with a "tool" argument. This argument would be of string type as probably not all models will have the OpenAI json formatted tool description.

Next, the a new "tool" message role should be added to support the tool results to be added to the "messages".

Finally, the chat response would need to have "tool_calls" argument to return the detected tool calls coming from the model response. This part will be addressed in the (3) Tool calls detection section of this Pull Request description.

(2) Model template extension

The prompt must support a way to define the available tools which in the mistral case require to add [AVAILABLE_TOOLS]...[/AVAILABLE_TOOLS] section before the [INST]...[/INST] section.

For the tool call results to be accepted by the model in the next completion cycle, the results must be enclosed with [TOOL_RESULTS]...[/TOOL_RESULTS].

Update mistral prompt template would look like this:

{{ if .Tools }}[AVAILABLE_TOOLS] {{ .Tools }} [/AVAILABLE_TOOLS] {{ end }}[INST] {{ if .System }}{{ .System }} {{ end }}{{ .Prompt }} [/INST]{{ if .Results }}[TOOL_RESULTS] {{ .Results }}[/TOOL_RESULTS]{{ end }} {{ .Response }}

For other models, .Tools and .Results variables would allow to specify different tokens enclosures or sequences to define available tools and to convert messages with "tool" role into the model supported prompt format.

Hermes 2 Pro Mistral 7B model

For this model the available tools prompt section could look as follows:

{{ if .Tools }}<tools>{{.Tools}}</tools>{{ end}}

(3) Tool calls detection

The model template extension is a straight forward one; however the tool call request parsing is the same.

In the case of the mistral model, the tool calls request starts with [TOOL_CALLS] token and is followed by a json array block. This would be the end of the story if the model would not occasionally hallucinate and add some text after the json array.

To mitigate this pitfall a "dumb" json array parser would be required. This parser would be combined with the start token [TOOL_CALLS] [...] to extract only that part of the response.

To keep ollama model independent, at the moment 2 types of tool call parsers could be added:

  1. [start_token] content [end_token] parser/extractor
  2. [start_token] json parser/extractor

Once the tool_calls have been detected and extracted, the content would be returned in the "tool_calls" response argument. As the format of the content will be highly dependent on the model safest would be to return it as a string.

The first one seems the most logical, however Mistral must have had their reasons to not include a end token.

The parser/extractor type as well as the start and stop tokens would be defined in the model parameters.

In case of Mistral, the parameters would look like this:

tool_calls_start     [TOOL_CALLS]
tool_calls_format json

In case the model returns both start and end token it would look like this:

tool_calls_start     [TOOL_CALLS]
tool_calls_end       [/TOOL_CALLS]
tool_calls_format json

(4) OpenAI function calling support

With the above addons to the ollama core and API, adding OpenAI function calling API support would mostly be straight forward.

  1. Add "tools" argument to the chat.completation endpoint and validate the correct format of the tools json value. This value would then be converted from json to a string and passed to Ollama.Chat api
  2. Parse the tool_calls and convert it the OpenAI tool calls json format.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/4946 **Author:** [@infinity0n3](https://github.com/infinity0n3) **Created:** 6/9/2024 **Status:** ❌ Closed **Base:** `main` ← **Head:** `tool_calls_support` --- ### 📝 Commits (2) - [`a1b2663`](https://github.com/ollama/ollama/commit/a1b2663c8c3c53942ce479cf626ee4b8635b7c9d) Support for tools requests - [`2be8223`](https://github.com/ollama/ollama/commit/2be8223a4529d53a27e438e62db5a12b8276acc0) Added support for tool_calls response parsing ### 📊 Changes **4 files changed** (+210 additions, -71 deletions) <details> <summary>View changed files</summary> 📝 `api/types.go` (+19 -25) 📝 `server/prompt.go` (+27 -9) 📝 `server/prompt_test.go` (+5 -2) 📝 `server/routes.go` (+159 -35) </details> ### 📄 Description Ollama API tools/tool_calls support The tool support would consist of three addons to the ollama core. 1. Adding "tools" to the chat request and "tool_calls" to the chat response 2. Extending the model template to support `.Tools` and `.Results` variables. 3. To support detection of the model responding with a tool call request. 4. Further work towards OpenAI function calling support The examples will be based on the Mistral-7b-v0.3 model it now supports tool calls. # (1) Ollama API extension The target handler would be the `chat` endpoint. First the request would be extended with a `"tool"` argument. This argument would be of `string` type as probably not all models will have the OpenAI `json` formatted tool description. Next, the a new `"tool"` message role should be added to support the tool results to be added to the `"messages"`. Finally, the chat response would need to have `"tool_calls"` argument to return the detected tool calls coming from the model response. This part will be addressed in the (3) Tool calls detection section of this Pull Request description. # (2) Model template extension The prompt must support a way to define the available tools which in the mistral case require to add `[AVAILABLE_TOOLS]...[/AVAILABLE_TOOLS]` section before the `[INST]...[/INST]` section. For the tool call results to be accepted by the model in the next completion cycle, the results must be enclosed with `[TOOL_RESULTS]...[/TOOL_RESULTS]`. Update mistral prompt template would look like this: ``` {{ if .Tools }}[AVAILABLE_TOOLS] {{ .Tools }} [/AVAILABLE_TOOLS] {{ end }}[INST] {{ if .System }}{{ .System }} {{ end }}{{ .Prompt }} [/INST]{{ if .Results }}[TOOL_RESULTS] {{ .Results }}[/TOOL_RESULTS]{{ end }} {{ .Response }} ``` For other models, `.Tools` and .`Results` variables would allow to specify different tokens enclosures or sequences to define available tools and to convert messages with "tool" role into the model supported prompt format. ## Hermes 2 Pro Mistral 7B model For this model the available tools prompt section could look as follows: ``` {{ if .Tools }}<tools>{{.Tools}}</tools>{{ end}} ``` # (3) Tool calls detection The model template extension is a straight forward one; however the tool call request parsing is the same. In the case of the mistral model, the tool calls request starts with `[TOOL_CALLS]` token and is followed by a `json array` block. This would be the end of the story if the model would not occasionally hallucinate and add some text after the json array. To mitigate this pitfall a "dumb" json array parser would be required. This parser would be combined with the start token `[TOOL_CALLS] [...]` to extract only that part of the response. To keep ollama model independent, at the moment 2 types of tool call parsers could be added: 1. [start_token] content [end_token] parser/extractor 2. [start_token] json parser/extractor Once the tool_calls have been detected and extracted, the content would be returned in the `"tool_calls"` response argument. As the format of the content will be highly dependent on the model safest would be to return it as a `string`. The first one seems the most logical, however Mistral must have had their reasons to not include a end token. The parser/extractor type as well as the start and stop tokens would be defined in the model parameters. In case of Mistral, the parameters would look like this: ``` tool_calls_start [TOOL_CALLS] tool_calls_format json ``` In case the model returns both start and end token it would look like this: ``` tool_calls_start [TOOL_CALLS] tool_calls_end [/TOOL_CALLS] tool_calls_format json ``` # (4) OpenAI function calling support With the above addons to the ollama core and API, adding OpenAI function calling API support would mostly be straight forward. 1. Add "tools" argument to the `chat.completation` endpoint and validate the correct format of the tools json value. This value would then be converted from json to a string and passed to Ollama.Chat api 2. Parse the tool_calls and convert it the OpenAI tool calls json format. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2025-11-12 15:26:40 -06:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
client2
cloud
compatibility
context-length
create
docker
documentation
embeddings
engine
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
install
integration
intel
js
linux
macos
memory
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama-ollama#10363
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?