github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #14967] Warn on tool name mismatch when tool_choice: "required" &/ support case-insensitive tool name matching #9624

New Issue
Open
opened 2026-04-12 22:31:34 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 22:31:34 -05:00
Owner
Copy Link

Originally created by @cicoyle on GitHub (Mar 19, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14967

When using tool_choice: "required" with the OpenAI-compatible /v1/chat/completions endpoint, if the system prompt instructs the model to call a tool name that doesn't exactly match any registered tool definition, the model silently returns this resp: finish_reason: "stop", instead of emitting tool calls. There is no warning or error.

This came up in this issue Tool calls silently drop with large system prompts (~1600+ tokens) where I spent time trying to figure out what was going on, and I thought it was an Ollama bug, but was actually a casing mismatch btw tool names in my system prompt (kebab-case: get-orders-at-risk-count) and the registered tool definitions (PascalCase: GetOrdersAtRiskCount). This is easy to hit when tool names are transformed between systems.

Either of the following ideas would be helpful:

  1. emit a warning when tool names don't match

When tool_choice: "required" is set and the model's resp references a function name that doesn't match any tool in the tools array, Ollama could:

  • Log a warning server-side (ex: WARN: model attempted to call 'get-orders-at-risk-count' but no matching tool found in registered tools)
  • Optionally include a warning in the API response

This would make debugging significantly faster.

  1. Support case-insensitive / normalized tool name matching

Apply normalization when matching tool names so that get-orders-at-risk-count, GetOrdersAtRiskCount, and getordersatriskcount all resolve to the same tool. This could be done by the following:

  • Strip hyphens/underscores and compare case-insensitively
  • Or use a simple fuzzy match on registered tool names

Currently,

Request: tool_choice: "required", tools: [GetOrdersAtRiskCount, ...], system prompt says "Call get-orders-at-risk-count"
Response: finish_reason: "stop", content: "I don't have access to tools...", tool_calls: none

Expected behavior:
Either

  • A warning is emitted that the model tried to call a tool that doesn't exist
  • Or the name is fuzzy-matched to the correct registered tool and the tool call succeeds

Issue for ref that prompted this issue to be created.

Originally created by @cicoyle on GitHub (Mar 19, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14967 When using `tool_choice: "required"` with the OpenAI-compatible `/v1/chat/completions` endpoint, if the system prompt instructs the model to call a tool name that doesn't exactly match any registered tool definition, the model silently returns this resp: `finish_reason: "stop"`, instead of emitting tool calls. There is no warning or error. This came up in this issue [Tool calls silently drop with large system prompts (~1600+ tokens)](https://github.com/ollama/ollama/issues/14958) where I spent time trying to figure out what was going on, and I thought it was an Ollama bug, but was actually a casing mismatch btw tool names in my system prompt (kebab-case: `get-orders-at-risk-count`) and the registered tool definitions (PascalCase: `GetOrdersAtRiskCount`). This is easy to hit when tool names are transformed between systems. Either of the following ideas would be helpful: 1. emit a warning when tool names don't match When `tool_choice: "required"` is set and the model's resp references a function name that doesn't match any tool in the tools array, Ollama could: - Log a warning server-side (ex: WARN: model attempted to call 'get-orders-at-risk-count' but no matching tool found in registered tools) - Optionally include a warning in the API response This would make debugging significantly faster. 2. Support case-insensitive / normalized tool name matching Apply normalization when matching tool names so that `get-orders-at-risk-count`, `GetOrdersAtRiskCount`, and `getordersatriskcount` all resolve to the same tool. This could be done by the following: - Strip hyphens/underscores and compare case-insensitively - Or use a simple fuzzy match on registered tool names Currently, ``` Request: tool_choice: "required", tools: [GetOrdersAtRiskCount, ...], system prompt says "Call get-orders-at-risk-count" Response: finish_reason: "stop", content: "I don't have access to tools...", tool_calls: none ``` Expected behavior: Either - A warning is emitted that the model tried to call a tool that doesn't exist - Or the name is fuzzy-matched to the correct registered tool and the tool call succeeds [Issue for ref that prompted this issue to be created.](https://github.com/ollama/ollama/issues/14958)
GiteaMirror commented 2026-04-12 22:31:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 19, 2026):

I could have sworn that there was code to emit a warning on a failed tool name match, but it doesn't appear to be there now. Either I imagined it or it got inadvertently removed in a refactor or something.

<!-- gh-comment-id:4093672607 --> @rick-github commented on GitHub (Mar 19, 2026): I could have sworn that there was code to emit a warning on a failed tool name match, but it doesn't appear to be there now. Either I imagined it or it got inadvertently removed in a refactor or something.
GiteaMirror commented 2026-04-12 22:31:36 -05:00
Author
Owner
Copy Link

@adrianabasbous commented on GitHub (Mar 30, 2026):

hello,
while looking for any information about future support for the tool_choice feature, I stumbled on this thread. How is it you're using tool_choice with the OpenAI compatible /v1/chat/completions endpoint if the documentation explicitly mentions that it isn't supported, nor does it appear to be in /api/chat https://docs.ollama.com/api/chat
thanks

<!-- gh-comment-id:4155231607 --> @adrianabasbous commented on GitHub (Mar 30, 2026): hello, while looking for any information about future support for the `tool_choice` feature, I stumbled on this thread. How is it you're using `tool_choice` with the OpenAI compatible `/v1/chat/completions` endpoint if the [documentation](https://docs.ollama.com/api/openai-compatibility) explicitly mentions that it isn't supported, nor does it appear to be in `/api/chat` https://docs.ollama.com/api/chat thanks
GiteaMirror commented 2026-04-12 22:31:36 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2026):

It's accepted because it's part of the OpenAI API specification but is currently ignored.

<!-- gh-comment-id:4155263599 --> @rick-github commented on GitHub (Mar 30, 2026): It's accepted because it's part of the OpenAI API specification but is currently ignored.
GiteaMirror commented 2026-04-12 22:31:37 -05:00
Author
Owner
Copy Link

@adrianabasbous commented on GitHub (Mar 30, 2026):

thanks - do you know if there's any plan to support it or if there's an equivalent to this parameter, which can force the model to call at least 1 tool?

<!-- gh-comment-id:4155282770 --> @adrianabasbous commented on GitHub (Mar 30, 2026): thanks - do you know if there's any plan to support it or if there's an equivalent to this parameter, which can force the model to call at least 1 tool?
GiteaMirror commented 2026-04-12 22:31:37 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2026):

There is no mechanism for forcing a model to use a tool. The only way to influence the model would be with prompt engineering, either through instruction in the user/system prompt, or by using pre-fill to add opening tool call tags to the context buffer. The latter may require additional support in the tool call parser in the ollama server, I haven't tried it to see if it works.

<!-- gh-comment-id:4155351984 --> @rick-github commented on GitHub (Mar 30, 2026): There is no mechanism for forcing a model to use a tool. The only way to influence the model would be with prompt engineering, either through instruction in the user/system prompt, or by using pre-fill to add opening tool call tags to the context buffer. The latter may require additional support in the tool call parser in the ollama server, I haven't tried it to see if it works.
GiteaMirror commented 2026-04-12 22:31:38 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2026):

Using structured output may also be a way to force tool use, send a schema with the request that conforms to a tool call.

<!-- gh-comment-id:4155393252 --> @rick-github commented on GitHub (Mar 30, 2026): Using structured output may also be a way to force tool use, send a schema with the request that conforms to a tool call.
GiteaMirror commented 2026-04-12 22:31:38 -05:00
Author
Owner
Copy Link

@adrianabasbous commented on GitHub (Mar 30, 2026):

thanks for confirming

<!-- gh-comment-id:4155745150 --> @adrianabasbous commented on GitHub (Mar 30, 2026): thanks for confirming
GiteaMirror commented 2026-04-12 22:31:39 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 30, 2026):

Using structured output may also be a way to force tool use, send a schema with the request that conforms to a tool call.

This worked surprisingly well. qwen2.5-coder is a tool capable model but is bad in actual use. Constraining the output to a tool call format makes it much more reliable.

#!/usr/bin/env python3

import ollama

def bash(command: str) -> str:
    """
    Execute a bash command on the system. Use this to run shell commands, check files, run programs, etc.

    Args:
      command: The bash command to execute
    """
    pass


format={
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "string",
  "pattern": r'^<tool_call>\n\{"name": "[^"]*", "arguments": .*?\}\n</tool_call>$'
}

def test(format=None):
  response = ollama.chat(
      model="qwen2.5-coder",
      messages=[
        {"role":"user", "content":"what is the time?"}
      ],
      tools=[bash],
      format=format
  )
  print(f"content  : {response.message.content.replace('\n','\\n')}\ntool_call: {response.message.tool_calls}")

print("==", "No format", "====")
for _ in range(5):
  test()

print("\n==", "Format", "====")
for _ in range(5):
  test(format=format)

== No format ====
content  : ```json\n{\n  "name": "bash",\n  "arguments": {\n    "command": "date"\n  }\n}\n```
tool_call: None
content  : ```xml\n{"name": "bash", "arguments": {"command":"date"}}\n```
tool_call: None
content  : ```json\n{"name": "bash", "arguments": {"command": "date"}}\n```
tool_call: None
content  : ```json\n{"name":"bash", "arguments":{"command":"date"}}\n```
tool_call: None
content  : ```xml\n{"name": "bash", "arguments": {"command":"date"}}\n```
tool_call: None

== Format ====
content  : "
tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))]
content  : "
tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))]
content  : "
tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))]
content  : "
tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))]
content  : "
tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))]
<!-- gh-comment-id:4156748669 --> @rick-github commented on GitHub (Mar 30, 2026): > Using structured output may also be a way to force tool use, send a schema with the request that conforms to a tool call. This worked surprisingly well. qwen2.5-coder is a tool capable model but is bad in actual use. Constraining the output to a tool call format makes it much more reliable. ```python #!/usr/bin/env python3 import ollama def bash(command: str) -> str: """ Execute a bash command on the system. Use this to run shell commands, check files, run programs, etc. Args: command: The bash command to execute """ pass format={ "$schema": "http://json-schema.org/draft-07/schema#", "type": "string", "pattern": r'^<tool_call>\n\{"name": "[^"]*", "arguments": .*?\}\n</tool_call>$' } def test(format=None): response = ollama.chat( model="qwen2.5-coder", messages=[ {"role":"user", "content":"what is the time?"} ], tools=[bash], format=format ) print(f"content : {response.message.content.replace('\n','\\n')}\ntool_call: {response.message.tool_calls}") print("==", "No format", "====") for _ in range(5): test() print("\n==", "Format", "====") for _ in range(5): test(format=format) ``` ``` == No format ==== content : ```json\n{\n "name": "bash",\n "arguments": {\n "command": "date"\n }\n}\n``` tool_call: None content : ```xml\n{"name": "bash", "arguments": {"command":"date"}}\n``` tool_call: None content : ```json\n{"name": "bash", "arguments": {"command": "date"}}\n``` tool_call: None content : ```json\n{"name":"bash", "arguments":{"command":"date"}}\n``` tool_call: None content : ```xml\n{"name": "bash", "arguments": {"command":"date"}}\n``` tool_call: None == Format ==== content : " tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))] content : " tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))] content : " tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))] content : " tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))] content : " tool_call: [ToolCall(function=Function(name='bash', arguments={'command': 'date'}))] ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#9624
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?