github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11158] What LLM output formats trigger Ollama tool call detection? #7360

New Issue
Open
opened 2026-04-12 19:25:01 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-12 19:25:01 -05:00
Owner
Copy Link

Originally created by @matty-fortune on GitHub (Jun 21, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11158

I want to learn more about how tool calls are detected in Ollama, but I've had a difficult time figuring out what the underlying LLM output(s) that triggers a tool call recognition in Ollama should look like. Can anyone help with that? Maybe there's already an example somewhere but I looked and haven't found it.

Originally created by @matty-fortune on GitHub (Jun 21, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11158 I want to learn more about how tool calls are detected in Ollama, but I've had a difficult time figuring out what the underlying LLM output(s) that triggers a tool call recognition in Ollama should look like. Can anyone help with that? Maybe there's already an example somewhere but I looked and haven't found it.
GiteaMirror added the question label 2026-04-12 19:25:01 -05:00
GiteaMirror commented 2026-04-12 19:25:02 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 21, 2025):

Look at the template of a tool using model. The system message will usually contain instructions to the model on how to format a tool call, and the body of the template will contain a .ToolCalls line that indicates the format that the parser should try to match. The parser will fall back to looking for a generic JSON object if no specific tool calling tag is found.

<!-- gh-comment-id:2993820755 --> @rick-github commented on GitHub (Jun 21, 2025): Look at the template of a [tool using model](https://ollama.com/search?c=tools). The system message will usually contain instructions to the model on how to format a tool call, and the body of the template will contain a `.ToolCalls` line that indicates the format that the parser should try to match. The parser will fall back to looking for a generic JSON object if no specific tool calling tag is found.
GiteaMirror commented 2026-04-12 19:25:03 -05:00
Author
Owner
Copy Link

@matty-fortune commented on GitHub (Jun 22, 2025):

@rick-github thank you. I see the template and system message for Devstral are as follows. The system message doesn't seem to contain instructions on how to format a tool call. When running it using a minimal Ollama set up using one of the SDKs, it only outputs a tool call some of the time when it should use a tool. Other times it tries to perform a tool call but uses a format that isn't detected as a tool call. Would this example below be a literal LLM output that would be expected in this case to trigger a tool call?

Example:

[TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}]

Template:

{{- range $index, $_ := .Messages }}
{{- if eq .Role "system" }}[SYSTEM_PROMPT]{{ .Content }}[/SYSTEM_PROMPT]
{{- else if eq .Role "user" }}
{{- if and (le (len (slice $.Messages $index)) 2) $.Tools }}[AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS]
{{- end }}[INST]{{ .Content }}[/INST]
{{- else if eq .Role "assistant" }}
{{- if .Content }}{{ .Content }}
{{- if not (eq (len (slice $.Messages $index)) 1) }}</s>
{{- end }}
{{- else if .ToolCalls }}[TOOL_CALLS][
{{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}
{{- end }}]</s>
{{- end }}
{{- else if eq .Role "tool" }}[TOOL_RESULTS]{"content": {{ .Content }}}[/TOOL_RESULTS]
{{- end }}
{{- end }}

System message:

You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks.

<ROLE>
Your primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed.
* If the user asks a question, like "why is X happening", don't try to fix the problem. Just give an answer to the question.
</ROLE>

<EFFICIENCY>
* Each action you take is somewhat expensive. Wherever possible, combine multiple actions into a single action, e.g. combine multiple bash commands into one, using sed and grep to edit/view multiple files at once.
* When exploring the codebase, use efficient tools like find, grep, and git commands with appropriate filters to minimize unnecessary operations.
</EFFICIENCY>

<FILE_SYSTEM_GUIDELINES>
* When a user provides a file path, do NOT assume it's relative to the current working directory. First explore the file system to locate the file before working on it.
* If asked to edit a file, edit the file directly, rather than creating a new file with a different filename.
* For global search-and-replace operations, consider using `sed` instead of opening file editors multiple times.
</FILE_SYSTEM_GUIDELINES>

<CODE_QUALITY>
* Write clean, efficient code with minimal comments. Avoid redundancy in comments: Do not repeat information that can be easily inferred from the code itself.
* When implementing solutions, focus on making the minimal changes needed to solve the problem.
* Before implementing any changes, first thoroughly understand the codebase through exploration.
* If you are adding a lot of code to a function or file, consider splitting the function or file into smaller pieces when appropriate.
</CODE_QUALITY>

<VERSION_CONTROL>
* When configuring git credentials, use "openhands" as the user.name and "openhands@all-hands.dev" as the user.email by default, unless explicitly instructed otherwise.
* Exercise caution with git operations. Do NOT make potentially dangerous changes (e.g., pushing to main, deleting repositories) unless explicitly asked to do so.
* When committing changes, use `git status` to see all modified files, and stage all files necessary for the commit. Use `git commit -a` whenever possible.
* Do NOT commit files that typically shouldn't go into version control (e.g., node_modules/, .env files, build directories, cache files, large binaries) unless explicitly instructed by the user.
* If unsure about committing certain files, check for the presence of .gitignore files or ask the user for clarification.
</VERSION_CONTROL>

<PULL_REQUESTS>
* When creating pull requests, create only ONE per session/issue unless explicitly instructed otherwise.
* When working with an existing PR, update it with new commits rather than creating additional PRs for the same issue.
* When updating a PR, preserve the original PR title and purpose, updating description only when necessary.
</PULL_REQUESTS>

<PROBLEM_SOLVING_WORKFLOW>
1. EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions
2. ANALYSIS: Consider multiple approaches and select the most promising one
3. TESTING:
   * For bug fixes: Create tests to verify issues before implementing fixes
   * For new features: Consider test-driven development when appropriate
   * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure
   * If the environment is not set up to run tests, consult with the user first before investing time to install all dependencies
4. IMPLEMENTATION: Make focused, minimal changes to address the problem
5. VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests.
</PROBLEM_SOLVING_WORKFLOW>

<SECURITY>
* Only use GITHUB_TOKEN and other credentials in ways the user has explicitly requested and would expect.
* Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing.
</SECURITY>

<ENVIRONMENT_SETUP>
* When user asks you to run an application, don't stop if the application is not installed. Instead, please install the application and run the command again.
* If you encounter missing dependencies:
  1. First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.)
  2. If dependency files exist, use them to install all dependencies at once (e.g., `pip install -r requirements.txt`, `npm install`, etc.)
  3. Only install individual packages directly if no dependency files are found or if only specific packages are needed
* Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible.
</ENVIRONMENT_SETUP>

<TROUBLESHOOTING>
* If you've made repeated attempts to solve a problem but tests still fail or the user reports it's still broken:
  1. Step back and reflect on 5-7 different possible sources of the problem
  2. Assess the likelihood of each possible cause
  3. Methodically address the most likely causes, starting with the highest probability
  4. Document your reasoning process
* When you run into any major issue while executing a plan from the user, please don't try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding.
</TROUBLESHOOTING>
<!-- gh-comment-id:2993855395 --> @matty-fortune commented on GitHub (Jun 22, 2025): @rick-github thank you. I see the template and system message for Devstral are as follows. The system message doesn't seem to contain instructions on how to format a tool call. When running it using a minimal Ollama set up using one of the SDKs, it only outputs a tool call some of the time when it should use a tool. Other times it tries to perform a tool call but uses a format that isn't detected as a tool call. Would this example below be a literal LLM output that would be expected in this case to trigger a tool call? Example: ``` [TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}] ``` Template: ``` {{- range $index, $_ := .Messages }} {{- if eq .Role "system" }}[SYSTEM_PROMPT]{{ .Content }}[/SYSTEM_PROMPT] {{- else if eq .Role "user" }} {{- if and (le (len (slice $.Messages $index)) 2) $.Tools }}[AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS] {{- end }}[INST]{{ .Content }}[/INST] {{- else if eq .Role "assistant" }} {{- if .Content }}{{ .Content }} {{- if not (eq (len (slice $.Messages $index)) 1) }}</s> {{- end }} {{- else if .ToolCalls }}[TOOL_CALLS][ {{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{- end }}]</s> {{- end }} {{- else if eq .Role "tool" }}[TOOL_RESULTS]{"content": {{ .Content }}}[/TOOL_RESULTS] {{- end }} {{- end }} ``` System message: ``` You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks. <ROLE> Your primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed. * If the user asks a question, like "why is X happening", don't try to fix the problem. Just give an answer to the question. </ROLE> <EFFICIENCY> * Each action you take is somewhat expensive. Wherever possible, combine multiple actions into a single action, e.g. combine multiple bash commands into one, using sed and grep to edit/view multiple files at once. * When exploring the codebase, use efficient tools like find, grep, and git commands with appropriate filters to minimize unnecessary operations. </EFFICIENCY> <FILE_SYSTEM_GUIDELINES> * When a user provides a file path, do NOT assume it's relative to the current working directory. First explore the file system to locate the file before working on it. * If asked to edit a file, edit the file directly, rather than creating a new file with a different filename. * For global search-and-replace operations, consider using `sed` instead of opening file editors multiple times. </FILE_SYSTEM_GUIDELINES> <CODE_QUALITY> * Write clean, efficient code with minimal comments. Avoid redundancy in comments: Do not repeat information that can be easily inferred from the code itself. * When implementing solutions, focus on making the minimal changes needed to solve the problem. * Before implementing any changes, first thoroughly understand the codebase through exploration. * If you are adding a lot of code to a function or file, consider splitting the function or file into smaller pieces when appropriate. </CODE_QUALITY> <VERSION_CONTROL> * When configuring git credentials, use "openhands" as the user.name and "openhands@all-hands.dev" as the user.email by default, unless explicitly instructed otherwise. * Exercise caution with git operations. Do NOT make potentially dangerous changes (e.g., pushing to main, deleting repositories) unless explicitly asked to do so. * When committing changes, use `git status` to see all modified files, and stage all files necessary for the commit. Use `git commit -a` whenever possible. * Do NOT commit files that typically shouldn't go into version control (e.g., node_modules/, .env files, build directories, cache files, large binaries) unless explicitly instructed by the user. * If unsure about committing certain files, check for the presence of .gitignore files or ask the user for clarification. </VERSION_CONTROL> <PULL_REQUESTS> * When creating pull requests, create only ONE per session/issue unless explicitly instructed otherwise. * When working with an existing PR, update it with new commits rather than creating additional PRs for the same issue. * When updating a PR, preserve the original PR title and purpose, updating description only when necessary. </PULL_REQUESTS> <PROBLEM_SOLVING_WORKFLOW> 1. EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions 2. ANALYSIS: Consider multiple approaches and select the most promising one 3. TESTING: * For bug fixes: Create tests to verify issues before implementing fixes * For new features: Consider test-driven development when appropriate * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure * If the environment is not set up to run tests, consult with the user first before investing time to install all dependencies 4. IMPLEMENTATION: Make focused, minimal changes to address the problem 5. VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests. </PROBLEM_SOLVING_WORKFLOW> <SECURITY> * Only use GITHUB_TOKEN and other credentials in ways the user has explicitly requested and would expect. * Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing. </SECURITY> <ENVIRONMENT_SETUP> * When user asks you to run an application, don't stop if the application is not installed. Instead, please install the application and run the command again. * If you encounter missing dependencies: 1. First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.) 2. If dependency files exist, use them to install all dependencies at once (e.g., `pip install -r requirements.txt`, `npm install`, etc.) 3. Only install individual packages directly if no dependency files are found or if only specific packages are needed * Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible. </ENVIRONMENT_SETUP> <TROUBLESHOOTING> * If you've made repeated attempts to solve a problem but tests still fail or the user reports it's still broken: 1. Step back and reflect on 5-7 different possible sources of the problem 2. Assess the likelihood of each possible cause 3. Methodically address the most likely causes, starting with the highest probability 4. Document your reasoning process * When you run into any major issue while executing a plan from the user, please don't try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding. </TROUBLESHOOTING> ```
GiteaMirror commented 2026-04-12 19:25:04 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 22, 2025):

The system message doesn't seem to contain instructions on how to format a tool call.

It's not required, a model may be sufficiently trained to generate the required output without explicit instruction. Adherence to tool generation can be enhanced with additional instruction.

Would this example below be a literal LLM output that would be expected in this case to trigger a tool call?

[TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}]

This example looks like it matches the .ToolCalls format. Did it not generate a tool call? How did you define the tool, and what was the prompt?

<!-- gh-comment-id:2994062967 --> @rick-github commented on GitHub (Jun 22, 2025): > The system message doesn't seem to contain instructions on how to format a tool call. It's not required, a model may be sufficiently trained to generate the required output without explicit instruction. Adherence to tool generation can be enhanced with additional instruction. > Would this example below be a literal LLM output that would be expected in this case to trigger a tool call? > > `[TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}]` This example looks like it matches the `.ToolCalls` format. Did it not generate a tool call? How did you define the tool, and what was the prompt?
GiteaMirror commented 2026-04-12 19:25:04 -05:00
Author
Owner
Copy Link

@matty-fortune commented on GitHub (Jun 22, 2025):

@rick-github thank you for the discussion. I'm just interested in learning about Devstral, OpenHands, and Ollama.

I've defined the tools according to the examples. My simple code based on that example does work because sometimes Devstral is able to call tools successfully. I'm trying simple prompts of the style "Solve the following equation: 5x + 7 = 22". Sometimes it tries to perform a tool call but uses a format that isn't detected as a tool call. Note, I've been using maths_solver here solely as an example to illustrate the response formats.

Here are some examples of incorrect tool call formats in responses. I've changed to using CmdRunAction in these examples. It's a basic OpenHands tool.

[{"name": "CmdRunAction", "arguments": {"command":"ls -la"}}]

<FunctionCall>
{
  "name": "CmdRunAction",
  "arguments": {
    "command": "ls -la"
  }
}
</FunctionCall>

<FunctionCall>
Name: CmdRunAction
Arguments:
command: ls -la
</FunctionCall>

<Functions#[CmdRunAction]{
  command": "ls -la"
}>

OpenHands is doing something very different. A contributor to that project says they're using a different, Jinja, template. They seemed to be implying it's inside of the official Devstral model files. OpenHands is also using a modified system message. They define the available tools at the end of it, followed by the tool call format below.

If you choose to call a function ONLY reply in the following format with NO suffix:

<function=example_function_name>
<parameter=example_parameter_1>value_1</parameter>
<parameter=example_parameter_2>
This is the value for the second parameter
that can span
multiple lines
</parameter>
</function>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after.
- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
</IMPORTANT>

While I was playing around with things more, I've noticed that if I set the temperature to 0.0 and the top_p to 1.0, as OpenHands does, Devstral starts reliably outputting the <function=example_function_name> format even without instructing it to do so. I'm assuming this is the default format the model was trained on? Of course, it doesn't get recognized by the Go template.

Anyway, I'm just trying to learn. Thank you for your help. I do have two somewhat related questions if you'd be so kind, and then we could close this issue unless you think there's a valid problem concerning the tool call format mismatch I've detailed.

(1) What does the format of $.Tools in [AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS] look like?

(2) Is there a way in Ollama to somehow log the raw text that ultimately gets sent to the LLM? If so I can answer these sorts of questions myself using that.

Edit: To be clear, I think [TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}] is the expected format in Ollama currently, and the model did not output the above, I wrote it as an example.

<!-- gh-comment-id:2994264568 --> @matty-fortune commented on GitHub (Jun 22, 2025): @rick-github thank you for the discussion. I'm just interested in learning about Devstral, OpenHands, and Ollama. I've defined the tools according to the [examples](https://github.com/ollama/ollama-python/blob/main/examples/tools.py). My simple code based on that example does work because sometimes Devstral is able to call tools successfully. I'm trying simple prompts of the style "Solve the following equation: 5x + 7 = 22". Sometimes it tries to perform a tool call but uses a format that isn't detected as a tool call. Note, I've been using `maths_solver` here solely as an example to illustrate the response formats. Here are some examples of incorrect tool call formats in responses. I've changed to using `CmdRunAction` in these examples. It's a basic [OpenHands tool](https://github.com/All-Hands-AI/OpenHands/blob/main/openhands/events/action/commands.py). ``` [{"name": "CmdRunAction", "arguments": {"command":"ls -la"}}] ``` ``` <FunctionCall> { "name": "CmdRunAction", "arguments": { "command": "ls -la" } } </FunctionCall> ``` ``` <FunctionCall> Name: CmdRunAction Arguments: command: ls -la </FunctionCall> ``` ``` <Functions#[CmdRunAction]{ command": "ls -la" }> ``` OpenHands is doing something very different. A contributor to that project says they're using a different, Jinja, template. They seemed to be implying it's inside of the official Devstral model files. OpenHands is also using a modified system message. They define the available tools at the end of it, followed by the tool call format below. ``` If you choose to call a function ONLY reply in the following format with NO suffix: <function=example_function_name> <parameter=example_parameter_1>value_1</parameter> <parameter=example_parameter_2> This is the value for the second parameter that can span multiple lines </parameter> </function> <IMPORTANT> Reminder: - Function calls MUST follow the specified format, start with <function= and end with </function> - Required parameters MUST be specified - Only call one function at a time - You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after. - If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls </IMPORTANT> ``` While I was playing around with things more, I've noticed that if I set the `temperature` to `0.0` and the `top_p` to `1.0`, as OpenHands does, Devstral starts reliably outputting the `<function=example_function_name>` format even without instructing it to do so. I'm assuming this is the default format the model was trained on? Of course, it doesn't get recognized by the Go template. Anyway, I'm just trying to learn. Thank you for your help. I do have two somewhat related questions if you'd be so kind, and then we could close this issue unless you think there's a valid problem concerning the tool call format mismatch I've detailed. (1) What does the format of `$.Tools` in `[AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS]` look like? (2) Is there a way in Ollama to somehow log the raw text that ultimately gets sent to the LLM? If so I can answer these sorts of questions myself using that. **Edit:** To be clear, I think `[TOOL_CALLS][{"name": "maths_solver", "arguments": {"equation": "5x + 7 = 22", "variable": "x"}}]` is the expected format in Ollama currently, and the model *did not* output the above, I wrote it as an example.
GiteaMirror commented 2026-04-12 19:25:05 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 22, 2025):

(1) What does the format of $.Tools in [AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS] look like?

It's the JSON function signature of the tool definition passed in through the tools argument of the API call.

(2) Is there a way in Ollama to somehow log the raw text that ultimately gets sent to the LLM? If so I can answer these sorts of questions myself using that.

Set OLLAMA_DEBUG=2 in the server environment.

<!-- gh-comment-id:2994275633 --> @rick-github commented on GitHub (Jun 22, 2025): > (1) What does the format of $.Tools in `[AVAILABLE_TOOLS]{{ $.Tools }}[/AVAILABLE_TOOLS]` look like? It's the JSON function signature of the tool definition passed in through the `tools` argument of the API call. > (2) Is there a way in Ollama to somehow log the raw text that ultimately gets sent to the LLM? If so I can answer these sorts of questions myself using that. Set `OLLAMA_DEBUG=2` in the server environment.
GiteaMirror commented 2026-04-12 19:25:05 -05:00
Author
Owner
Copy Link

@raghavan-v commented on GitHub (Jun 22, 2025):

I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different.

<!-- gh-comment-id:2994281069 --> @raghavan-v commented on GitHub (Jun 22, 2025): I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different.
GiteaMirror commented 2026-04-12 19:25:06 -05:00
Author
Owner
Copy Link

@matty-fortune commented on GitHub (Jun 22, 2025):

@raghavan-v well it's dependent on how the specific model was trained, or do you have another case where the preferred tool call format of the model isn't reflected in the Go template? You can see the template and system message by running the command ollama show <model-name> --modelfile with <model-name> replaced with the specific model name, which can be seen with ollama list. It's normal that different models will have different tool call formats since there's not (yet) a universal standard.

<!-- gh-comment-id:2994284542 --> @matty-fortune commented on GitHub (Jun 22, 2025): @raghavan-v well it's dependent on how the specific model was trained, or do you have another case where the preferred tool call format of the model isn't reflected in the Go template? You can see the template and system message by running the command `ollama show <model-name> --modelfile` with `<model-name>` replaced with the specific model name, which can be seen with `ollama list`. It's normal that different models will have different tool call formats since there's not (yet) a universal standard.
GiteaMirror commented 2026-04-12 19:25:06 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 22, 2025):

I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different.

Could you clarify? The API is consistent across all models, the tool definition is set via tools and the model specific template formats the tool instructions in a way that hides the gory details from the client.

<!-- gh-comment-id:2994285081 --> @rick-github commented on GitHub (Jun 22, 2025): > I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different. Could you clarify? The API is consistent across all models, the tool definition is set via `tools` and the model specific template formats the tool instructions in a way that hides the gory details from the client.
GiteaMirror commented 2026-04-12 19:25:07 -05:00
Author
Owner
Copy Link

@matty-fortune commented on GitHub (Jun 22, 2025):

I've done more testing. It's a little hit-or-miss (it's probabilistic after all), but I feel reasonably confident on the following points about using Devstral in Ollama. Maybe I'll continue to be able to refine this view as I play with it more.

  1. Setting the temperature to 0.0 and the top_p to 1.0, as OpenHands does, Devstral in Ollama starts frequently outputting the <function=example_function_name> format even without instructing it to do so, and even instructing it to use the [TOOL_CALLS][{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}] format as specified in the Ollama template for Devstral, it still has a strong tendency to use the <function=example_function_name> format instead.

  2. With the temperature set to 0.0, the top_p set to 1.0 and the instructions to use the <function=example_function_name> format added to the end of the system message, it seems to behave reasonably consistently in using that format and properly calling tools (which don't get correctly identified by Ollama).

  3. For all set ups, the complexity of the prompt seems to matter, with something like Execute the hello.sh script more reliably resulting in correct tool use than something like Create a basic todo app in React.

  4. Using the changes from point 2 seems to result in performance closer to what would be expected for Devstral, however, the default Go template doesn't correctly identify tool calls in the <function=example_function_name> format, so it'd require custom tool call parsing.

  5. The looser the settings, with default everything being the loosest, the more the variety and frequency of hallucinations of improper tool call formats.

  6. It's not clear if the Go templates can handle strings on multiple lines in a tool call, as is being done in the following, but I suppose the system message could just instruct the model to use "\n" within strings in the JSON and maybe that'd work reliably.

If you choose to call a function ONLY reply in the following format with NO suffix:

<function=example_function_name>
<parameter=example_parameter_1>value_1</parameter>
<parameter=example_parameter_2>
This is the value for the second parameter
that can span
multiple lines
</parameter>
</function>

<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after.
- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
</IMPORTANT>

So I'd consider that either the Ollama template for Devstral should be changed to use the <function=example_function_name> format, assuming that's possible, or the system prompt should be modified to exhibit reliable usage of the [TOOL_CALLS][{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}] format, but at least so far in my testing, the built-in tool support in Ollama doesn't seem especially usable for Devstral.

<!-- gh-comment-id:2994378296 --> @matty-fortune commented on GitHub (Jun 22, 2025): I've done more testing. It's a little hit-or-miss (it's probabilistic after all), but I feel reasonably confident on the following points about using Devstral in Ollama. Maybe I'll continue to be able to refine this view as I play with it more. 1. Setting the temperature to 0.0 and the top_p to 1.0, as OpenHands does, Devstral in Ollama starts frequently outputting the `<function=example_function_name>` format even without instructing it to do so, and even instructing it to use the `[TOOL_CALLS][{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}]` format as specified in the Ollama template for Devstral, it still has a strong tendency to use the `<function=example_function_name>` format instead. 2. With the temperature set to 0.0, the top_p set to 1.0 *and* the instructions to use the `<function=example_function_name>` format added to the end of the system message, it seems to behave reasonably consistently in using that format and properly calling tools (which don't get correctly identified by Ollama). 3. For all set ups, the complexity of the prompt seems to matter, with something like `Execute the hello.sh script` more reliably resulting in correct tool use than something like `Create a basic todo app in React`. 4. Using the changes from point 2 seems to result in performance closer to what would be expected for Devstral, however, the default Go template doesn't correctly identify tool calls in the `<function=example_function_name>` format, so it'd require custom tool call parsing. 5. The looser the settings, with default everything being the loosest, the more the variety and frequency of hallucinations of improper tool call formats. 6. It's not clear if the Go templates can handle strings on multiple lines in a tool call, as is being done in the following, but I suppose the system message could just instruct the model to use "\n" within strings in the JSON and maybe that'd work reliably. ``` If you choose to call a function ONLY reply in the following format with NO suffix: <function=example_function_name> <parameter=example_parameter_1>value_1</parameter> <parameter=example_parameter_2> This is the value for the second parameter that can span multiple lines </parameter> </function> <IMPORTANT> Reminder: - Function calls MUST follow the specified format, start with <function= and end with </function> - Required parameters MUST be specified - Only call one function at a time - You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after. - If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls </IMPORTANT> ``` So I'd consider that either the Ollama template for Devstral should be changed to use the `<function=example_function_name>` format, assuming that's possible, or the system prompt should be modified to exhibit reliable usage of the `[TOOL_CALLS][{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}]` format, but at least so far in my testing, the built-in tool support in Ollama doesn't seem especially usable for Devstral.
GiteaMirror commented 2026-04-12 19:25:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 22, 2025):

The tool parser is currently being re-worked to allow it to be more flexible. Prior to 0.8.0 it was pretty rigid in requiring a JSON response, as more work is done it may be able to support the devstral tool format without too much arm twisting.

<!-- gh-comment-id:2994386706 --> @rick-github commented on GitHub (Jun 22, 2025): The tool parser is currently being re-worked to allow it to be more flexible. Prior to 0.8.0 it was pretty rigid in requiring a JSON response, as more work is done it may be able to support the devstral tool format without too much arm twisting.
GiteaMirror commented 2026-04-12 19:25:07 -05:00
Author
Owner
Copy Link

@ParthSareen commented on GitHub (Jul 1, 2025):

I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different.

Hey! What exactly were you running into? @raghavan-v

So I'd consider that either the Ollama template for Devstral should be changed to use the <function=example_function_name> format, assuming that's possible,...
@matty-fortune when we worked with the Devstral team the template you see is what was supplied to us. However, if you're finding the results to be inconsistent we can take another look and see what the model wants to output for its function calling behavior. I don't think parsing is an issue in this case - it's more so what the model outputs for the function call

<!-- gh-comment-id:3025819961 --> @ParthSareen commented on GitHub (Jul 1, 2025): > I face similar issue where each model from Ollama behaves differently with tool call. It’s bit frustrating to see that documentation and behavior are different. Hey! What exactly were you running into? @raghavan-v > So I'd consider that either the Ollama template for Devstral should be changed to use the <function=example_function_name> format, assuming that's possible,... @matty-fortune when we worked with the Devstral team the template you see is what was supplied to us. However, if you're finding the results to be inconsistent we can take another look and see what the model _wants_ to output for its function calling behavior. I don't think parsing is an issue in this case - it's more so what the model outputs for the function call
GiteaMirror commented 2026-04-12 19:25:08 -05:00
Author
Owner
Copy Link

@matty-fortune commented on GitHub (Jul 2, 2025):

@ParthSareen yes, take a look, thanks. Can you get Devstral to reliably produce tool calls in the format [TOOL_CALLS][{"name": "execute_bash", "arguments": {"command": "ls -la"}}] that's expected by the template? If you can, please provide the settings used to do so, as I haven't been able to get it to work reliably. As I stated above, the model seems to want to produce tool calls in the XML-like format below. This is the format that OpenHands uses, with their expanded system prompt instructing the model to do so as well, and Devstral was trained on that set up. Do you think Ollama can currently parse the format below? I don't and I think that's likely why a more standard template might have been provided. I'm definitely hoping you can show a way to get the default template to work reliably for tool calls 🤞

<function=example_function_name>
<parameter=example_parameter_1>value_1</parameter>
<parameter=example_parameter_2>
This is the value for the second parameter
that can span
multiple lines
</parameter>
</function>
<!-- gh-comment-id:3028435147 --> @matty-fortune commented on GitHub (Jul 2, 2025): @ParthSareen yes, take a look, thanks. Can you get Devstral to reliably produce tool calls in the format `[TOOL_CALLS][{"name": "execute_bash", "arguments": {"command": "ls -la"}}]` that's expected by the template? If you can, please provide the settings used to do so, as I haven't been able to get it to work reliably. As I stated above, the model seems to _want_ to produce tool calls in the XML-like format below. This is the format that OpenHands uses, with their expanded system prompt instructing the model to do so as well, and Devstral was trained on that set up. Do you think Ollama can currently parse the format below? I don't and I think that's likely why a more standard template might have been provided. I'm definitely hoping you can show a way to get the default template to work reliably for tool calls 🤞 ``` <function=example_function_name> <parameter=example_parameter_1>value_1</parameter> <parameter=example_parameter_2> This is the value for the second parameter that can span multiple lines </parameter> </function> ```
GiteaMirror commented 2026-04-12 19:25:09 -05:00
Author
Owner
Copy Link

@sumitkm commented on GitHub (Jan 20, 2026):

I am experiencing similar issues with devstral. Thanks to this thread, now I know I am not going nuts.

Also does anyone know what's the difference between calling Ollama Python libraries' client.chat vs calling chat directly? Calling client.chat sends Ollama into a spin and my 4090 screaming until I break out of the executing.

Calling chat gets better responses, but so far no luck with tool calling. It does list out all the tools it has, so it is responding to the tool definitions, but the response doesn't have any tool_calls populated so a generic implementation doesn't work.

Initially I was getting role: function with correct fuction_name and function_parameters but now after I refactored my code, all I am getting is assistant responses.

I'll try with custom system prompt as suggested above.

I found this link on Mistral's site which seems to indicate they do support Ollama's tool calling, but they are using the REST api not the python library https://docs.mistral.ai/cookbooks/third_party-ollama-function_calling_local and doing custom parsing of the tool_calling response.

<!-- gh-comment-id:3775193605 --> @sumitkm commented on GitHub (Jan 20, 2026): I am experiencing similar issues with devstral. Thanks to this thread, now I know I am not going nuts. Also does anyone know what's the difference between calling Ollama Python libraries' `client.chat` vs calling `chat` directly? Calling `client.chat` sends Ollama into a spin and my 4090 screaming until I break out of the executing. Calling `chat` gets better responses, but so far no luck with tool calling. It does list out all the tools it has, so it is responding to the tool definitions, but the response doesn't have any tool_calls populated so a generic implementation doesn't work. Initially I was getting `role: function` with correct `fuction_name` and `function_parameters` but now after I refactored my code, all I am getting is `assistant` responses. I'll try with custom system prompt as suggested above. I found this link on Mistral's site which seems to indicate they do support Ollama's tool calling, but they are using the REST api not the python library https://docs.mistral.ai/cookbooks/third_party-ollama-function_calling_local and doing custom parsing of the tool_calling response.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7360
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?