github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6419] Ollama Tools - random results without providing tools in second call #81716

New Issue
Closed
opened 2026-05-09 12:05:52 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-05-09 12:05:52 -05:00
Owner
Copy Link

Originally created by @jprogramista on GitHub (Aug 19, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6419

What is the issue?

Not always, but very often this script: https://github.com/ollama/ollama-python/blob/main/examples/tools/main.py gives me random answers (I have modified 'NYC-LAX': {'departure': '08:00 AM', 'arrival': '11:30 PM', 'duration': '15h 30m'} and I expect to get 15h 30m from tools, but get 5h 30m or 3h 50m), like tools not included in query - I observe that sometimes the tool answer is included, but more times not. If I include tools in final_response I get tools response included:

final_response = client.chat(model=model, messages=messages, tools=tools)

I have also tried with sync client, and observe same behaviour.

I tried also a REST call:

{
    "model": "llama3.1",
    "messages": [
        {
            "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": {
                            "arrival": "LAX",
                            "departure": "NYC"
                        },
                        "name": "get_flight_info"
                    }
                }
            ]
        },
        {
            "content": "{"from": "NYC", "to": "LAX", "duration": "15h 30m", "price": "150 USD"}",
            "role": "tool"
        }
    ],
    "options": {},
    "stream": false,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_flight_info",
                "description": "Get the flight data",
                "parameters": {
                    "properties": {
                        "arrival": {
                            "description": "The arrival ",
                            "departure": "string"
                        },
                        "departure": {
                            "description": "The departure",
                            "type": "string"
                        }
                    },
                    "required": [
                        "arrival",
                        "departure"
                    ]
                }
            }
        }
    ]
}

which give me 100% times correct results if "tools" node provided, and not correct or 90% of not correct if don't.

Model is 8b.

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.3.6

Originally created by @jprogramista on GitHub (Aug 19, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6419 ### What is the issue? Not always, but very often this script: https://github.com/ollama/ollama-python/blob/main/examples/tools/main.py gives me random answers (I have modified `'NYC-LAX': {'departure': '08:00 AM', 'arrival': '11:30 PM', 'duration': '15h 30m'}` and I expect to get 15h 30m from tools, but get 5h 30m or 3h 50m), like tools not included in query - I observe that sometimes the tool answer is included, but more times not. If I include tools in final_response I get tools response included: ``` final_response = client.chat(model=model, messages=messages, tools=tools) ``` I have also tried with sync client, and observe same behaviour. I tried also a REST call: ``` { "model": "llama3.1", "messages": [ { "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?", "role": "user" }, { "content": "", "role": "assistant", "tool_calls": [ { "function": { "arguments": { "arrival": "LAX", "departure": "NYC" }, "name": "get_flight_info" } } ] }, { "content": "{"from": "NYC", "to": "LAX", "duration": "15h 30m", "price": "150 USD"}", "role": "tool" } ], "options": {}, "stream": false, "tools": [ { "type": "function", "function": { "name": "get_flight_info", "description": "Get the flight data", "parameters": { "properties": { "arrival": { "description": "The arrival ", "departure": "string" }, "departure": { "description": "The departure", "type": "string" } }, "required": [ "arrival", "departure" ] } } } ] } ``` which give me 100% times correct results if "tools" node provided, and not correct or 90% of not correct if don't. Model is 8b. ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.3.6
GiteaMirror added the bug label 2026-05-09 12:05:52 -05:00
GiteaMirror commented 2026-05-09 12:05:57 -05:00
Author
Owner
Copy Link

@mxyng commented on GitHub (Aug 22, 2024):

this seems to be the expected output from llama3.1. it has a tendency to fact check the tool response. for example, changing duration to 15h30m produces additional context in the llama3.1 response clarifying while the typical duration is 5h30m, actual time may depend on factors such as layovers and delays:

The flight time from New York (JFK/LGA/EWR) to Los Angeles (LAX) is approximately 5 hours and 30 minutes. However, please note that this time may vary depending on several factors such as the airline, flight schedule, layovers, and any potential delays.

Here are some general flight times for your reference:

* Non-stop flights from JFK/LGA/EWR to LAX: 5h 30m
* Flights with a layover: 8-10 hours (depending on the layover duration)

Please check with the airlines or a travel booking website like Expedia, Kayak, or Skyscanner for the most up-to-date and accurate flight information.

mistral on the other hand will take the tool response as is without additional clarifications

 It takes approximately 15 hours and 30 minutes to fly from New York (NYC) to Los Angeles (LAX), with a departure time of 8:00 AM.
<!-- gh-comment-id:2303408375 --> @mxyng commented on GitHub (Aug 22, 2024): this seems to be the expected output from llama3.1. it has a tendency to fact check the tool response. for example, changing duration to 15h30m produces additional context in the llama3.1 response clarifying while the typical duration is 5h30m, actual time may depend on factors such as layovers and delays: ``` The flight time from New York (JFK/LGA/EWR) to Los Angeles (LAX) is approximately 5 hours and 30 minutes. However, please note that this time may vary depending on several factors such as the airline, flight schedule, layovers, and any potential delays. Here are some general flight times for your reference: * Non-stop flights from JFK/LGA/EWR to LAX: 5h 30m * Flights with a layover: 8-10 hours (depending on the layover duration) Please check with the airlines or a travel booking website like Expedia, Kayak, or Skyscanner for the most up-to-date and accurate flight information. ``` mistral on the other hand will take the tool response as is without additional clarifications ``` It takes approximately 15 hours and 30 minutes to fly from New York (NYC) to Los Angeles (LAX), with a departure time of 8:00 AM. ```
GiteaMirror commented 2026-05-09 12:05:58 -05:00
Author
Owner
Copy Link

@jprogramista commented on GitHub (Aug 22, 2024):

@mxyng Thank you for your response, maybe I wasn't precise in my question I have no problem with fact checking by llama3.1, but with using tools json node in secondary request, which seems not to be necessary by design (or at least I see it in examples), but in my case it obligatory to get correct result.

Here are 2 examples - the difference is only in adding tools section:

for i in {1..10}; do curl --location 'http://localhost:11434/api/chat' \
--header 'Content-Type: application/json' \
--data '{
    "model": "llama3.1",
    "messages": [
        {
            "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": {
                            "arrival": "LAX",
                            "departure": "NYC"
                        },
                        "name": "get_flight_info"
                    }
                }
            ]
        },
        {
            "content": "{\"from\": \"NYC\", \"to\": \"LAX\", \"duration\": \"15h 30m\", \"price\": \"150 USD\"}",
            "role": "tool"
        }
    ],
    "options": {},
    "stream": false,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_flight_info",
                "description": "Get the flight data",
                "parameters": {
                    "properties": {
                        "arrival": {
                            "description": "The arrival ",
                            "departure": "string"
                        },
                        "departure": {
                            "description": "The departure",
                            "type": "string"
                        }
                    },
                    "required": [
                        "arrival",
                        "departure"
                    ]
                }
            }
        }
    ]
}' | grep -o "15 hours and 30 minutes"; done | wc -l

Response is 10, so tool response was used and no fact checking was done, even if real airports were provided.

for i in {1..10}; do curl --location 'http://localhost:11434/api/chat' \
--header 'Content-Type: application/json' \
--data '{
    "model": "llama3.1",
    "messages": [
        {
            "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?",
            "role": "user"
        },
        {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": {
                            "arrival": "LAX",
                            "departure": "NYC"
                        },
                        "name": "get_flight_info"
                    }
                }
            ]
        },
        {
            "content": "{\"from\": \"NYC\", \"to\": \"LAX\", \"duration\": \"15h 30m\", \"price\": \"150 USD\"}",
            "role": "tool"
        }
    ],
    "options": {},
    "stream": false
}' | grep -o "15 hours and 30 minutes"; done | wc -l

Response is 0, so tool response now is ignored (same if grep is 15h, usual answer is I can’t provide the flight price as it varies depending on airline ... if I don't use loop).

I tried also the version with non-existent airports and get the same results. I also just updated model to be sure that I'm on latest version (llama3.1:latest 91ab477bec9d).

The difference as you can see is only by removing /tools from JSON, and this seems to be not required when I look into examples ( https://github.com/ollama/ollama-python/blob/main/examples/tools/main.py ).

<!-- gh-comment-id:2303625986 --> @jprogramista commented on GitHub (Aug 22, 2024): @mxyng Thank you for your response, maybe I wasn't precise in my question I have no problem with fact checking by llama3.1, but with using tools json node in secondary request, which seems not to be necessary by design (or at least I see it in examples), but in my case it obligatory to get correct result. Here are 2 examples - the difference is only in adding tools section: ``` for i in {1..10}; do curl --location 'http://localhost:11434/api/chat' \ --header 'Content-Type: application/json' \ --data '{ "model": "llama3.1", "messages": [ { "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?", "role": "user" }, { "content": "", "role": "assistant", "tool_calls": [ { "function": { "arguments": { "arrival": "LAX", "departure": "NYC" }, "name": "get_flight_info" } } ] }, { "content": "{\"from\": \"NYC\", \"to\": \"LAX\", \"duration\": \"15h 30m\", \"price\": \"150 USD\"}", "role": "tool" } ], "options": {}, "stream": false, "tools": [ { "type": "function", "function": { "name": "get_flight_info", "description": "Get the flight data", "parameters": { "properties": { "arrival": { "description": "The arrival ", "departure": "string" }, "departure": { "description": "The departure", "type": "string" } }, "required": [ "arrival", "departure" ] } } } ] }' | grep -o "15 hours and 30 minutes"; done | wc -l ``` Response is 10, so tool response was used and no fact checking was done, even if real airports were provided. ``` for i in {1..10}; do curl --location 'http://localhost:11434/api/chat' \ --header 'Content-Type: application/json' \ --data '{ "model": "llama3.1", "messages": [ { "content": "What is the flight price from New York (NYC) to Los Angeles (LAX)?", "role": "user" }, { "content": "", "role": "assistant", "tool_calls": [ { "function": { "arguments": { "arrival": "LAX", "departure": "NYC" }, "name": "get_flight_info" } } ] }, { "content": "{\"from\": \"NYC\", \"to\": \"LAX\", \"duration\": \"15h 30m\", \"price\": \"150 USD\"}", "role": "tool" } ], "options": {}, "stream": false }' | grep -o "15 hours and 30 minutes"; done | wc -l ``` Response is 0, so tool response now is ignored (same if grep is 15h, usual answer is `I can’t provide the flight price as it varies depending on airline ...` if I don't use loop). I tried also the version with non-existent airports and get the same results. I also just updated model to be sure that I'm on latest version (llama3.1:latest 91ab477bec9d). The difference as you can see is only by removing `/tools` from JSON, and this seems to be not required when I look into examples ( https://github.com/ollama/ollama-python/blob/main/examples/tools/main.py ).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#81716
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?