github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 08:30:05 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #15850] Token counts not updated for input tokens before a tool call. #72161

New Issue
Open
opened 2026-05-05 03:34:25 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-05 03:34:25 -05:00
Owner
Copy Link

Originally created by @xxxajk on GitHub (Apr 28, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15850

What is the issue?

I have been running some simple tests, and found a really interesting anomaly.
The failure is that the input tokens are never calculated before a tool call.
Output counts are good as far as I can tell from the tests.

I am interfacing using pydantic and the openai API, non-streaming mode,

I have four tools, two rules for the llm, and five easy tests that show the bug:

def token_test(ctx: RunContext[None]) -> Dict[str, str]:
    """
        This tool returns token statistics to the LLM.
        
        Returns:
            A dictionary containing token statistics.
    """
    print(f"Stats:\n in:{ctx.usage.input_tokens}\nout:{ctx.usage.output_tokens}\nall:{ctx.usage.total_
    return { 
        'input_tokens': ctx.usage.input_tokens,
        'output_tokens': ctx.usage.output_tokens,
        'total_tokens': ctx.usage.total_tokens
    }

def chat(message: str = None) -> None:
    """
    Prints the agent's thinking messages to the console for the human to see.

    Args:
        message: Chat message
        
    Returns:
        None
    """
    if(message is not None):
        print(f"\nAgent:\n{message}\n")

def get_cwd() -> Dict[str, Any]:
    """
    Get the current working directory for the agent.

    Returns:
        Dictionary with:
            current working directory 'path' and 'status' of the command
            'error' reason for failure
            MIGHT include "next_step_instruction" if present, the next tasks the LLM **MUST** do immediately.
    """
    print("Agent tool get_cwd")
    try:
        path = os.getcwd()
        if is_longtime():
            return {
                "next_step_instruction": "You must stop, save all persistent data and checkpoint_subgoal now.",
                'status': "success",
                'path': path
            }
        return {
            'status': "success",
            'path': path
        }
    except Exception as e:
        return {
            "next_step_instruction": "Try again with proper parameters.",
            'status': "error",
            'path': ".",
            'error': str(e)
        }
    
def set_cwd(path: str = None) -> Dict[str, Any]:
    """
    Change working directory for the agent.

    Args:
        path: directory to change to
    Returns:
        Dictionary with:
            current working directory 'path', and 'status'.
            'error' reason for failure
            MIGHT include "next_step_instruction" if present, the next tasks the LLM **MUST** do immediately.
    """
    print(f"Agent tool set_cwd path='{path}'")
    if (path is not None):
        try:
            os.chdir(path)
            if is_longtime():
                return {
                    "next_step_instruction": "You must stop, save all persistent data and checkpoint_subgoal now.",
                    'status': "success",
                    'path': os.getcwd()
                }
            return {
                'status': "success",
                'path': os.getcwd()
            }
        except Exception as e:
            return {
                "next_step_instruction": "Try again with proper parameters.",
                'status': "error",
                'path': os.getcwd(),
                'error': str(e)
            }
    return {
        "next_step_instruction": "Try again with proper parameters.",
        'status': "error",
        'path': os.getcwd(),
        'error': "Path not specified."
    }

Rules in system prompt:

### **Mandatory** important rules you **must** **always follow** **exactly**:
    ::: You **MUST** **ALWAYS** **Replace any internal monologue tags with `chat` tool calls.** **The user can not see internal thinking blocks.**
    ::: **ALWAYS** use the `chat` tool only for displaying intermediate thoughts, reasoning, and thought processes. Do not use it for final answers.

tests:

test1:
call token_test
call token_test 

test2:
call token_test, then think about the results
call token_test 

test3: 
call token_test, then use chat tool to show me the result
call token_test

test4:
call token_test
get your current directory
call token_test
change directory to "."
call token_test

test5:
call token_test
get your current directory
call token_test
change to the reported current directory
call token_test

Results from the tool tests:
test1: No input or output change.

Stats:
 in:4534
out:21
all:4555

Stats:
 in:4534
out:21
all:4555

test2: Force user to use the chat tool

Stats:
 in:4702
out:13
all:4715

Agent:
The first call to `token_test` shows 4,702 input tokens and 13 output tokens. This provides a snapshot of the current context window usage. I will now proceed to the second call as requested.
Stats:
 in:9451
out:84
all:9535

test3: explicitly tell the llm to use the tool

Stats:
 in:4971
out:13
all:4984

Agent:
The result of the first token_test call is: {"input_tokens": 4971, "output_tokens": 13, "total_tokens": 4984}

Stats:
 in:9989
out:79
all:10068

test4: get information, and don't use it

Stats:
 in:5994
out:51
all:6045


Agent tool get_cwd
Stats:
 in:5994
out:51
all:6045


Agent tool set_cwd path='.'
Stats:
 in:5994
out:51
all:6045

test5: Get information and use it Model will anticipate/plan to use the directory it discovered.

Stats:
 in:6385
out:21
all:6406


Agent tool get_cwd
Stats:
 in:12857
out:66
all:12923


Agent tool set_cwd path='/home/xxxajk/Cotton'
Stats:
 in:12857
out:66
all:12923

Relevant log output

OS

Linux

GPU

Nvidia, AMD

CPU

AMD

Ollama version

0.21.2

Originally created by @xxxajk on GitHub (Apr 28, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15850 ### What is the issue? I have been running some simple tests, and found a really interesting anomaly. The failure is that the input tokens are never calculated before a tool call. Output counts are good as far as I can tell from the tests. I am interfacing using pydantic and the openai API, non-streaming mode, I have four tools, two rules for the llm, and five easy tests that show the bug: ``` def token_test(ctx: RunContext[None]) -> Dict[str, str]: """ This tool returns token statistics to the LLM. Returns: A dictionary containing token statistics. """ print(f"Stats:\n in:{ctx.usage.input_tokens}\nout:{ctx.usage.output_tokens}\nall:{ctx.usage.total_ return { 'input_tokens': ctx.usage.input_tokens, 'output_tokens': ctx.usage.output_tokens, 'total_tokens': ctx.usage.total_tokens } def chat(message: str = None) -> None: """ Prints the agent's thinking messages to the console for the human to see. Args: message: Chat message Returns: None """ if(message is not None): print(f"\nAgent:\n{message}\n") def get_cwd() -> Dict[str, Any]: """ Get the current working directory for the agent. Returns: Dictionary with: current working directory 'path' and 'status' of the command 'error' reason for failure MIGHT include "next_step_instruction" if present, the next tasks the LLM **MUST** do immediately. """ print("Agent tool get_cwd") try: path = os.getcwd() if is_longtime(): return { "next_step_instruction": "You must stop, save all persistent data and checkpoint_subgoal now.", 'status': "success", 'path': path } return { 'status': "success", 'path': path } except Exception as e: return { "next_step_instruction": "Try again with proper parameters.", 'status': "error", 'path': ".", 'error': str(e) } def set_cwd(path: str = None) -> Dict[str, Any]: """ Change working directory for the agent. Args: path: directory to change to Returns: Dictionary with: current working directory 'path', and 'status'. 'error' reason for failure MIGHT include "next_step_instruction" if present, the next tasks the LLM **MUST** do immediately. """ print(f"Agent tool set_cwd path='{path}'") if (path is not None): try: os.chdir(path) if is_longtime(): return { "next_step_instruction": "You must stop, save all persistent data and checkpoint_subgoal now.", 'status': "success", 'path': os.getcwd() } return { 'status': "success", 'path': os.getcwd() } except Exception as e: return { "next_step_instruction": "Try again with proper parameters.", 'status': "error", 'path': os.getcwd(), 'error': str(e) } return { "next_step_instruction": "Try again with proper parameters.", 'status': "error", 'path': os.getcwd(), 'error': "Path not specified." } ``` Rules in system prompt: ``` ### **Mandatory** important rules you **must** **always follow** **exactly**: ::: You **MUST** **ALWAYS** **Replace any internal monologue tags with `chat` tool calls.** **The user can not see internal thinking blocks.** ::: **ALWAYS** use the `chat` tool only for displaying intermediate thoughts, reasoning, and thought processes. Do not use it for final answers. ``` tests: ``` test1: call token_test call token_test test2: call token_test, then think about the results call token_test test3: call token_test, then use chat tool to show me the result call token_test test4: call token_test get your current directory call token_test change directory to "." call token_test test5: call token_test get your current directory call token_test change to the reported current directory call token_test ``` Results from the tool tests: test1: No input or output change. ``` Stats: in:4534 out:21 all:4555 Stats: in:4534 out:21 all:4555 ``` test2: Force user to use the chat tool ``` Stats: in:4702 out:13 all:4715 Agent: The first call to `token_test` shows 4,702 input tokens and 13 output tokens. This provides a snapshot of the current context window usage. I will now proceed to the second call as requested. Stats: in:9451 out:84 all:9535 ``` test3: explicitly tell the llm to use the tool ``` Stats: in:4971 out:13 all:4984 Agent: The result of the first token_test call is: {"input_tokens": 4971, "output_tokens": 13, "total_tokens": 4984} Stats: in:9989 out:79 all:10068 ``` test4: get information, and don't use it ``` Stats: in:5994 out:51 all:6045 Agent tool get_cwd Stats: in:5994 out:51 all:6045 Agent tool set_cwd path='.' Stats: in:5994 out:51 all:6045 ``` test5: Get information and use it Model will anticipate/plan to use the directory it discovered. ``` Stats: in:6385 out:21 all:6406 Agent tool get_cwd Stats: in:12857 out:66 all:12923 Agent tool set_cwd path='/home/xxxajk/Cotton' Stats: in:12857 out:66 all:12923 ``` ### Relevant log output ```shell ``` ### OS Linux ### GPU Nvidia, AMD ### CPU AMD ### Ollama version 0.21.2
GiteaMirror added the bug label 2026-05-05 03:34:25 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#72161
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?