github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-18 10:48:00 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

issue: Incorrect Token Usage Calculation using Function Calling in Native Mode #4913

New Issue
Open
opened 2025-11-11 16:06:21 -06:00 by GiteaMirror · 1 comment
GiteaMirror commented 2025-11-11 16:06:21 -06:00
Owner
Copy Link

Originally created by @ChenxingLi on GitHub (Apr 20, 2025).

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.5

Ollama Version (if applicable)

Not related

Operating System

Ubuntu 22.04

Browser (if applicable)

Not related

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have listed steps to reproduce the bug in detail.

I have not included additional browser or server logs, as I believe the provided network packet records offer more relevant insights into this issue than logs. That said, if you think additional log details might be useful, please don’t hesitate to let me know, and I would be glad to share them.

Principle

Native function calling involves multiple calls to the Completion API:

  • In the first call, the LLM ends with a function call instruction.
  • In the second call, the history (including previous messages and function call results) is sent as the prompt to determine the final response.
  • If multiple function calls are required, additional API calls will be made following the same process.

Expected Behavior

  1. Each API call should include "include_usage": true in the request payload.
  2. The usage data returned by every API call should be captured and aggregated correctly.
  3. For the Filter tool functions, both outlet and stream interface inputs should reflect the full token usage, including all API calls, ensuring accurate reporting and cost calculation across the entire workflow.

Actual Behavior

Open WebUI does not correctly track or calculate token usage during native function calling:

  1. During the second API call (and any subsequent calls), the request does not include "include_usage": true.
  2. As a result, the usage data is not returned for these calls, causing the token usage from subsequent calls to be omitted.
  3. The outlet function of the Filter Tool functions only reflects the token usage from the first API call, while the stream function inputs lack any token usage data entirely.

This behavior results in incomplete and incorrect token cost calculations for workflows involving native function calling.

Steps to Reproduce

  1. Set Up LLM Model

    • Go to Admin Panel > Settings > Connections > Open AI API
    • Add an external OpenAI model link (ensure the model supports function calling, e.g., gpt-4o or claude-3.7).

  2. Set Up a Tool Server

    • Navigate to Admin Panel > Tools > Manage Tool Servers
    • Add an external tool server (e.g., the simplest example of get_current_time from the tutorial).

  3. Enable Usage Capabilities for the Model

    • Go to Admin Panel > Settings > Models.
    • Select the model you are using.
    • Enable Capabilities > Usage for the model.

  4. Configure Conversation Settings

    • Enter the conversation interface.
    • Enable the time tool
    • In conversation settings, select Function Calling mode as Native.

  5. Ask a Function-Calling-Related Question

    • e.g., "Using function calling, tell me the current time in xx timezone."

  6. Capture Network Traffic

    • Use an HTTP network packet capture tool to observe the outgoing API calls from OpenWebUI.
    • You will notice multiple /chat/completions calls made to the OpenAI API. The second call(s) will not include the expected token usage calculation ("include_usage": true).

  7. Check Filter Tool Logs

    • Navigate to Admin Panel > Functions.
    • Add a Filter function that logs the stream and outlet handler outputs.
    • Observe the logs and verify:
      • The outlet function input only includes token usage from the first API call.
      • The stream function input contains no token usage data for the second API call.

Logs & Screenshots

The screenshots below show the HTTP packets exchanged between OpenWebUI and the OpenAI API server during the aforementioned steps. It can be observed that the second communication fails to correctly request token usage.

Image
Image
Image
Image

Additional Information

I am not using the official native API but rather a third-party API. Therefore, I am unsure whether the official native API can still return token usage during the second call, even if include_usage is not requested. However, after testing multiple OpenAI-compatible APIs, I found that none of them return token usage if usage is not explicitly requested.

Originally created by @ChenxingLi on GitHub (Apr 20, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.5 ### Ollama Version (if applicable) Not related ### Operating System Ubuntu 22.04 ### Browser (if applicable) Not related ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. I have not included additional browser or server logs, as I believe the provided **network packet records** offer more relevant insights into this issue than logs. That said, if you think additional log details might be useful, please don’t hesitate to let me know, and I would be glad to share them. ### Principle Native function calling involves multiple calls to the Completion API: - In the first call, the LLM ends with a function call instruction. - In the second call, the history (including previous messages and function call results) is sent as the prompt to determine the final response. - If multiple function calls are required, additional API calls will be made following the same process. ### Expected Behavior 1. Each API call should include `"include_usage": true` in the request payload. 2. The `usage` data returned by every API call should be captured and aggregated correctly. 3. For the `Filter` tool functions, both `outlet` and `stream` interface inputs should reflect the full token usage, including all API calls, ensuring accurate reporting and cost calculation across the entire workflow. ### Actual Behavior Open WebUI does not correctly track or calculate token usage during native function calling: 1. During the second API call (and any subsequent calls), the request does not include `"include_usage": true`. 2. As a result, the `usage` data is not returned for these calls, causing the token usage from subsequent calls to be omitted. 3. The `outlet` function of the `Filter` Tool functions only reflects the token usage from the first API call, while the stream function inputs lack any token usage data entirely. This behavior results in incomplete and incorrect token cost calculations for workflows involving native function calling. ### Steps to Reproduce 1. **Set Up LLM Model** - Go to **Admin Panel > Settings > Connections > Open AI API** - Add an external OpenAI model link (ensure the model supports function calling, e.g., `gpt-4o` or `claude-3.7`). 2. **Set Up a Tool Server** - Navigate to **Admin Panel > Tools > Manage Tool Servers** - Add an external tool server (e.g., the simplest example of `get_current_time` from the tutorial). 3. **Enable Usage Capabilities for the Model** - Go to Admin Panel > Settings > Models. - Select the model you are using. - Enable Capabilities > Usage for the model. 4. **Configure Conversation Settings** - Enter the conversation interface. - Enable the time tool - In conversation settings, select `Function Calling` mode as **Native**. 5. **Ask a Function-Calling-Related Question** - e.g., "Using function calling, tell me the current time in xx timezone." 6. **Capture Network Traffic** - Use an HTTP network packet capture tool to observe the outgoing API calls from OpenWebUI. - You will notice multiple `/chat/completions` calls made to the OpenAI API. The second call(s) will not include the expected token usage calculation (`"include_usage": true`). 7. **Check Filter Tool Logs** - Navigate to **Admin Panel > Functions**. - Add a `Filter` function that logs the `stream` and `outlet` handler outputs. - Observe the logs and verify: - The `outlet` function input only includes token usage from the first API call. - The `stream` function input contains no token usage data for the second API call. ### Logs & Screenshots The screenshots below show the HTTP packets exchanged between OpenWebUI and the OpenAI API server during the aforementioned steps. It can be observed that the second communication fails to correctly request token usage. ![Image](https://github.com/user-attachments/assets/c95976ca-104d-4a9f-87da-799047223591) ![Image](https://github.com/user-attachments/assets/cf71232d-cf45-46d3-aa7f-f6df7eb4f1ce) ![Image](https://github.com/user-attachments/assets/7a97784b-9f33-43d7-a6b9-ce88bd6cf995) ![Image](https://github.com/user-attachments/assets/c5930ab5-9efc-48d9-837c-ab2949375778) ### Additional Information I am not using the official native API but rather a third-party API. Therefore, I am unsure whether the official native API can still return token usage during the second call, even if `include_usage` is not requested. However, after testing multiple OpenAI-compatible APIs, I found that none of them return token usage if `usage` is not explicitly requested.
GiteaMirror added the bug label 2025-11-11 16:06:21 -06:00
GiteaMirror commented 2025-11-11 16:06:24 -06:00
Author
Owner
Copy Link

@rgaricano commented on GitHub (Apr 23, 2025):

On CommVersion Outlet filters are not triggered on API calls.
https://github.com/open-webui/open-webui/discussions/8722#discussioncomment-12193783

@rgaricano commented on GitHub (Apr 23, 2025): On CommVersion Outlet filters are not triggered on API calls. https://github.com/open-webui/open-webui/discussions/8722#discussioncomment-12193783
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#4913
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?