mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 23:21:44 -05:00
[GH-ISSUE #24950] feat: lazy loading / selective injection of tool schemas to reduce token usage #123750
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#123750
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @rpostulart on GitHub (May 20, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24950
Summary
When tools are configured on a model in OpenWebUI, their full JSON schemas are included in every API request — even when the user doesn't use them and even when the user has toggled them off in the chat UI. There is no lazy loading or selective injection.
Current behavior
All configured tool schemas are always sent in the
toolsarray of every request, regardless of:Measured impact
Tested on Claude Sonnet 4.6 via AWS Bedrock (production environment):
~3,500 extra tokens per request just from tool schema definitions being sent upfront, before any output is generated.
With heavier tool sets (e.g. 39 MCP tools via Jira + Confluence): ~1,700 extra tokens. With full MCP gateway (~88 tools): ~11,000 extra tokens per request.
Expected / requested behavior
One or more of the following improvements would significantly reduce unnecessary token usage:
Workaround
Currently the only effective workaround is maintaining separate model configurations — one without tools (low token cost) and one with tools (opt-in). This is cumbersome for end users.
Related
Environment
@owui-terminator[bot] commented on GitHub (May 20, 2026):
⚠️ Missing Issue Title Prefix
@rpostulart, your issue title is missing a categorising prefix (e.g.
bug:,feat:,docs:).Please update the title to lead with one of:
Example:
bug: Login fails when the password contains special characters@owui-terminator[bot] commented on GitHub (May 20, 2026):
🔍 Related Issues Found
I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:
🟣 #15853 feat: select tools from OpenAPI Tool Server
This request is about selectively choosing which OpenAPI/tool schemas are available in a conversation instead of sending every configured tool. That overlaps with the same core need to reduce unnecessary tool injection, though it focuses more on manual selection than token cost.
by kenkwt921
🟣 #17957 feat: Align Model Configuration Logic for External Tools and Internal Features/Capabilities (WebSearch, Code Execution, Image Generation.)
This issue discusses model-level control over external tools and making tool availability consistent with internal features. It is related because the new request also asks for per-model/per-turn control over when tools are active and injected, rather than always including all configured tools.
by Maximilian-Pichler
🟣 #21760 ** OpenAPI tool servers — frontend sends empty tool_servers=[] so middleware never injects schemas**
This is the closest technical bug: the request payload's tool server data is not being passed correctly, so middleware never injects tool schemas. While the direction is opposite of the new issue, it is directly about the same tool-schema injection path and request payload fields.
by antonioromero-pm ·
bug🟣 #24019 issue: self.toggle does not work anymore
This is the related chat UI toggle bug referenced in the new issue. It covers the failure of the toggle state to affect behavior, which is part of the requested per-turn tool activation control.
by AIWintermuteAI ·
bug🟣 #19390 issue: tools will double the token cost
This issue reports that tools significantly increase token cost, which is the same impact being highlighted here. It doesn't address lazy loading specifically, but it is directly related to the token-usage problem caused by tool handling.
by FujinoXiao ·
bug💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.
This comment was generated automatically. React with 👍 if helpful, 👎 if not.
@Classic298 commented on GitHub (May 20, 2026):
This implies there has to be a task model that decides if loading the tools is necessary. Costs one extra completion call to a task model per assistant turn, which might just negate any gains since you have to send the last few turns to the task model to determine if it's worth injecting the tools.
So even with super cheap models like gemini 2.5 flash lite (where you will also get mixed results because its a very dumb model), you will pay, lets estimate 20k tokens for the last 3 turns you send to the task model, you still pay 0.001$ for the input tokens alone for each request. No caching applies here, because you cannot cache it. The model only gets the last N messages (much older messages are irrelevant to the tool provision decision AND introduce extra costs), and since the last N messages are always a unique input you don't get caching benefits on those. So you always pay the full input price on them. Which, even on a cheap model, still adds up. Gemini 2.5 Flash Lite costs 0.10
per 1 mil tokens. Which is nothing. But even that is a lot. My example with roughly 20k input tokens over the last few turns is realistic, i would argue and comes out to 0.001just for the input tokens. (I didnt calculate output here).If you instead send the 3500 extra tokens to your main model, e.g. GPT 5.4, that will be an extra 0.008$, which is more, BUT, you only pay that the first time. All subsequent messages will have the tool prompts/tool schema be cached input. So that will be 0,0008$ which is actually CHEAPER than gemini 2.5 flash lite - and not only that, the tool calling will be reliable because the smart main big model decides which tools to call, it always has the tools available and can just use them if needed.
Furthermore, on dynamic tool injection, you WILL break your KV cache, introducing massive costs on your main model.
Do you remove the tools in the next turn? Do you keep them? If you keep them, what is the benefit of even deciding whether to manually add them? If you remove them again, you will break KV cache AGAIN.
So this might not be a good idea.
Also you introduce extra latency to have the task model evaluate this for each turn,
So: it's actually more expensive, you will break your KV cache on your main model causing even more costs, it will be unreliable and it will introduce latency. Just sending the tool schema straight out will be CHEAPER thanks to caching on the main model, more reliable and not introduce latency.
How can I reproduce this, because I can't. If i disable a tool in the UI, it's tool prompt/schema is simply not included in the request.
Also i just realized - the issue you quote isn't even about the chat UI tool toggle not being respected. It is about tools being injected in API requests. And this can be turned off by sending the respective feature flags in your api request.
Already the case. Disabling web search disables all search and web fetch related tools. Disabling Image disables all image creation and image editing tools. And for custom tools you installed, it's your job to group them into one tool file, which then appears as one tool toggle, and disabling it will stop sending all the tool schema to a model.
You omitted any steps to reproduce for your claim that the toggles aren't being respected and frankly I can't reproduce it. If i disable some of my tools in a fresh chat and in another fresh chat leave them enabled, i clearly see the difference in input tokens
This is far from the latest version. If you experience any issues with tools, it is likely because you are on a 1+ year old version.
If you truly have too many tools, i recommend following Anthropic's tool search tool schema, which is a tool the model can call to search for tools. This way, none are available by default (or very little) and via the tool search tool, the model can find available tools and call them through a dedicated tool execute tool.
@rpostulart commented on GitHub (May 20, 2026):
Thanks for the detailed response — some clarifications:
On the version: I'm on 0.9.5, not 0.6.x. Apologies for the imprecise version string in the issue.
On lazy loading breaking KV cache: You're right that a separate task model call is a bad idea for exactly the reasons you describe. That's not what I had in mind. The approach I'm suggesting is different:
Instead of sending full tool schemas upfront, send a lightweight tool index — just name + one-line description for each tool — and expose a single
get_tool_schema(tool_name)meta-tool. When the model determines it needs a specific tool, it callsget_tool_schemato retrieve the full schema, then invokes the tool.This keeps the static index small (cache-friendly), doesn't require an extra LLM call, and lets the main model make the decision. Similar to how Anthropic's tool search pattern works, but without the search complexity for smaller tool sets.
For our setup (39 MCP tools = ~1,700 tokens), a name+description index would be maybe 200–300 tokens instead. The full schema is only fetched when actually needed.
On the toggle bug: I'll re-test on 0.9.5. The behavior I measured earlier may have been specific to our configuration (built-in tools configured at admin model level, not per-chat).
@Classic298 commented on GitHub (May 20, 2026):
So if you want this for your own external tools, why not do it via a tool?
You can have a tool that exposes all your MCP tools exactly in this way to the model as you described it - tool name and short description - and has a tool which the model can call to get the schema