[PR #24707] [CLOSED] fix: server-side native tool-calling loop for REST API requests #115105

New Issue

GiteaMirror · 2026-05-18T16:03:41-05:00

GiteaMirror commented

2026-05-18 16:03:41 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/24707
Author: @PedroFCM
Created: 5/14/2026
Status: ❌ Closed

Base: dev ← Head: fix/native-tool-calling-api-loop-v2

📝 Commits (1)

fe80419 fix: real-time SSE streaming for native tool-calling loop

📊 Changes

1 file changed (+169 additions, -7 deletions)

View changed files

📝 backend/open_webui/main.py (+169 -7)

📄 Description

Summary

Problem: REST API consumers using function_calling: native with tool_ids get raw tool_calls back — tools are never executed because the execution loop requires a WebSocket session managed by the frontend. This makes the REST API unusable for native tool calling, which is a common need for programmatic integrations, automation pipelines, and custom clients.

This PR adds a server-side tool-calling loop that activates only for REST API requests (no session_id) with function_calling == 'native' and tools present. It does not change the WebSocket path at all.

Why this follow-up to #23581

The previous PR was closed with concerns about scope and approach. This implementation addresses those concerns:

Minimal scope — only activates for REST API requests without a session. Zero impact on the WebSocket/frontend path.
Reuses existing infrastructure — calls execute_tool_server, _split_tool_calls, process_tool_result, and get_updated_tool_function from the middleware. No new abstractions.
MCP-compatible — runs the tool loop eagerly in the request context so MCP client lifecycle (anyio task groups) is respected.
Production-validated — deployed and load-tested on AWS ECS Fargate with MCP tool servers (0% errors, 100% tool execution reliability across 14 requests with 2 concurrent users).
Community demand — #9435 and multiple users on #23581 confirm this is a widely needed feature.

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions to discuss your idea/fix with the community before creating a pull request, and describe your changes before submitting a pull request.

This is to ensure large feature PRs are discussed with the community first, before starting work on it. If the community does not want this feature or it is not relevant for Open WebUI as a project, it can be identified in the discussion before working on the feature and submitting the PR.

Before submitting, make sure you've checked the following:

Linked Issue/Discussion: Relates to #9435 — Follow-up to #23581 (maintainer concerns addressed in this comment)
Target branch: dev
Description: See changelog below
Changelog: See below
Documentation: N/A — internal API behavior, no new env vars or UI changes
Dependencies: None
Testing: Manually tested non-streaming and streaming tool-calling with MCP tool servers. Load tested with k6 (2 VUs, 0% errors, 100% tool execution reliability). Tested on a production-like ECS Fargate deployment.
Agentic AI Code: Human-reviewed and manually tested. AI-assisted development with full human oversight.
Code review: Self-reviewed + team code review (4 reviewers)
Design & Architecture: Minimal change — adds tool-calling loop only for REST API requests without a WebSocket session. No new settings, no UI changes.
Git Hygiene: Single atomic commit on latest dev
Title Prefix: fix

Changelog Entry

Description

REST API consumers using /api/chat/completions with function_calling: native and tool_ids never get tools executed server-side — tool calls are returned raw because the execution loop relies on a WebSocket session managed by the frontend. This PR adds a server-side tool-calling loop for the REST API path (no session_id), making native tool calling work for API-only consumers.

This is a follow-up to #23581 which was closed. The community interest remains (#9435), and this implementation addresses the feedback from @MrMakkarone about multiple tool calls being broken.

How to reproduce the issue (before this fix):

curl -X POST http://localhost:8080/api/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [{"role": "user", "content": "Search for X"}],
    "tool_ids": ["server:mcp:your-mcp-server"],
    "params": {"function_calling": "native"},
    "stream": false
  }'

Result: Response contains raw tool_calls that are never executed. The LLM asks to call tools but nobody executes them.

After this fix: Tools are executed server-side, results are fed back to the LLM, and the final answer is returned to the caller.

Added

Server-side tool-calling loop for REST API requests (activated when: no session_id + function_calling == 'native' + tools present)
Parallel tool execution via asyncio.gather with per-tool 30s timeout
MCP/direct tool server support via execute_tool_server
Concatenated JSON argument splitting via _split_tool_calls
Streaming support: tool loop runs eagerly, then final LLM response streams back

Changed

function_calling parameter passes through configured value as-is instead of only recognizing 'native' and falling back to 'default'

Deprecated

N/A

Removed

N/A

Fixed

REST API consumers can now use native tool calling without a WebSocket session
Multiple parallel tool calls in a single LLM response are handled correctly
tool_result_files and tool_result_embeds are preserved in tool messages

Security

N/A

Breaking Changes

None — the change only activates for requests that previously returned unexecuted tool calls

Additional Information

The tool loop runs eagerly in the request context (not via async generator) because MCP clients are disconnected in the finally block after the response is returned — a lazy generator would lose the MCP session before tool execution begins
Argument parsing uses ast.literal_eval primary with json.loads fallback, matching the middleware's WebSocket path
Citation extraction is not implemented for this path (WebSocket-only feature tied to event emitter) — documented as known limitation
Tools execute in parallel (unlike the sequential WebSocket path) for lower latency; tools with ordering dependencies within a single batch may be non-deterministic
Tested in production on AWS ECS Fargate with MCP tool servers

Screenshots or Videos

N/A — API-only change, no UI impact. Validated via curl and k6 load tests.

Testing Evidence

Manually tested on AWS ECS Fargate with OpenWebUI 0.9.2 + this patch, using MCP tool servers (streamable HTTP).

✅ Non-streaming: tools executed server-side, final answer includes tool results
✅ Streaming: SSE stream with content tokens after tool loop completes
✅ k6 load test (2 VUs, 2 min): 0% errors, 14/14 requests successful, p95 duration 20.8s

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/24707 **Author:** [@PedroFCM](https://github.com/PedroFCM) **Created:** 5/14/2026 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `fix/native-tool-calling-api-loop-v2` --- ### 📝 Commits (1) - [`fe80419`](https://github.com/open-webui/open-webui/commit/fe8041902cbecda6806c5a6c268c0ec3d47ada10) fix: real-time SSE streaming for native tool-calling loop ### 📊 Changes **1 file changed** (+169 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/main.py` (+169 -7) </details> ### 📄 Description  ## Summary **Problem:** REST API consumers using `function_calling: native` with `tool_ids` get raw `tool_calls` back — tools are never executed because the execution loop requires a WebSocket session managed by the frontend. This makes the REST API unusable for native tool calling, which is a common need for programmatic integrations, automation pipelines, and custom clients. **This PR adds a server-side tool-calling loop** that activates only for REST API requests (no `session_id`) with `function_calling == 'native'` and tools present. It does not change the WebSocket path at all. ### Why this follow-up to #23581 The [previous PR](https://github.com/open-webui/open-webui/pull/23581) was closed with concerns about scope and approach. This implementation addresses those concerns: 1. **Minimal scope** — only activates for REST API requests without a session. Zero impact on the WebSocket/frontend path. 2. **Reuses existing infrastructure** — calls `execute_tool_server`, `_split_tool_calls`, `process_tool_result`, and `get_updated_tool_function` from the middleware. No new abstractions. 3. **MCP-compatible** — runs the tool loop eagerly in the request context so MCP client lifecycle (anyio task groups) is respected. 4. **Production-validated** — deployed and load-tested on AWS ECS Fargate with MCP tool servers (0% errors, 100% tool execution reliability across 14 requests with 2 concurrent users). 5. **Community demand** — [#9435](https://github.com/open-webui/open-webui/discussions/9435) and [multiple users on #23581](https://github.com/open-webui/open-webui/pull/23581#issuecomment-4296376255) confirm this is a widely needed feature. --- # Pull Request Checklist ### Note to first-time contributors: Please open a discussion post in [Discussions](https://github.com/open-webui/open-webui/discussions) to discuss your idea/fix with the community before creating a pull request, and describe your changes before submitting a pull request. This is to ensure large feature PRs are discussed with the community first, before starting work on it. If the community does not want this feature or it is not relevant for Open WebUI as a project, it can be identified in the discussion before working on the feature and submitting the PR.  **Before submitting, make sure you've checked the following:** - [x] **Linked Issue/Discussion:** Relates to [#9435](https://github.com/open-webui/open-webui/discussions/9435) — Follow-up to [#23581](https://github.com/open-webui/open-webui/pull/23581) (maintainer concerns addressed in [this comment](https://github.com/open-webui/open-webui/pull/23581#issuecomment-REPLACE_WITH_YOUR_COMMENT_ID)) - [x] **Target branch:** `dev` - [x] **Description:** See changelog below - [x] **Changelog:** See below - [x] **Documentation:** N/A — internal API behavior, no new env vars or UI changes - [x] **Dependencies:** None - [x] **Testing:** Manually tested non-streaming and streaming tool-calling with MCP tool servers. Load tested with k6 (2 VUs, 0% errors, 100% tool execution reliability). Tested on a production-like ECS Fargate deployment. - [x] **Agentic AI Code:** Human-reviewed and manually tested. AI-assisted development with full human oversight. - [x] **Code review:** Self-reviewed + team code review (4 reviewers) - [x] **Design & Architecture:** Minimal change — adds tool-calling loop only for REST API requests without a WebSocket session. No new settings, no UI changes. - [x] **Git Hygiene:** Single atomic commit on latest `dev` - [x] **Title Prefix:** `fix` # Changelog Entry ### Description REST API consumers using `/api/chat/completions` with `function_calling: native` and `tool_ids` never get tools executed server-side — tool calls are returned raw because the execution loop relies on a WebSocket session managed by the frontend. This PR adds a server-side tool-calling loop for the REST API path (no `session_id`), making native tool calling work for API-only consumers. This is a follow-up to [#23581](https://github.com/open-webui/open-webui/pull/23581) which was closed. The community interest remains ([#9435](https://github.com/open-webui/open-webui/discussions/9435)), and this implementation addresses the feedback from [@MrMakkarone](https://github.com/open-webui/open-webui/pull/23581#issuecomment-4296376255) about multiple tool calls being broken. **How to reproduce the issue (before this fix):** ```bash curl -X POST http://localhost:8080/api/chat/completions \ -H "Authorization: Bearer <key>" \ -H "Content-Type: application/json" \ -d '{ "model": "your-model", "messages": [{"role": "user", "content": "Search for X"}], "tool_ids": ["server:mcp:your-mcp-server"], "params": {"function_calling": "native"}, "stream": false }' ``` Result: Response contains raw `tool_calls` that are never executed. The LLM asks to call tools but nobody executes them. **After this fix:** Tools are executed server-side, results are fed back to the LLM, and the final answer is returned to the caller. ### Added - Server-side tool-calling loop for REST API requests (activated when: no `session_id` + `function_calling == 'native'` + tools present) - Parallel tool execution via `asyncio.gather` with per-tool 30s timeout - MCP/direct tool server support via `execute_tool_server` - Concatenated JSON argument splitting via `_split_tool_calls` - Streaming support: tool loop runs eagerly, then final LLM response streams back ### Changed - `function_calling` parameter passes through configured value as-is instead of only recognizing `'native'` and falling back to `'default'` ### Deprecated - N/A ### Removed - N/A ### Fixed - REST API consumers can now use native tool calling without a WebSocket session - Multiple parallel tool calls in a single LLM response are handled correctly - `tool_result_files` and `tool_result_embeds` are preserved in tool messages ### Security - N/A ### Breaking Changes - None — the change only activates for requests that previously returned unexecuted tool calls --- ### Additional Information - The tool loop runs eagerly in the request context (not via async generator) because MCP clients are disconnected in the `finally` block after the response is returned — a lazy generator would lose the MCP session before tool execution begins - Argument parsing uses `ast.literal_eval` primary with `json.loads` fallback, matching the middleware's WebSocket path - Citation extraction is not implemented for this path (WebSocket-only feature tied to event emitter) — documented as known limitation - Tools execute in parallel (unlike the sequential WebSocket path) for lower latency; tools with ordering dependencies within a single batch may be non-deterministic - Tested in production on AWS ECS Fargate with MCP tool servers ### Screenshots or Videos N/A — API-only change, no UI impact. Validated via curl and k6 load tests. ### Testing Evidence Manually tested on AWS ECS Fargate with OpenWebUI 0.9.2 + this patch, using MCP tool servers (streamable HTTP). - ✅ Non-streaming: tools executed server-side, final answer includes tool results - ✅ Streaming: SSE stream with content tokens after tool loop completes - ✅ k6 load test (2 VUs, 2 min): **0% errors**, 14/14 requests successful, p95 duration 20.8s ### Contributor License Agreement  - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-18 16:03:41 -05:00

GiteaMirror closed this issue

2026-05-18 16:03:44 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#115105