[PR #15617] [CLOSED] feat: cache tool servers in redis when available #127548

Closed
opened 2026-05-21 09:29:51 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/15617
Author: @signorettif
Created: 7/9/2025
Status: ❌ Closed

Base: dev ← Head: fs-redis-in-memory-tools


📝 Commits (4)

  • 8820ac1 feat: cache tool servers in redis when available
  • 3ec77bf accept initialised values from env variable
  • 91707ef do not return in-memory stuff if it is not current spec
  • 1442424 fix circularity

📊 Changes

6 files changed (+222 additions, -111 deletions)

View changed files

📝 backend/open_webui/functions.py (+1 -1)
📝 backend/open_webui/routers/configs.py (+81 -36)
📝 backend/open_webui/routers/tools.py (+25 -34)
📝 backend/open_webui/utils/middleware.py (+1 -1)
📝 backend/open_webui/utils/tools.py (+22 -39)
➕ backend/open_webui/utils/tools_cache.py (+92 -0)

📄 Description

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions and describe your changes before submitting a pull request.

Before submitting, make sure you've checked the following:

  • Target branch: Please verify that the pull request targets the dev branch.
  • Description: Provide a concise description of the changes made in this pull request.
  • Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
  • Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
  • Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
  • Testing: Have you written and run sufficient tests to validate the changes?
  • Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
  • Prefix: feat: Introduces a new feature or enhancement to the codebase

Changelog Entry

Description

Implements Redis-based caching for Tool Servers to support multi-pod backend deployments. Previously, tool servers were cached in memory per pod, causing inconsistencies when multiple backend pods were running. This enhancement stores tool servers in Redis for shared access across all pods, with fallback to memory cache when Redis is unavailable.

We don't really need to keep the TOOL_SERVERS_CONNECTION in memory. We can derive the information from tool_servers.

Added

  • Redis-based tool server caching system (open_webui/utils/tools_cache.py)
    • get_cached_tool_servers(): Retrieves tool servers from Redis → memory cache
    • set_and_cache_tool_servers(): Immediately fetches and caches tool servers when configuration is set
    • invalidate_tool_servers_cache(): Clears both Redis and memory cache
  • Self-contained tool server data with access_control and __raw properties
  • cahcing invalidation when tool server configuration is updated

Changed

  • Tool server configuration flow now immediately fetches and caches servers instead of storing connections
  • Tool server data structure enhanced with access_control property and __raw original connection data
  • Tools router simplified to only read from cache, eliminating runtime config dependencies
  • Configuration endpoints now retrieve connections from cached tool servers' __raw property
  • Configuration flow change: Tool servers are now cached immediately when configuration is set, rather than on-demand

Removed

  • Dependency on request.app.state.config.TOOL_SERVER_CONNECTIONS for runtime tool server access
  • Import of TOOL_SERVER_CONNECTIONS from main.py
  • Complex conditional logic in cache retrieval functions

Fixed

  • Multi-pod tool server inconsistency - tool servers now shared via Redis cache across all backend pods
  • Tool server updates not propagating between pods in multi-instance deployments

Breaking Changesn/a


Additional Information

Technical Implementation:

  • Redis Key: open-webui:tool_servers with 5-minute TTL
  • Fallback Strategy: Redis → Memory → Empty list (no more on-demand fetching)
  • Data Structure: Tool servers now include access_control and __raw (original connection) properties
  • Multi-pod Safe: All pods share the same Redis cache for consistent tool server state

Migration Path:

  • Existing deployments will automatically migrate to the new caching system
  • Single-pod deployments continue to work with memory fallback
  • Multi-pod deployments now properly share tool server state via Redis

Files Modified:

  • open_webui/utils/tools_cache.py - New Redis caching module
  • open_webui/utils/tools.py - Enhanced tool server data structure
  • open_webui/routers/configs.py - Updated configuration endpoints
  • open_webui/routers/tools.py - Simplified cache-only access + refresh endpoint
  • open_webui/main.py - Removed TOOL_SERVER_CONNECTIONS dependency

Testing Recommendations:

  • Verify tool server functionality with Redis enabled/disabled
  • Test multi-pod deployment tool server consistency
  • Validate cache refresh endpoint functionality
  • Confirm existing tool server configurations continue working

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/15617 **Author:** [@signorettif](https://github.com/signorettif) **Created:** 7/9/2025 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `fs-redis-in-memory-tools` --- ### 📝 Commits (4) - [`8820ac1`](https://github.com/open-webui/open-webui/commit/8820ac163b4cc4f9e810b16d62f41fdb133063f5) feat: cache tool servers in redis when available - [`3ec77bf`](https://github.com/open-webui/open-webui/commit/3ec77bfe0d178a68996882098d477ee2a9422ec2) accept initialised values from env variable - [`91707ef`](https://github.com/open-webui/open-webui/commit/91707efc8b9b0dca5e376469123fd9236db9c4a2) do not return in-memory stuff if it is not current spec - [`1442424`](https://github.com/open-webui/open-webui/commit/1442424bc572245d8db1966c173e38b7d593ef4d) fix circularity ### 📊 Changes **6 files changed** (+222 additions, -111 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/functions.py` (+1 -1) 📝 `backend/open_webui/routers/configs.py` (+81 -36) 📝 `backend/open_webui/routers/tools.py` (+25 -34) 📝 `backend/open_webui/utils/middleware.py` (+1 -1) 📝 `backend/open_webui/utils/tools.py` (+22 -39) ➕ `backend/open_webui/utils/tools_cache.py` (+92 -0) </details> ### 📄 Description # Pull Request Checklist ### Note to first-time contributors: Please open a discussion post in [Discussions](https://github.com/open-webui/open-webui/discussions) and describe your changes before submitting a pull request. **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Please verify that the pull request targets the `dev` branch. - [x] **Description:** Provide a concise description of the changes made in this pull request. - [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [ ] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources? - [x] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation? - [ ] **Testing:** Have you written and run sufficient tests to validate the changes? - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Prefix:** **feat**: Introduces a new feature or enhancement to the codebase # Changelog Entry ### Description Implements Redis-based caching for Tool Servers to support multi-pod backend deployments. Previously, tool servers were cached in memory per pod, causing inconsistencies when multiple backend pods were running. This enhancement stores tool servers in Redis for shared access across all pods, with fallback to memory cache when Redis is unavailable. We don't really need to keep the TOOL_SERVERS_CONNECTION in memory. We can derive the information from `tool_servers`. ### Added - **Redis-based tool server caching system** (`open_webui/utils/tools_cache.py`) - `get_cached_tool_servers()`: Retrieves tool servers from Redis → memory cache - `set_and_cache_tool_servers()`: Immediately fetches and caches tool servers when configuration is set - `invalidate_tool_servers_cache()`: Clears both Redis and memory cache - **Self-contained tool server data** with `access_control` and `__raw` properties - **cahcing invalidation** when tool server configuration is updated ### Changed - **Tool server configuration flow** now immediately fetches and caches servers instead of storing connections - **Tool server data structure** enhanced with `access_control` property and `__raw` original connection data - **Tools router** simplified to only read from cache, eliminating runtime config dependencies - **Configuration endpoints** now retrieve connections from cached tool servers' `__raw` property - **Configuration flow change**: Tool servers are now cached immediately when configuration is set, rather than on-demand ### Removed - **Dependency on `request.app.state.config.TOOL_SERVER_CONNECTIONS`** for runtime tool server access - **Import of `TOOL_SERVER_CONNECTIONS`** from main.py - **Complex conditional logic** in cache retrieval functions ### Fixed - **Multi-pod tool server inconsistency** - tool servers now shared via Redis cache across all backend pods - **Tool server updates not propagating** between pods in multi-instance deployments ### Breaking Changesn/a --- ### Additional Information **Technical Implementation:** - **Redis Key**: `open-webui:tool_servers` with 5-minute TTL - **Fallback Strategy**: Redis → Memory → Empty list (no more on-demand fetching) - **Data Structure**: Tool servers now include `access_control` and `__raw` (original connection) properties - **Multi-pod Safe**: All pods share the same Redis cache for consistent tool server state **Migration Path:** - Existing deployments will automatically migrate to the new caching system - Single-pod deployments continue to work with memory fallback - Multi-pod deployments now properly share tool server state via Redis **Files Modified:** - `open_webui/utils/tools_cache.py` - New Redis caching module - `open_webui/utils/tools.py` - Enhanced tool server data structure - `open_webui/routers/configs.py` - Updated configuration endpoints - `open_webui/routers/tools.py` - Simplified cache-only access + refresh endpoint - `open_webui/main.py` - Removed TOOL_SERVER_CONNECTIONS dependency **Testing Recommendations:** - Verify tool server functionality with Redis enabled/disabled - Test multi-pod deployment tool server consistency - Validate cache refresh endpoint functionality - Confirm existing tool server configurations continue working ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-21 09:29:51 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#127548