[GH-ISSUE #15074] issue: Redis in Cluster Mode - Keys in request don't hash to the same slot #32985

Closed
opened 2026-04-25 06:49:49 -05:00 by GiteaMirror · 7 comments
Owner

Originally created by @taylorwilsdon on GitHub (Jun 17, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/15074

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.15

Ollama Version (if applicable)

No response

Operating System

Ubuntu

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

Configuring redis maintains state for application config and orchestrates tasks and websockets.

Actual Behavior

Image

Steps to Reproduce

Configure Open WebUI with websockets enabled and point at a cluster mode or serverless Redis engine 8 compatible instance:

    {
      name  = "ENABLE_WEBSOCKET_SUPPORT"
      value = "true"
    },
    {
      name  = "REDIS_URL"
      value = "rediss://myinstance.serverless.usw2.cache.amazonaws.com:6379/0"
    },
    {
      name  = "WEBSOCKET_REDIS_URL"
      value = "rediss://myinstance.serverless.usw2.cache.amazonaws.com:6379/0"
    },
    {
      name  = "WEBSOCKET_MANAGER"
      value = "redis"
    }

Logs & Screenshots

Image

Additional Information

There is a relatively simple fix that should just drop in without issue:

just adding {open-webui}: prefix to existing key patterns (ie {open-webui}:config:* instead of open-webui:config:* will allow multi-key operations work for config keys when operating in cluster mode.

        if isinstance(value, PersistentConfig):
            self._state[key] = value
        else:
            self._state[key].value = value
            self._state[key].save()

            if self._redis:
                redis_key = f"open-webui:config:{key}"
                self._redis.set(redis_key, json.dumps(self._state[key].value))

    def __getattr__(self, key):
        if key not in self._state:
            raise AttributeError(f"Config key '{key}' not found")

        # If Redis is available, check for an updated value
        if self._redis:
            redis_key = f"open-webui:config:{key}"
            redis_value = self._redis.get(redis_key)
Originally created by @taylorwilsdon on GitHub (Jun 17, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/15074 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.15 ### Ollama Version (if applicable) _No response_ ### Operating System Ubuntu ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior Configuring redis maintains state for application config and orchestrates tasks and websockets. ### Actual Behavior <img width="958" alt="Image" src="https://github.com/user-attachments/assets/98b86317-b597-4575-8a73-e6f649cc7114" /> ### Steps to Reproduce Configure Open WebUI with websockets enabled and point at a cluster mode or serverless Redis engine 8 compatible instance: ``` { name = "ENABLE_WEBSOCKET_SUPPORT" value = "true" }, { name = "REDIS_URL" value = "rediss://myinstance.serverless.usw2.cache.amazonaws.com:6379/0" }, { name = "WEBSOCKET_REDIS_URL" value = "rediss://myinstance.serverless.usw2.cache.amazonaws.com:6379/0" }, { name = "WEBSOCKET_MANAGER" value = "redis" } ``` ### Logs & Screenshots <img width="956" alt="Image" src="https://github.com/user-attachments/assets/9c08ea84-f882-4b70-a292-d5a99d74a72e" /> ### Additional Information There is a relatively simple fix that should just drop in without issue: just adding `{open-webui}:` prefix to existing key patterns (ie `{open-webui}:config:*` instead of `open-webui:config:*` will allow multi-key operations work for config keys when operating in cluster mode. ``` def __setattr__(self, key, value): if isinstance(value, PersistentConfig): self._state[key] = value else: self._state[key].value = value self._state[key].save() if self._redis: redis_key = f"open-webui:config:{key}" self._redis.set(redis_key, json.dumps(self._state[key].value)) def __getattr__(self, key): if key not in self._state: raise AttributeError(f"Config key '{key}' not found") # If Redis is available, check for an updated value if self._redis: redis_key = f"open-webui:config:{key}" redis_value = self._redis.get(redis_key) ```
GiteaMirror added the bug label 2026-04-25 06:49:49 -05:00
Author
Owner

@taylorwilsdon commented on GitHub (Jun 17, 2025):

Hey @tjbck I can push up a fix but wanted to check with you first which approach you'd prefer:

  • Direct rename (ie just update the fstring in the code there)
  • Configurable as env var (ie REDIS_KEY_PREFIX) so that those who don't care can use as-is and those who need cluster mode can drop in {open-webui}:config:
<!-- gh-comment-id:2980393601 --> @taylorwilsdon commented on GitHub (Jun 17, 2025): Hey @tjbck I can push up a fix but wanted to check with you first which approach you'd prefer: * Direct rename (ie just update the fstring in the code there) * Configurable as env var (ie `REDIS_KEY_PREFIX`) so that those who don't care can use as-is and those who need cluster mode can drop in `{open-webui}:config:`
Author
Owner

@tjbck commented on GitHub (Jun 18, 2025):

REDIS_KEY_PREFIX sounds reasonable, PR welcome!

<!-- gh-comment-id:2983226502 --> @tjbck commented on GitHub (Jun 18, 2025): `REDIS_KEY_PREFIX` sounds reasonable, PR welcome!
Author
Owner

@srdosramos commented on GitHub (Jun 26, 2025):

Similar error happens on a kubernetes with multiple instances of the docker image, the chats throws

Image

Since version 6.1.4, i tracked the error to this code

Image

backend/open_webui/tasks.py

at the beginnig of the file they define the prefix but they are hardcoded no readed from enviroment variables

Image

<!-- gh-comment-id:3010193424 --> @srdosramos commented on GitHub (Jun 26, 2025): Similar error happens on a kubernetes with multiple instances of the docker image, the chats throws ![Image](https://github.com/user-attachments/assets/ca8ac265-71de-448b-99f6-e11076946808) Since version 6.1.4, i tracked the error to this code ![Image](https://github.com/user-attachments/assets/6fd944d2-ebc0-4fd1-96b3-ddf15b72c506) backend/open_webui/tasks.py at the beginnig of the file they define the prefix but they are hardcoded no readed from enviroment variables ![Image](https://github.com/user-attachments/assets/e49b7346-cfce-45c7-aa90-3c4ac5d99654)
Author
Owner

@IgorFZ commented on GitHub (Jun 27, 2025):

Same error here, version 0.6.15:

Image

2025-06-27 11:46:44.826 | ERROR    | asyncio.runners:run:118 - Task exception was never retrieved
future: <Task finished name='Task-2512' coro=<cleanup_task() done, defined at /app/backend/open_webui/tasks.py:79> exception=ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-fccdfd3bf28d) of pipeline caused error: {exception.args}')> - {}
Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1454, in _execute_transaction
    response = await self.parse_response(connection, "_")
                     │    │              └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
                     │    └ <function Pipeline.parse_response at 0x7439f5426700>
                     └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw...
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1537, in parse_response
    result = await super().parse_response(connection, command_name, **options)
                                          │           │               └ {}
                                          │           └ '_'
                                          └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 698, in parse_response
    response = await connection.read_response()
                     │          └ <function AbstractConnection.read_response at 0x7439f555d9e0>
                     └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 622, in read_response
    raise response from None
          └ ExecAbortError('Transaction discarded because of previous errors.')

redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

> File "/app/backend/open_webui/tasks.py", line 84, in cleanup_task
    await redis_cleanup_task(request.app.state.redis, task_id, id)
          │                  │       │                │        └ '67330471-5a62-4ac4-9bc6-6e347ee7bf9f'
          │                  │       │                └ 'bc37dca8-da3c-45cc-90af-fccdfd3bf28d'
          │                  │       └ <property object at 0x7439f48cce00>
          │                  └ <starlette.requests.Request object at 0x7439b457f050>
          └ <function redis_cleanup_task at 0x7439bb7704a0>

  File "/app/backend/open_webui/tasks.py", line 62, in redis_cleanup_task
    if (await pipe.scard(f"{REDIS_CHAT_TASKS_KEY}:{chat_id}").execute())[-1] == 0:
              │    └ <function SetCommands.scard at 0x7439f53fefc0>
              └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw...

  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1597, in execute
    return await conn.retry.call_with_retry(
                 │    │     └ <function Retry.call_with_retry at 0x7439f54ed440>
                 │    └ <redis.asyncio.retry.Retry object at 0x7439b48bdac0>
                 └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/retry.py", line 71, in call_with_retry
    return await do()
                 └ <function Pipeline.execute.<locals>.<lambda> at 0x7439b97aa660>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1457, in _execute_transaction
    raise errors[0][1] from err
          └ [(1, ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-9...
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1447, in _execute_transaction
    await self.parse_response(connection, "_")
          │    │              └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
          │    └ <function Pipeline.parse_response at 0x7439f5426700>
          └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw...
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1537, in parse_response
    result = await super().parse_response(connection, command_name, **options)
                                          │           │               └ {}
                                          │           └ '_'
                                          └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 698, in parse_response
    response = await connection.read_response()
                     │          └ <function AbstractConnection.read_response at 0x7439f555d9e0>
                     └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)>
  File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 622, in read_response
    raise response from None
          └ ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-f...

redis.exceptions.ClusterCrossSlotError: Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-fccdfd3bf28d) of pipeline caused error: {exception.args}
<!-- gh-comment-id:3012797512 --> @IgorFZ commented on GitHub (Jun 27, 2025): Same error here, version 0.6.15: ![Image](https://github.com/user-attachments/assets/c1031a64-f7ad-4782-8f4f-01bb16cd4e81) ```bash 2025-06-27 11:46:44.826 | ERROR | asyncio.runners:run:118 - Task exception was never retrieved future: <Task finished name='Task-2512' coro=<cleanup_task() done, defined at /app/backend/open_webui/tasks.py:79> exception=ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-fccdfd3bf28d) of pipeline caused error: {exception.args}')> - {} Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1454, in _execute_transaction response = await self.parse_response(connection, "_") │ │ └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> │ └ <function Pipeline.parse_response at 0x7439f5426700> └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw... File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1537, in parse_response result = await super().parse_response(connection, command_name, **options) │ │ └ {} │ └ '_' └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 698, in parse_response response = await connection.read_response() │ └ <function AbstractConnection.read_response at 0x7439f555d9e0> └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 622, in read_response raise response from None └ ExecAbortError('Transaction discarded because of previous errors.') redis.exceptions.ExecAbortError: Transaction discarded because of previous errors. The above exception was the direct cause of the following exception: Traceback (most recent call last): > File "/app/backend/open_webui/tasks.py", line 84, in cleanup_task await redis_cleanup_task(request.app.state.redis, task_id, id) │ │ │ │ └ '67330471-5a62-4ac4-9bc6-6e347ee7bf9f' │ │ │ └ 'bc37dca8-da3c-45cc-90af-fccdfd3bf28d' │ │ └ <property object at 0x7439f48cce00> │ └ <starlette.requests.Request object at 0x7439b457f050> └ <function redis_cleanup_task at 0x7439bb7704a0> File "/app/backend/open_webui/tasks.py", line 62, in redis_cleanup_task if (await pipe.scard(f"{REDIS_CHAT_TASKS_KEY}:{chat_id}").execute())[-1] == 0: │ └ <function SetCommands.scard at 0x7439f53fefc0> └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw... File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1597, in execute return await conn.retry.call_with_retry( │ │ └ <function Retry.call_with_retry at 0x7439f54ed440> │ └ <redis.asyncio.retry.Retry object at 0x7439b48bdac0> └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/retry.py", line 71, in call_with_retry return await do() └ <function Pipeline.execute.<locals>.<lambda> at 0x7439b97aa660> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1457, in _execute_transaction raise errors[0][1] from err └ [(1, ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-9... File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1447, in _execute_transaction await self.parse_response(connection, "_") │ │ └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> │ └ <function Pipeline.parse_response at 0x7439f5426700> └ <redis.asyncio.client.Pipeline(<redis.asyncio.connection.ConnectionPool(<redis.asyncio.connection.Connection(host=cache-openw... File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 1537, in parse_response result = await super().parse_response(connection, command_name, **options) │ │ └ {} │ └ '_' └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 698, in parse_response response = await connection.read_response() │ └ <function AbstractConnection.read_response at 0x7439f555d9e0> └ <redis.asyncio.connection.Connection(host=redis_url_removed,port=10000,db=0)> File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 622, in read_response raise response from None └ ClusterCrossSlotError('Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-f... redis.exceptions.ClusterCrossSlotError: Command # 2 (SREM open-webui:tasks:chat:67330471-5a62-4ac4-9bc6-6e347ee7bf9f bc37dca8-da3c-45cc-90af-fccdfd3bf28d) of pipeline caused error: {exception.args} ```
Author
Owner

@taylorwilsdon commented on GitHub (Jun 28, 2025):

ok - PR up @ https://github.com/open-webui/open-webui/pull/15380
@IgorFZ or @srdosramos let me know if this resolves your issues!

<!-- gh-comment-id:3015817513 --> @taylorwilsdon commented on GitHub (Jun 28, 2025): ok - PR up @ https://github.com/open-webui/open-webui/pull/15380 @IgorFZ or @srdosramos let me know if this resolves your issues!
Author
Owner

@davidshen84 commented on GitHub (Jul 18, 2025):

Hi, I upgraded to the latest Docker image v0.6.16, but I still got this error message. Do I need to change any configuration after the upgrade?

<!-- gh-comment-id:3087424182 --> @davidshen84 commented on GitHub (Jul 18, 2025): Hi, I upgraded to the latest Docker image v0.6.16, but I still got this error message. Do I need to change any configuration after the upgrade?
Author
Owner

@taylorwilsdon commented on GitHub (Jul 18, 2025):

Hi, I upgraded to the latest Docker image v0.6.16, but I still got this error message. Do I need to change any configuration after the upgrade?

By default, the PR introduces the ability to change the key - it doesn't actually change it unless you set the env var

Setting env REDIS_KEY_PREFIX to {open-webui} should be your solve if the issue was the same as mine (cluster mode redis)

<!-- gh-comment-id:3089116855 --> @taylorwilsdon commented on GitHub (Jul 18, 2025): > Hi, I upgraded to the latest Docker image v0.6.16, but I still got this error message. Do I need to change any configuration after the upgrade? By default, the PR introduces the ability to change the key - it doesn't actually change it unless you set the env var Setting env REDIS_KEY_PREFIX to `{open-webui}` should be your solve if the issue was the same as mine (cluster mode redis)
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#32985