[GH-ISSUE #19840] issue: Bug: RedisCluster publish Fails on POST /api/tasks/stop/{task_id} #57681

Open
opened 2026-05-05 21:23:23 -05:00 by GiteaMirror · 20 comments
Owner

Originally created by @Salamastik on GitHub (Dec 9, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/19840

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.40

Ollama Version (if applicable)

No response

Operating System

debiangnu/linux 12

Browser (if applicable)

chrome 111.0.5563.65

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

The POST /api/tasks/stop/{task_id} endpoint should reliably stop the task, which requires the control message to reach all potential worker nodes in the cluster.
When using Redis Cluster for Pub/Sub, the message should be broadcast to all master nodes in the cluster to ensure that all consumers (subscribers) receive the command, regardless of which node they are connected to.

Actual Behavior

When Open WebUI is configured to use a Redis Cluster, executing a request to the task-stopping API endpoint results in a fatal AttributeError because the underlying Redis client is a RedisCluster instance which lacks a direct publish method.

The issue is triggered specifically when calling:

POST /api/tasks/stop/{task_id}

This request leads to the following code being executed in /backend/open_webui/tasks.py:

async def redis_send_command(redis: Redis, command: dict):
await redis.publish(REDIS_PUBSUB_CHANNEL, json.dumps(command))

Even if a custom implementation allowed publish on the cluster object, standard Redis Cluster architecture dictates that PUBLISH commands are node-local and do not broadcast across the cluster. Therefore, a single publish call would only reach subscribers on the single, randomly selected node, leading to inconsistent application state across workers/nodes.

Steps to Reproduce

The RedisCluster client object does not implement publish directly; instead, it requires iterating through its nodes and calling publish on each node's client.

The Redis publishing function should be robust against both standard Redis connections and RedisCluster connections. When a RedisCluster is detected, the message must be manually broadcast to every master node in the cluster to ensure all subscribers (workers) receive the command.

Logs & Screenshots

AttributeError: 'RedisCluster' object has no attribute 'publish'

Additional Information

No response

Originally created by @Salamastik on GitHub (Dec 9, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/19840 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.40 ### Ollama Version (if applicable) _No response_ ### Operating System debiangnu/linux 12 ### Browser (if applicable) chrome 111.0.5563.65 ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior The POST /api/tasks/stop/{task_id} endpoint should reliably stop the task, which requires the control message to reach all potential worker nodes in the cluster. When using Redis Cluster for Pub/Sub, the message should be broadcast to all master nodes in the cluster to ensure that all consumers (subscribers) receive the command, regardless of which node they are connected to. ### Actual Behavior When Open WebUI is configured to use a Redis Cluster, executing a request to the task-stopping API endpoint results in a fatal AttributeError because the underlying Redis client is a RedisCluster instance which lacks a direct publish method. The issue is triggered specifically when calling: POST /api/tasks/stop/{task_id} This request leads to the following code being executed in /backend/open_webui/tasks.py: async def redis_send_command(redis: Redis, command: dict): await redis.publish(REDIS_PUBSUB_CHANNEL, json.dumps(command)) Even if a custom implementation allowed publish on the cluster object, standard Redis Cluster architecture dictates that PUBLISH commands are node-local and do not broadcast across the cluster. Therefore, a single publish call would only reach subscribers on the single, randomly selected node, leading to inconsistent application state across workers/nodes. ### Steps to Reproduce The RedisCluster client object does not implement publish directly; instead, it requires iterating through its nodes and calling publish on each node's client. The Redis publishing function should be robust against both standard Redis connections and RedisCluster connections. When a RedisCluster is detected, the message must be manually broadcast to every master node in the cluster to ensure all subscribers (workers) receive the command. ### Logs & Screenshots AttributeError: 'RedisCluster' object has no attribute 'publish' ### Additional Information _No response_
GiteaMirror added the bug label 2026-05-05 21:23:23 -05:00
Author
Owner

@owui-terminator[bot] commented on GitHub (Dec 9, 2025):

🔍 Similar Issues Found

I found some existing issues that might be related to this one. Please check if any of these are duplicates or contain helpful solutions:

  1. #19480 issue: v0.6.40 cannot start anymore with TLS redis configured (ValueError: Unsupported Redis URL schema: rediss)
    by nick-hicx • Nov 25, 2025 • bug

  2. #19496 issue: 500 internal server error appears in v0.6.40
    by cloudtuotuo • Nov 26, 2025 • bug

  3. #19777 issue:
    by Yaute7 • Dec 05, 2025 • bug

  4. #16157 issue: Backend cannot connect to Redis in cluster mode
    by erhhung • Jul 30, 2025 • bug

  5. #16236 issue: Tasks don't work on large context prompts (Renaming chat, etc)
    by frenzybiscuit • Aug 02, 2025 • bug

Show 5 more related issues
  1. #18073 issue: Redis ReadOnlyError with new Open WebUI Version
    by koflerm • Oct 06, 2025 • bug

  2. #19563 issue:
    by naruto7g • Nov 28, 2025 • bug

  3. #14068 issue: database is locked error on a multi-node cluster
    by JTHesse • May 20, 2025 • bug

  4. #15074 issue: Redis in Cluster Mode - Keys in request don't hash to the same slot
    by taylorwilsdon • Jun 17, 2025 • bug

  5. #19211 issue:
    by Byrnes9 • Nov 16, 2025 • bug


💡 Tips:

  • If this is a duplicate, please consider closing this issue and adding any additional details to the existing one
  • If you found a solution in any of these issues, please share it here to help others

This comment was generated automatically by a bot. Please react with a 👍 if this comment was helpful, or a 👎 if it was not.

<!-- gh-comment-id:3631688943 --> @owui-terminator[bot] commented on GitHub (Dec 9, 2025): 🔍 **Similar Issues Found** I found some existing issues that might be related to this one. Please check if any of these are duplicates or contain helpful solutions: 1. [#19480](https://github.com/open-webui/open-webui/issues/19480) **issue: v0.6.40 cannot start anymore with TLS redis configured (ValueError: Unsupported Redis URL schema: rediss)** *by nick-hicx • Nov 25, 2025 • `bug`* 2. [#19496](https://github.com/open-webui/open-webui/issues/19496) **issue: 500 internal server error appears in v0.6.40** *by cloudtuotuo • Nov 26, 2025 • `bug`* 3. [#19777](https://github.com/open-webui/open-webui/issues/19777) **issue:** *by Yaute7 • Dec 05, 2025 • `bug`* 4. [#16157](https://github.com/open-webui/open-webui/issues/16157) **issue: Backend cannot connect to Redis in cluster mode** *by erhhung • Jul 30, 2025 • `bug`* 5. [#16236](https://github.com/open-webui/open-webui/issues/16236) **issue: Tasks don't work on large context prompts (Renaming chat, etc)** *by frenzybiscuit • Aug 02, 2025 • `bug`* <details> <summary>Show 5 more related issues</summary> 6. [#18073](https://github.com/open-webui/open-webui/issues/18073) **issue: Redis ReadOnlyError with new Open WebUI Version** *by koflerm • Oct 06, 2025 • `bug`* 7. [#19563](https://github.com/open-webui/open-webui/issues/19563) **issue:** *by naruto7g • Nov 28, 2025 • `bug`* 8. [#14068](https://github.com/open-webui/open-webui/issues/14068) **issue: database is locked error on a multi-node cluster** *by JTHesse • May 20, 2025 • `bug`* 9. [#15074](https://github.com/open-webui/open-webui/issues/15074) **issue: Redis in Cluster Mode - Keys in request don't hash to the same slot** *by taylorwilsdon • Jun 17, 2025 • `bug`* 10. [#19211](https://github.com/open-webui/open-webui/issues/19211) **issue:** *by Byrnes9 • Nov 16, 2025 • `bug`* </details> --- 💡 **Tips:** - If this is a duplicate, please consider closing this issue and adding any additional details to the existing one - If you found a solution in any of these issues, please share it here to help others *This comment was generated automatically by a bot.* Please react with a 👍 if this comment was helpful, or a 👎 if it was not.
Author
Owner

@Salamastik commented on GitHub (Dec 10, 2025):

I checked the existing issues and they are not related to this specific problem. They describe different scenarios and do not contain a solution.

<!-- gh-comment-id:3636044092 --> @Salamastik commented on GitHub (Dec 10, 2025): I checked the existing issues and they are not related to this specific problem. They describe different scenarios and do not contain a solution.
Author
Owner

@Classic298 commented on GitHub (Dec 10, 2025):

@Ithanil do you know anything here?

<!-- gh-comment-id:3636164353 --> @Classic298 commented on GitHub (Dec 10, 2025): @Ithanil do you know anything here?
Author
Owner

@Ithanil commented on GitHub (Dec 10, 2025):

@Ithanil do you know anything here?

No, I have no experience with a Redis cluster setup.

But I think the path to a solution is outlined in the issue:

When a RedisCluster is detected, the message must be manually broadcast to every master node in the cluster to ensure all subscribers (workers) receive the command.

<!-- gh-comment-id:3636183943 --> @Ithanil commented on GitHub (Dec 10, 2025): > [@Ithanil](https://github.com/Ithanil) do you know anything here? No, I have no experience with a Redis cluster setup. But I think the path to a solution is outlined in the issue: >When a RedisCluster is detected, the message must be manually broadcast to every master node in the cluster to ensure all subscribers (workers) receive the command.
Author
Owner

@rgaricano commented on GitHub (Dec 10, 2025):

@Salamastik,
Could you try with this fix: 98f5f683a6 ?

<!-- gh-comment-id:3636593567 --> @rgaricano commented on GitHub (Dec 10, 2025): @Salamastik, Could you try with this fix: https://github.com/rgaricano/open-webui/commit/98f5f683a60c9a4240f0c3bdc53177c2dd7da137 ?
Author
Owner

@Salamastik commented on GitHub (Dec 10, 2025):

I tried the suggested fix: rgaricano@98f5f68

Unfortunately, it failed with the following error:

File "/app/backendopen_webui/tasks.py", line 80, in redis_send_command
for node in redis.get_master_nodes():

AttributeError: 'RedisCluster' object has no attribute 'get_master_nodes'

<!-- gh-comment-id:3637213442 --> @Salamastik commented on GitHub (Dec 10, 2025): I tried the suggested fix: rgaricano@98f5f68 Unfortunately, it failed with the following error: File "/app/backendopen_webui/tasks.py", line 80, in redis_send_command for node in redis.get_master_nodes(): AttributeError: 'RedisCluster' object has no attribute 'get_master_nodes'
Author
Owner

@rgaricano commented on GitHub (Dec 10, 2025):

@Salamastik,
Sorry, for the mistake. I revised and it's easiars as I did, just doing a share publising of the message
(https://redis.io/docs/latest/commands/spublish/ )

But also I can't test it now (not using cluster)

async def redis_send_command(redis: Redis, command: dict):  
    command_json = json.dumps(command)  
      
    # Check if it's a RedisCluster  
    if hasattr(redis, 'nodes_manager'):  
        # Use sharded pub/sub for cluster mode  
        await redis.spublish(REDIS_PUBSUB_CHANNEL, command_json)  
    else:  
        # Standard Redis connection  
        await redis.publish(REDIS_PUBSUB_CHANNEL, command_json)

(I think that in the referenced commit I lost the last parenthesis )

<!-- gh-comment-id:3638414306 --> @rgaricano commented on GitHub (Dec 10, 2025): @Salamastik, Sorry, for the mistake. I revised and it's easiars as I did, just doing a share publising of the message (https://redis.io/docs/latest/commands/spublish/ ) But also I can't test it now (not using cluster) ``` async def redis_send_command(redis: Redis, command: dict): command_json = json.dumps(command) # Check if it's a RedisCluster if hasattr(redis, 'nodes_manager'): # Use sharded pub/sub for cluster mode await redis.spublish(REDIS_PUBSUB_CHANNEL, command_json) else: # Standard Redis connection await redis.publish(REDIS_PUBSUB_CHANNEL, command_json) ``` (I think that in the referenced commit I lost the last parenthesis )
Author
Owner

@Salamastik commented on GitHub (Dec 11, 2025):

@rgaricano

Thank you for the suggestion!

When attempting to execute the suggested code, it failed with the following error:

AttributeError: 'RedisCluster' object has no attribute 'spublish'

1. Environment Details

  • Redis Client Library: redis (version 7.1.0)
  • Version: 7.1.0
  • Client Type: Connected via redis.cluster.RedisCluster.from_url(...)
  • Problem: The RedisCluster object does not seem to expose the spublish method directly in this version of the client library.

2. Workarounds Attempted and Results

I subsequently tried two different common methods for publishing messages via a RedisCluster instance, which led to mixed results:

  1. redis.pubsub_publish(REDIS_PUBSUB_CHANNEL, command_json)

    Failed: AttributeError: 'RedisCluster' object has no attribute 'pubsub_publish'

  2. redis.execute_command('PUBLISH', REDIS_PUBSUB_CHANNEL, command_json)

    This command executed successfully without raising an exception. However, the intended functionality (the task was not stopping) is still not working as intended.

<!-- gh-comment-id:3641201891 --> @Salamastik commented on GitHub (Dec 11, 2025): @rgaricano Thank you for the suggestion! When attempting to execute the suggested code, it failed with the following error: AttributeError: 'RedisCluster' object has no attribute 'spublish' ### 1. Environment Details * **Redis Client Library:** `redis` (version 7.1.0) * **Version:** 7.1.0 * **Client Type:** Connected via `redis.cluster.RedisCluster.from_url(...)` * **Problem:** The `RedisCluster` object does not seem to expose the `spublish` method directly in this version of the client library. ### 2. Workarounds Attempted and Results I subsequently tried two different common methods for publishing messages via a `RedisCluster` instance, which led to mixed results: 1. redis.pubsub_publish(REDIS_PUBSUB_CHANNEL, command_json) Failed: AttributeError: 'RedisCluster' object has no attribute 'pubsub_publish' 2. redis.execute_command('PUBLISH', REDIS_PUBSUB_CHANNEL, command_json) This command executed successfully without raising an exception. However, the intended functionality (the task was not stopping) is still not working as intended.
Author
Owner

@koflerm commented on GitHub (Dec 22, 2025):

+1 for this issue. We see the same behavior with our setup, as well using Redis Cluster.

<!-- gh-comment-id:3682401635 --> @koflerm commented on GitHub (Dec 22, 2025): +1 for this issue. We see the same behavior with our setup, as well using Redis Cluster.
Author
Owner

@rgaricano commented on GitHub (Dec 22, 2025):

@Salamastik,

yes, the redis-py library's RedisCluster object doesn't have an spublish method. In Redis cluster mode, sharded pub/sub commands (SPUBLISH, SSUBSCRIBE) are server-side commands, but the Python client library doesn't expose them as methods on the cluster client object.
This is why you get the AttributeError.
The cluster client requires using execute_command for commands that aren't directly exposed as methods.

workaround:

async def redis_send_command(redis: Redis, command: dict):    
    command_json = json.dumps(command)    
        
    # Check if it's a RedisCluster    
    if hasattr(redis, 'nodes_manager'):    
        # Use execute_command for sharded pub/sub in cluster mode  
        await redis.execute_command('SPUBLISH', REDIS_PUBSUB_CHANNEL, command_json)    
    else:    
        # Standard Redis connection    
        await redis.publish(REDIS_PUBSUB_CHANNEL, command_json)
<!-- gh-comment-id:3682740022 --> @rgaricano commented on GitHub (Dec 22, 2025): @Salamastik, yes, the redis-py library's RedisCluster object doesn't have an spublish method. In Redis cluster mode, sharded pub/sub commands (SPUBLISH, SSUBSCRIBE) are server-side commands, but the Python client library doesn't expose them as methods on the cluster client object. This is why you get the AttributeError. The cluster client requires using execute_command for commands that aren't directly exposed as methods. workaround: ``` async def redis_send_command(redis: Redis, command: dict): command_json = json.dumps(command) # Check if it's a RedisCluster if hasattr(redis, 'nodes_manager'): # Use execute_command for sharded pub/sub in cluster mode await redis.execute_command('SPUBLISH', REDIS_PUBSUB_CHANNEL, command_json) else: # Standard Redis connection await redis.publish(REDIS_PUBSUB_CHANNEL, command_json) ```
Author
Owner

@Salamastik commented on GitHub (Dec 29, 2025):

@rgaricano
Thanks for the comment! I tried it, but it behaves the same as: redis.execute_command('PUBLISH', REDIS_PUBSUB_CHANNEL, command_json)

The code executes successfully without any exceptions, but the task is still not stopping. It’s still not working as intended.
@koflerm, maybe you can try it too and see if you get the same result?

<!-- gh-comment-id:3696788831 --> @Salamastik commented on GitHub (Dec 29, 2025): @rgaricano Thanks for the comment! I tried it, but it behaves the same as: redis.execute_command('PUBLISH', REDIS_PUBSUB_CHANNEL, command_json) The code executes successfully without any exceptions, but the task is still not stopping. It’s still not working as intended. @koflerm, maybe you can try it too and see if you get the same result?
Author
Owner

@Classic298 commented on GitHub (Dec 29, 2025):

@Salamastik
please test this

https://github.com/open-webui/open-webui/pull/20246

<!-- gh-comment-id:3697584464 --> @Classic298 commented on GitHub (Dec 29, 2025): @Salamastik please test this https://github.com/open-webui/open-webui/pull/20246
Author
Owner

@Salamastik commented on GitHub (Dec 31, 2025):

@Classic298
Thanks for the suggestion! Unfortunately, it doesn't seem to work. There are no error messages in the console, but the response never stops.

<!-- gh-comment-id:3701785587 --> @Salamastik commented on GitHub (Dec 31, 2025): @Classic298 Thanks for the suggestion! Unfortunately, it doesn't seem to work. There are no error messages in the console, but the response never stops.
Author
Owner

@Classic298 commented on GitHub (Dec 31, 2025):

@Salamastik can you give it another shot please?

<!-- gh-comment-id:3702031654 --> @Classic298 commented on GitHub (Dec 31, 2025): @Salamastik can you give it another shot please?
Author
Owner

@Salamastik commented on GitHub (Jan 4, 2026):

@Classic298
It is still not working, and there are no errors in the console...

<!-- gh-comment-id:3708102456 --> @Salamastik commented on GitHub (Jan 4, 2026): @Classic298 It is still not working, and there are no errors in the console...
Author
Owner

@Salamastik commented on GitHub (Jan 21, 2026):

Confirmed! This is working perfectly now. Thanks @HANIHALILI for the fix, much appreciated! 🙌

<!-- gh-comment-id:3776468879 --> @Salamastik commented on GitHub (Jan 21, 2026): Confirmed! This is working perfectly now. Thanks @HANIHALILI for the fix, much appreciated! 🙌
Author
Owner

@tjbck commented on GitHub (Feb 12, 2026):

@Salamastik could you confirm if this issue has been addressed in the latest dev?

<!-- gh-comment-id:3893482368 --> @tjbck commented on GitHub (Feb 12, 2026): @Salamastik could you confirm if this issue has been addressed in the latest dev?
Author
Owner

@Salamastik commented on GitHub (Feb 13, 2026):

Hi @tjbck,

I tested this exact approach (redis.execute_command('PUBLISH', ...)) about a month ago. As mentioned in the issue, while the command executes without errors, it does not reliably stop tasks in a RedisCluster setup — the cancellation message does not consistently propagate across nodes.

In my testing, the state-based approach introduced in this PR by @HANIHALILI behaves consistently in a clustered environment and successfully stops tasks across nodes.

If there were additional changes beyond this specific modification, I’d be happy to review them in case they impact the behavior. However, based on my current testing, this particular change alone does not resolve the issue

<!-- gh-comment-id:3896758805 --> @Salamastik commented on GitHub (Feb 13, 2026): Hi @tjbck, I tested this exact approach (redis.execute_command('PUBLISH', ...)) about a month ago. As mentioned in the issue, while the command executes without errors, it does not reliably stop tasks in a RedisCluster setup — the cancellation message does not consistently propagate across nodes. In my testing, the state-based approach introduced in this PR by @HANIHALILI behaves consistently in a clustered environment and successfully stops tasks across nodes. If there were additional changes beyond this specific modification, I’d be happy to review them in case they impact the behavior. However, based on my current testing, this particular change alone does not resolve the issue
Author
Owner

@luke-wren commented on GitHub (Mar 6, 2026):

I'm experiencing the same issues on my instance. I can confirm the fix in https://github.com/open-webui/open-webui/pull/20803 resolves the problem.

<!-- gh-comment-id:4010847118 --> @luke-wren commented on GitHub (Mar 6, 2026): I'm experiencing the same issues on my instance. I can confirm the fix in https://github.com/open-webui/open-webui/pull/20803 resolves the problem.
Author
Owner

@nwon910 commented on GitHub (Apr 21, 2026):

I'm experiencing the same issues, using Redis Cluster as well, on the most current version of v0.9.1

<!-- gh-comment-id:4291372768 --> @nwon910 commented on GitHub (Apr 21, 2026): I'm experiencing the same issues, using Redis Cluster as well, on the most current version of v0.9.1
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#57681