[GH-ISSUE #14340] run in K8S with multiple PODs, can not stop response #120527

New Issue

GiteaMirror · 2026-05-20T22:08:41-05:00

GiteaMirror commented

2026-05-20 22:08:41 -05:00

Originally created by @heifade on GitHub (May 26, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/14340

Originally assigned to: @tjbck on GitHub.

Check Existing Issues

I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.

Installation Method

Pip Install

Open WebUI Version

0.6.10

Ollama Version (if applicable)

No response

Operating System

K8S

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have listed steps to reproduce the bug in detail.

Expected Behavior

run in K8S with multiple PODs
ask one question
wait for one moment
click stop
Error message： Task ID with XXX not fond and Output has not stopped

Actual Behavior

run in K8S with multiple PODs
ask one question
wait for one moment
click stop
Error message： Task ID with XXX not fond and Output has not stopped

Steps to Reproduce

run in K8S with multiple PODs
ask one question
wait for one moment
click stop
Error message： Task ID with XXX not fond and Output has not stopped

Logs & Screenshots

None

Additional Information

None

Originally created by @heifade on GitHub (May 26, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/14340 Originally assigned to: @tjbck on GitHub. ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Pip Install ### Open WebUI Version 0.6.10 ### Ollama Version (if applicable) _No response_ ### Operating System K8S ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior 1. run in K8S with multiple PODs 2. ask one question 3. wait for one moment 4. click stop Error message： Task ID with XXX not fond and Output has not stopped ### Actual Behavior 1. run in K8S with multiple PODs 2. ask one question 3. wait for one moment 4. click stop Error message： Task ID with XXX not fond and Output has not stopped ### Steps to Reproduce 1. run in K8S with multiple PODs 2. ask one question 3. wait for one moment 4. click stop Error message： Task ID with XXX not fond and Output has not stopped ### Logs & Screenshots None ### Additional Information None

GiteaMirror added the bug label 2026-05-20 22:08:41 -05:00

GiteaMirror closed this issue

2026-05-20 22:08:42 -05:00

GiteaMirror commented

2026-05-20 22:08:43 -05:00

@Classic298 commented on GitHub (May 26, 2025):

you run k8s with a pip installation? How? Why? Especially why?

And did you configure all the env variables necessary for parallel operations?

@Classic298 commented on GitHub (May 26, 2025): you run k8s with a pip installation? How? Why? Especially why? And did you configure all the env variables necessary for parallel operations?

GiteaMirror commented

2026-05-20 22:08:44 -05:00

@heifade commented on GitHub (May 26, 2025):

I feel that when there are multiple PODs, the reason why the task cannot be found in the pod called by the API. Because tasks are stored in memory

When there is only one pod on K8S, this problem will not occur

@app.post("/api/tasks/stop/{task_id}")
async def stop_task_endpoint(task_id: str, user=Depends(get_verified_user)):
try:
result = await stop_task(task_id)
return result
except ValueError as e:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(e))

@heifade commented on GitHub (May 26, 2025): I feel that when there are multiple PODs, the reason why the task cannot be found in the pod called by the API. Because tasks are stored in memory When there is only one pod on K8S, this problem will not occur @app.post("/api/tasks/stop/{task_id}") async def stop_task_endpoint(task_id: str, user=Depends(get_verified_user)): try: result = await stop_task(task_id) return result except ValueError as e: raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(e))

GiteaMirror commented

2026-05-20 22:08:44 -05:00

@Ithanil commented on GitHub (May 26, 2025):

you run k8s with a pip installation? How? Why? Especially why?

And did you configure all the env variables necessary for parallel operations?

So you don't have that problem with a multi-replica setup? Because I can confirm this, it's an old problem that I somehow failed to report yet.

@Ithanil commented on GitHub (May 26, 2025): > you run k8s with a pip installation? How? Why? Especially why? > > And did you configure all the env variables necessary for parallel operations? So you don't have that problem with a multi-replica setup? Because I can confirm this, it's an old problem that I somehow failed to report yet.

GiteaMirror commented

2026-05-20 22:08:44 -05:00

@tjbck commented on GitHub (Jun 8, 2025):

Should be addressed in dev!

@tjbck commented on GitHub (Jun 8, 2025): Should be addressed in dev!

GiteaMirror commented

2026-05-20 22:08:45 -05:00

@tjbck commented on GitHub (Jun 8, 2025):

@jackthgu testing wanted here

@tjbck commented on GitHub (Jun 8, 2025): @jackthgu testing wanted here

GiteaMirror commented

2026-05-20 22:08:45 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

@tjbck Although there are no errors, it still cannot be stopped; I am using docker swarm

@EntropyYue commented on GitHub (Jun 9, 2025): @tjbck Although there are no errors, it still cannot be stopped; I am using docker swarm

GiteaMirror commented

2026-05-20 22:08:46 -05:00

@mikertillman commented on GitHub (Jun 9, 2025):

Also getting the same behavior in Windows 11 > WSL2 > Docker Desktop > OpenWebUI + Ollama containers.
I also had to manually stop and start the containers and restart them.

@mikertillman commented on GitHub (Jun 9, 2025): Also getting the same behavior in Windows 11 > WSL2 > Docker Desktop > OpenWebUI + Ollama containers. I also had to manually stop and start the containers and restart them.

GiteaMirror commented

2026-05-20 22:08:46 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

I can see the correct logs on the backend, but the messages are still being sent

@EntropyYue commented on GitHub (Jun 9, 2025): I can see the correct logs on the backend, but the messages are still being sent ![Image](https://github.com/user-attachments/assets/9149fa1f-e2e6-4d7d-82b8-2e93f2826908)

GiteaMirror commented

2026-05-20 22:08:47 -05:00

@tjbck commented on GitHub (Jun 9, 2025):

@EntropyYue did you configure REDIS_URL as well?

@tjbck commented on GitHub (Jun 9, 2025): @EntropyYue did you configure REDIS_URL as well?

GiteaMirror commented

2026-05-20 22:08:47 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

I configured REDIS_URL

- REDIS_URL=redis://redis:6379/0
- ENABLE_WEBSOCKET_SUPPORT=true
- WEBSOCKET_MANAGER=redis
- WEBSOCKET_REDIS_URL=redis://redis:6379/0

redis:
  image: redis:7.2
  networks:
    - open-webui
  restart: always

The above is my docker compose

@EntropyYue commented on GitHub (Jun 9, 2025): I configured REDIS_URL ``` - REDIS_URL=redis://redis:6379/0 - ENABLE_WEBSOCKET_SUPPORT=true - WEBSOCKET_MANAGER=redis - WEBSOCKET_REDIS_URL=redis://redis:6379/0 ``` ``` redis: image: redis:7.2 networks: - open-webui restart: always ``` The above is my docker compose

GiteaMirror commented

2026-05-20 22:08:47 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

I can see that redis is working

@EntropyYue commented on GitHub (Jun 9, 2025): I can see that redis is working ![Image](https://github.com/user-attachments/assets/e179e740-ab5f-4b5c-84a4-eb72ae2d1c0b)

GiteaMirror commented

2026-05-20 22:08:48 -05:00

@tjbck commented on GitHub (Jun 9, 2025):

Should be fixed with db3c26ab7a

@tjbck commented on GitHub (Jun 9, 2025): Should be fixed with db3c26ab7a3cbe30109a6397ef1ee851e7999ebb

GiteaMirror commented

2026-05-20 22:08:48 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

Encountering task not found again

@EntropyYue commented on GitHub (Jun 9, 2025): Encountering task not found again ![Image](https://github.com/user-attachments/assets/2e350066-fbb4-4dd4-96dc-0986820a5184)

GiteaMirror commented

2026-05-20 22:08:49 -05:00

@tjbck commented on GitHub (Jun 9, 2025):

@EntropyYue is your redis configured correctly?

@tjbck commented on GitHub (Jun 9, 2025): @EntropyYue is your redis configured correctly?

GiteaMirror commented

2026-05-20 22:08:50 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

I configured REDIS_URL

- REDIS_URL=redis://redis:6379/0
- ENABLE_WEBSOCKET_SUPPORT=true
- WEBSOCKET_MANAGER=redis
- WEBSOCKET_REDIS_URL=redis://redis:6379/0

redis:
  image: redis:7.2
  networks:
    - open-webui
  restart: always

The above is my docker compose

It can work, it looks like the configuration is fine

@EntropyYue commented on GitHub (Jun 9, 2025): > I configured REDIS_URL > > ``` > - REDIS_URL=redis://redis:6379/0 > - ENABLE_WEBSOCKET_SUPPORT=true > - WEBSOCKET_MANAGER=redis > - WEBSOCKET_REDIS_URL=redis://redis:6379/0 > ``` > > ``` > redis: > image: redis:7.2 > networks: > - open-webui > restart: always > ``` > > The above is my docker compose It can work, it looks like the configuration is fine ![Image](https://github.com/user-attachments/assets/86f979cd-7cea-4f23-a062-479001c777f7)

GiteaMirror commented

2026-05-20 22:08:50 -05:00

@tjbck commented on GitHub (Jun 9, 2025):

@EntropyYue

async def stop_task(request, task_id: str):
    """
    Cancel a running task and remove it from the global task list.
    """
    if is_redis(request):
        # PUBSUB: All instances check if they have this task, and stop if so.
        redis_send_command(
            request.app.state.redis,
            {
                "action": "stop",
                "task_id": task_id,
            },
        )
        # Optionally check if task_id still in Redis a few moments later for feedback?
        return {"status": True, "message": f"Stop signal sent for {task_id}"}

    task = tasks.get(task_id)
    if not task:
        raise ValueError(f"Task with ID {task_id} not found.")

    task.cancel()  # Request task cancellation
    try:
        await task  # Wait for the task to handle the cancellation
    except asyncio.CancelledError:
        # Task successfully canceled
        tasks.pop(task_id, None)  # Remove it from the dictionary
        return {"status": True, "message": f"Task {task_id} successfully stopped."}

    return {"status": False, "message": f"Failed to stop task {task_id}."}

is the relevant code, you'd only get 404 response if redis is not detected, are you sure you're on the lastest dev?

@tjbck commented on GitHub (Jun 9, 2025): @EntropyYue ```py async def stop_task(request, task_id: str): """ Cancel a running task and remove it from the global task list. """ if is_redis(request): # PUBSUB: All instances check if they have this task, and stop if so. redis_send_command( request.app.state.redis, { "action": "stop", "task_id": task_id, }, ) # Optionally check if task_id still in Redis a few moments later for feedback? return {"status": True, "message": f"Stop signal sent for {task_id}"} task = tasks.get(task_id) if not task: raise ValueError(f"Task with ID {task_id} not found.") task.cancel() # Request task cancellation try: await task # Wait for the task to handle the cancellation except asyncio.CancelledError: # Task successfully canceled tasks.pop(task_id, None) # Remove it from the dictionary return {"status": True, "message": f"Task {task_id} successfully stopped."} return {"status": False, "message": f"Failed to stop task {task_id}."} ``` is the relevant code, you'd only get 404 response if redis is not detected, are you sure you're on the lastest dev?

GiteaMirror commented

2026-05-20 22:08:51 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

I am sure I am on the latest dev
I have enabled debug-level logging, and here are the logs related to Redis

# docker compose logs open-webui | grep Redis
open-webui-1  | DEBUG [open_webui.socket.main] Using Redis to manage websockets.
open-webui-3  | DEBUG [open_webui.socket.main] Using Redis to manage websockets.
open-webui-4  | DEBUG [open_webui.socket.main] Using Redis to manage websockets.
open-webui-2  | DEBUG [open_webui.socket.main] Using Redis to manage websockets.

@EntropyYue commented on GitHub (Jun 9, 2025): I am sure I am on the latest dev I have enabled debug-level logging, and here are the logs related to Redis ``` # docker compose logs open-webui | grep Redis open-webui-1 | DEBUG [open_webui.socket.main] Using Redis to manage websockets. open-webui-3 | DEBUG [open_webui.socket.main] Using Redis to manage websockets. open-webui-4 | DEBUG [open_webui.socket.main] Using Redis to manage websockets. open-webui-2 | DEBUG [open_webui.socket.main] Using Redis to manage websockets. ```

GiteaMirror commented

2026-05-20 22:08:51 -05:00

@Ithanil commented on GitHub (Jun 9, 2025):

I've merged latest dev and for me it seems to work. Using Redis Sentinel setup.

(however I once again hit an issue with usage pool cleanup preventing containers to come up properly that is present since one of the most recent version, will open an issue now)

@Ithanil commented on GitHub (Jun 9, 2025): I've merged latest dev and for me it seems to work. Using Redis Sentinel setup. (however I once again hit an issue with usage pool cleanup preventing containers to come up properly that is present since one of the most recent version, will open an issue now)

GiteaMirror commented

2026-05-20 22:08:52 -05:00

@tjbck commented on GitHub (Jun 9, 2025):

@EntropyYue @Ithanil Should actually be fixed with ea8dc333ee now, testing wanted here!

@tjbck commented on GitHub (Jun 9, 2025): @EntropyYue @Ithanil Should actually be fixed with ea8dc333ee808719d3756015808986531fb9e562 now, testing wanted here!

GiteaMirror commented

2026-05-20 22:08:52 -05:00

@Ithanil commented on GitHub (Jun 9, 2025):

Still works at at ea8dc333ee , but already did with db3c26ab7a for me.

@Ithanil commented on GitHub (Jun 9, 2025): Still works at at ea8dc333ee808719d3756015808986531fb9e562 , but already did with https://github.com/open-webui/open-webui/commit/db3c26ab7a3cbe30109a6397ef1ee851e7999ebb for me.

GiteaMirror commented

2026-05-20 22:08:53 -05:00

@EntropyYue commented on GitHub (Jun 9, 2025):

It work

@EntropyYue commented on GitHub (Jun 9, 2025): It work

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#120527