[GH-ISSUE #22734] issue: Intermittent model lookup failures due to RedisDict.set() race condition #123112

New Issue

GiteaMirror · 2026-05-21T02:19:13-05:00

GiteaMirror commented

2026-05-21 02:19:13 -05:00

Originally created by @m1g32 on GitHub (Mar 16, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/22734

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.8.8

Ollama Version (if applicable)

No response

Operating System

Kubernetes 1.32

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

• The selected model should always be found and consistently usable for all tasks.
• Both main response generation and follow-up tasks (follow-up question and chat title generation) should reliably use the same model without intermittent lookup failures.

Actual Behavior

• Intermittent “model not found” errors occur, even though the model does exist and is accessible at other times.
• Two failure patterns appear:
– During main response generation, the request fails entirely with “model not found”.
– During follow-up tasks (generate_follow_ups), the main response is produced successfully using the model, but follow-up question and/or chat title generation fails with “model not found” for the same model.
• The problem occurs randomly and is not tied to any specific model.
• The issue appears only in deployments under heavy load with many concurrent users, where RedisDict.set is called very frequently, and model-list refreshes become extremely common.

Steps to Reproduce

Deploy Open WebUI on Kubernetes with multiple replicas.
Deploy a Redis Cluster and configure Open WebUI accordingly.
Add several base models, then add a large number of custom models (hundreds).
Optionally set MODELS_CACHE_TTL to a non-zero value such as 300 seconds.
Leave “Cache Base Model List” enabled or disabled (the issue occurs either way).
Do not configure any dedicated local or external task model.
Under intense concurrent usage, generate responses using any available model.
Observe that intermittently:
– The main response fails with “model not found”, or
– The main response succeeds, but follow-up generation (follow-up question or chat title) fails with “model not found” for the same model.
Repeat several times; the failures occur unpredictably based on timing conditions.

Logs & Screenshots

| ERROR | open_webui.routers.tasks:generate_follow_ups:328 - Exception occurred
Traceback (most recent call last):

File "", line 1, in
File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
│ │ └ 4
│ └ 21
└ <function _main at 0x7febd5f1e160>
File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 135, in _main
return self._bootstrap(parent_sentinel)
│ │ └ 4
│ └ <function BaseProcess._bootstrap at 0x7febd6287740>
└
File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
│ └ <function BaseProcess.run at 0x7febd6286ca0>
└
File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ │ └ {'config': <uvicorn.config.Config object at 0x7febd5f1ad50>, 'target': <bound method Process.target of <uvicorn.supervisors.m...
│ │ │ │ └
│ │ │ └ ()
│ │ └
│ └ <function subprocess_started at 0x7febd542d440>
└
File "/usr/local/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
target(sockets=sockets)
│ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>]
└ <bound method Process.target of <uvicorn.supervisors.multiprocess.Process object at 0x7febd5359c50>>
File "/usr/local/lib/python3.11/site-packages/uvicorn/supervisors/multiprocess.py", line 64, in target
return self.real_target(sockets)
│ │ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>]
│ └ <bound method Server.run of <uvicorn.server.Server object at 0x7febd5359d90>>
└ <uvicorn.supervisors.multiprocess.Process object at 0x7febd5359c50>
File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 67, in run
return asyncio_run(self.serve(sockets=sockets), loop_factory=self.config.get_loop_factory())
│ │ │ │ │ │ └ <function Config.get_loop_factory at 0x7febd55bc400>
│ │ │ │ │ └ <uvicorn.config.Config object at 0x7febd5f1ad50>
│ │ │ │ └ <uvicorn.server.Server object at 0x7febd5359d90>
│ │ │ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>]
│ │ └ <function Server.serve at 0x7febd542c220>
│ └ <uvicorn.server.Server object at 0x7febd5359d90>
└ <function asyncio_run at 0x7febd5568a40>
File "/usr/local/lib/python3.11/site-packages/uvicorn/_compat.py", line 30, in asyncio_run
return runner.run(main)
│ │ └ <coroutine object Server.serve at 0x7febd52b7b50>
│ └ <function Runner.run at 0x7febd57d94e0>
└ <asyncio.runners.Runner object at 0x7febd5359910>
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
│ │ │ └ <Task pending name='Task-1' coro=<Server.serve() running at /usr/local/lib/python3.11/site-packages/uvicorn/server.py:71> wai...
│ │ └ <cyfunction Loop.run_until_complete at 0x7febd507a980>
│ └ <uvloop.Loop running=True closed=False debug=False>
└ <asyncio.runners.Runner object at 0x7febd5359910>

File "/app/backend/open_webui/main.py", line 1868, in process_chat
return await process_chat_response(response, ctx)
│ │ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message...
│ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50>
└ <function process_chat_response at 0x7fea4723a840>

File "/app/backend/open_webui/utils/middleware.py", line 4730, in process_chat_response
return await streaming_chat_response_handler(response, ctx)
│ │ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message...
│ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50>
└ <function streaming_chat_response_handler at 0x7fea4723a7a0>

File "/app/backend/open_webui/utils/middleware.py", line 4678, in streaming_chat_response_handler
return await response_handler(response, events)
│ │ └ []
│ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50>
└ <function streaming_chat_response_handler..response_handler at 0x7fea3e812d40>

File "/app/backend/open_webui/utils/middleware.py", line 4659, in response_handler
await background_tasks_handler(ctx)
│ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message...
└ <function background_tasks_handler at 0x7fea4723a660>

File "/app/backend/open_webui/utils/middleware.py", line 2836, in background_tasks_handler
res = await generate_follow_ups(
└ <function generate_follow_ups at 0x7fea4972db20>

File "/app/backend/open_webui/routers/tasks.py", line 326, in generate_follow_ups
return await generate_chat_completion(request, form_data=payload, user=user)
│ │ │ └ UserModel(id='', email='', username=None, role='user', ...
│ │ └ {'model': 'gpt-5.2', 'messages': [{'role': 'user', 'content': '### Task:\nSuggest 3-5 relevant follow-up questions or prompts...
│ └ <starlette.requests.Request object at 0x7fea3e9ba710>
└ <function generate_chat_completion at 0x7fea4972d120>

File "/app/backend/open_webui/utils/chat.py", line 188, in generate_chat_completion
raise Exception("Model not found")

Exception: Model not found

Additional Information

Root Cause Analysis:
This issue is caused by a race condition in RedisDict.set at backend/open_webui/socket/utils.py. This is a severe problem that becomes visible only in deployments with high load, many replicas, and a large number of activity and models.

Any time a pod refreshes the model list (via /api/models, admin changes, get_all_models, or cache refresh), it executes request.app.state.MODELS.set(models_dict).
RedisDict.set uses a Redis pipeline:

DELETE the entire models hash
HSET the new model mapping

Redis pipelines are not atomic. There is a moment between DELETE and HSET where the hash does not exist.
During this small window, any pod checking for a model (such as in generate_follow_ups or generate_chat_completion) may see the hash as empty, causing “model not found”.

I am not a Redis expert, so the suggested fix courtesy of Claude should absolutely be taken with a grain of salt and could be completely useless. The exact implementation of the fix should be reviewed and validated by someone with actual Redis expertise:

def set(self, mapping: dict):
    # Fetch existing field names only (not values) to compute stale keys.
    # HKEYS is cheap — it transfers only short model ID strings, not the large JSON values.
    existing_keys = set(self.redis.hkeys(self.name))
    new_keys = set(mapping.keys())
    keys_to_remove = existing_keys - new_keys

    # HSET first (add/update all new values), then HDEL (remove stale keys).
    # Critically, we never DELETE the whole hash — this eliminates the race window
    # where concurrent readers would see an empty models dict between a DELETE and HSET.
    pipe = self.redis.pipeline()

    if mapping:
        pipe.hset(self.name, mapping={k: json.dumps(v) for k, v in mapping.items()})

    if keys_to_remove:
        pipe.hdel(self.name, *keys_to_remove)

    pipe.execute()

Originally created by @m1g32 on GitHub (Mar 16, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/22734 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.8.8 ### Ollama Version (if applicable) _No response_ ### Operating System Kubernetes 1.32 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior • The selected model should always be found and consistently usable for all tasks. • Both main response generation and follow-up tasks (follow-up question and chat title generation) should reliably use the same model without intermittent lookup failures. ### Actual Behavior • Intermittent “model not found” errors occur, even though the model does exist and is accessible at other times. • Two failure patterns appear: – During main response generation, the request fails entirely with “model not found”. – During follow-up tasks (generate_follow_ups), the main response is produced successfully using the model, but follow-up question and/or chat title generation fails with “model not found” for the same model. • The problem occurs randomly and is not tied to any specific model. • The issue appears only in deployments under heavy load with many concurrent users, where RedisDict.set is called very frequently, and model-list refreshes become extremely common. ### Steps to Reproduce Deploy Open WebUI on Kubernetes with multiple replicas. Deploy a Redis Cluster and configure Open WebUI accordingly. Add several base models, then add a large number of custom models (hundreds). Optionally set MODELS_CACHE_TTL to a non-zero value such as 300 seconds. Leave “Cache Base Model List” enabled or disabled (the issue occurs either way). Do not configure any dedicated local or external task model. Under intense concurrent usage, generate responses using any available model. Observe that intermittently: – The main response fails with “model not found”, or – The main response succeeds, but follow-up generation (follow-up question or chat title) fails with “model not found” for the same model. Repeat several times; the failures occur unpredictably based on timing conditions. ### Logs & Screenshots | ERROR | open_webui.routers.tasks:generate_follow_ups:328 - Exception occurred Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main exitcode = _main(fd, parent_sentinel) │ │ └ 4 │ └ 21 └ <function _main at 0x7febd5f1e160> File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 135, in _main return self._bootstrap(parent_sentinel) │ │ └ 4 │ └ <function BaseProcess._bootstrap at 0x7febd6287740> └ <SpawnProcess name='SpawnProcess-4' parent=1 started> File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() │ └ <function BaseProcess.run at 0x7febd6286ca0> └ <SpawnProcess name='SpawnProcess-4' parent=1 started> File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) │ │ │ │ │ └ {'config': <uvicorn.config.Config object at 0x7febd5f1ad50>, 'target': <bound method Process.target of <uvicorn.supervisors.m... │ │ │ │ └ <SpawnProcess name='SpawnProcess-4' parent=1 started> │ │ │ └ () │ │ └ <SpawnProcess name='SpawnProcess-4' parent=1 started> │ └ <function subprocess_started at 0x7febd542d440> └ <SpawnProcess name='SpawnProcess-4' parent=1 started> File "/usr/local/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started target(sockets=sockets) │ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>] └ <bound method Process.target of <uvicorn.supervisors.multiprocess.Process object at 0x7febd5359c50>> File "/usr/local/lib/python3.11/site-packages/uvicorn/supervisors/multiprocess.py", line 64, in target return self.real_target(sockets) │ │ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>] │ └ <bound method Server.run of <uvicorn.server.Server object at 0x7febd5359d90>> └ <uvicorn.supervisors.multiprocess.Process object at 0x7febd5359c50> File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 67, in run return asyncio_run(self.serve(sockets=sockets), loop_factory=self.config.get_loop_factory()) │ │ │ │ │ │ └ <function Config.get_loop_factory at 0x7febd55bc400> │ │ │ │ │ └ <uvicorn.config.Config object at 0x7febd5f1ad50> │ │ │ │ └ <uvicorn.server.Server object at 0x7febd5359d90> │ │ │ └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8080)>] │ │ └ <function Server.serve at 0x7febd542c220> │ └ <uvicorn.server.Server object at 0x7febd5359d90> └ <function asyncio_run at 0x7febd5568a40> File "/usr/local/lib/python3.11/site-packages/uvicorn/_compat.py", line 30, in asyncio_run return runner.run(main) │ │ └ <coroutine object Server.serve at 0x7febd52b7b50> │ └ <function Runner.run at 0x7febd57d94e0> └ <asyncio.runners.Runner object at 0x7febd5359910> File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) │ │ │ └ <Task pending name='Task-1' coro=<Server.serve() running at /usr/local/lib/python3.11/site-packages/uvicorn/server.py:71> wai... │ │ └ <cyfunction Loop.run_until_complete at 0x7febd507a980> │ └ <uvloop.Loop running=True closed=False debug=False> └ <asyncio.runners.Runner object at 0x7febd5359910> File "/app/backend/open_webui/main.py", line 1868, in process_chat return await process_chat_response(response, ctx) │ │ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message... │ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50> └ <function process_chat_response at 0x7fea4723a840> File "/app/backend/open_webui/utils/middleware.py", line 4730, in process_chat_response return await streaming_chat_response_handler(response, ctx) │ │ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message... │ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50> └ <function streaming_chat_response_handler at 0x7fea4723a7a0> File "/app/backend/open_webui/utils/middleware.py", line 4678, in streaming_chat_response_handler return await response_handler(response, events) │ │ └ [] │ └ <starlette.responses.StreamingResponse object at 0x7fea3dba3e50> └ <function streaming_chat_response_handler.<locals>.response_handler at 0x7fea3e812d40> File "/app/backend/open_webui/utils/middleware.py", line 4659, in response_handler await background_tasks_handler(ctx) │ └ {'request': <starlette.requests.Request object at 0x7fea3e9ba710>, 'form_data': {'stream': True, 'model': 'gpt-5.2', 'message... └ <function background_tasks_handler at 0x7fea4723a660> File "/app/backend/open_webui/utils/middleware.py", line 2836, in background_tasks_handler res = await generate_follow_ups( └ <function generate_follow_ups at 0x7fea4972db20> > File "/app/backend/open_webui/routers/tasks.py", line 326, in generate_follow_ups return await generate_chat_completion(request, form_data=payload, user=user) │ │ │ └ UserModel(id='<ID>', email='<EMAIL>', username=None, role='user', ... │ │ └ {'model': 'gpt-5.2', 'messages': [{'role': 'user', 'content': '### Task:\nSuggest 3-5 relevant follow-up questions or prompts... │ └ <starlette.requests.Request object at 0x7fea3e9ba710> └ <function generate_chat_completion at 0x7fea4972d120> File "/app/backend/open_webui/utils/chat.py", line 188, in generate_chat_completion raise Exception("Model not found") Exception: Model not found ### Additional Information Root Cause Analysis: This issue is caused by a race condition in RedisDict.set at backend/open_webui/socket/utils.py. This is a severe problem that becomes visible only in deployments with high load, many replicas, and a large number of activity and models. Any time a pod refreshes the model list (via /api/models, admin changes, get_all_models, or cache refresh), it executes request.app.state.MODELS.set(models_dict). RedisDict.set uses a Redis pipeline: - DELETE the entire models hash - HSET the new model mapping Redis pipelines are not atomic. There is a moment between DELETE and HSET where the hash does not exist. During this small window, any pod checking for a model (such as in generate_follow_ups or generate_chat_completion) may see the hash as empty, causing “model not found”. I am not a Redis expert, so the suggested fix courtesy of Claude should absolutely be taken with a grain of salt and could be completely useless. The exact implementation of the fix should be reviewed and validated by someone with actual Redis expertise: def set(self, mapping: dict): # Fetch existing field names only (not values) to compute stale keys. # HKEYS is cheap — it transfers only short model ID strings, not the large JSON values. existing_keys = set(self.redis.hkeys(self.name)) new_keys = set(mapping.keys()) keys_to_remove = existing_keys - new_keys # HSET first (add/update all new values), then HDEL (remove stale keys). # Critically, we never DELETE the whole hash — this eliminates the race window # where concurrent readers would see an empty models dict between a DELETE and HSET. pipe = self.redis.pipeline() if mapping: pipe.hset(self.name, mapping={k: json.dumps(v) for k, v in mapping.items()}) if keys_to_remove: pipe.hdel(self.name, *keys_to_remove) pipe.execute()

GiteaMirror added the bug label 2026-05-21 02:19:13 -05:00

GiteaMirror closed this issue

2026-05-21 02:19:14 -05:00

GiteaMirror commented

2026-05-21 02:19:17 -05:00

@tjbck commented on GitHub (Mar 22, 2026):

Addressed in dev.

@tjbck commented on GitHub (Mar 22, 2026): Addressed in dev.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#123112