Replicas do not update certain settings until pod restart - split brain issues #3948

New Issue

GiteaMirror · 2025-11-11T15:43:03-06:00

GiteaMirror commented

2025-11-11 15:43:03 -06:00

Originally created by @tkg61 on GitHub (Feb 18, 2025).

Bug Report

When having more than 1 instance of openwebui certain settings will not sync between replicas when logging in with the same user until the pod is restarted.

i haven't gone through every single setting to see which are dynamically refreshed and which ones aren't but here are a few that i think should be working but do not:

Settings:
- Interface Banners
- model connection urls
- setting task model
- model direct connection flag
- channels (setting to turn on as well as any existing channels)

The bigger issue in my opinion, rather than a small inconvenience, is that this creates a possible split brain issue because which setting is used depends on which replica pod was used to save settings last and then which pods get restarted to then load the last saved state.

This prevents multi-admins from maintaining the system since you don't know who is making what change on which replica...

The bandaid for this is for admins to restart every pod after every backend change which is not-preferred since you don't want to accidently save settings on the wrong replica and turn off a feature that others were using because it hadn't been restarted since the last time something was changed.
The other bandaid is to only use env settings to manage the OWUI instance but that isn't always feasible/preferred.

There are things that will obviously sync just fine dynamically (web page refresh shows update) e.g.:

groups (additions/deletions)
chat threads
available models

Installation Method

K8s, latest helm chart

Environment

Open WebUI Version: 0.5.14

This has been happening even since 0.4.8

Confirmation:

I have read and followed all the instructions provided in the README.md.
I am on the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

When settings are saved all replicas will update any setting on refresh

Actual Behavior:

Pods will not pick up updates until the pod is restarted which can lead to split brain issues.

Description

Bug Summary:
Certain settings are not synced between replicas

Reproduction Details

Steps to Reproduce:

Setup a k8s deployment via helm chart with web sockets turned on using redis and external postgres deployment
port forward to each individual replica
change the setting on one and it will not sync to the others even after refresh or log out/login
restart the pod to see the changes finally pulled from the DB
if you forget to restart all pods and get routed to the wrong replica later and save improper settings, things will be lost

Logs and Screenshots

Screenshots/Screen Recordings (if applicable):
Channel setting:

after the pod on the left restarts, it pulls the channel = off setting since the pod on the right was last used to update the config. if you wanted channels to be on, you should have restarted the pods first to apply the setting everywhere

Originally created by @tkg61 on GitHub (Feb 18, 2025). # Bug Report When having more than 1 instance of openwebui certain settings will not sync between replicas when logging in with the same user until the pod is restarted. i haven't gone through every single setting to see which are dynamically refreshed and which ones aren't but here are a few that i think should be working but do not: - Settings: - Interface Banners - model connection urls - setting task model - model direct connection flag - channels (setting to turn on as well as any existing channels) **The bigger issue in my opinion, rather than a small inconvenience, is that this creates a possible split brain issue because which setting is used depends on which replica pod was used to save settings last and then which pods get restarted to then load the last saved state.** This prevents multi-admins from maintaining the system since you don't know who is making what change on which replica... The bandaid for this is for admins to restart every pod after every backend change which is not-preferred since you don't want to accidently save settings on the wrong replica and turn off a feature that others were using because it hadn't been restarted since the last time something was changed. The other bandaid is to only use env settings to manage the OWUI instance but that isn't always feasible/preferred. There are things that will obviously sync just fine dynamically (web page refresh shows update) e.g.: - groups (additions/deletions) - chat threads - available models ## Installation Method K8s, latest helm chart ## Environment - **Open WebUI Version:** 0.5.14 This has been happening even since 0.4.8 **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [x] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: When settings are saved all replicas will update any setting on refresh ## Actual Behavior: Pods will not pick up updates until the pod is restarted which can lead to split brain issues. ## Description **Bug Summary:** Certain settings are not synced between replicas ## Reproduction Details **Steps to Reproduce:** - Setup a k8s deployment via helm chart with web sockets turned on using redis and external postgres deployment - port forward to each individual replica - change the setting on one and it will not sync to the others even after refresh or log out/login - restart the pod to see the changes finally pulled from the DB - if you forget to restart all pods and get routed to the wrong replica later and save improper settings, things will be lost ## Logs and Screenshots **Screenshots/Screen Recordings (if applicable):** Channel setting: ![Image](https://github.com/user-attachments/assets/197de48a-1a47-4538-86b9-11fce6e90b96) after the pod on the left restarts, it pulls the channel = off setting since the pod on the right was last used to update the config. if you wanted channels to be on, you should have restarted the pods first to apply the setting everywhere ![Image](https://github.com/user-attachments/assets/c877fb26-f654-4945-b13e-b25cd296b102)

GiteaMirror closed this issue

2025-11-11 15:43:04 -06:00

GiteaMirror commented

2025-11-11 15:43:04 -06:00

@tjbck commented on GitHub (Feb 19, 2025):

#10365

@tjbck commented on GitHub (Feb 19, 2025): #10365

GiteaMirror referenced this issue

2025-11-11 17:46:39 -06:00

[PR #3948] [MERGED] 0.3.10 #8163

Sign in to join this conversation.