mirror of
https://github.com/open-webui/open-webui.git
synced 2026-05-07 11:28:35 -05:00
[GH-ISSUE #16158] issue: Processing does not continue after open_webui.retrieval.utils:generate_openai_batch_embeddings call #17807
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#17807
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @BAngelis on GitHub (Jul 30, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/16158
Check Existing Issues
Installation Method
Docker
Open WebUI Version
v0.6.18
Ollama Version (if applicable)
n/a
Operating System
Ubuntu 22.04 LTS
Browser (if applicable)
Chrome 138.0.7204.169 (Official Build) (arm64)
Confirmation
README.md.Expected Behavior
After tens of minutes of having successful request responses, once leaving the system idle for 30 minutes and making a request to OpenWeb UI you expect a response back to the browser, but it never comes.
Actual Behavior
After tens of minutes of having successful request/response handled, once leaving the system idle for 30 minutes and making a request to OpenWeb UI does not return a response to the browser, and the browser app eventually times out. Looking at the openwebui log entries, you can see the last log entry is : ai4l-openwebui | 2025-07-30T00:06:14.538244099Z 2025-07-30 00:06:14.537 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 1 - {}
Then, 3 minutes later the openwebui container goes "unhealthy" and is restarted by my watchdog.
Here are the logs leading up to the issue:
Steps to Reproduce
Overall deployment environment:
Steps to reproduce
Here is my docker-compose.yml
Define the logging configuration as an anchor
x-logging: &default-logging
driver: json-file
options:
max-size: "5m"
max-file: "3"
compress: "true"
services:
openwebui:
image: ghcr.io/open-webui/open-webui:latest
container_name: ai4l-openwebui
logging: *default-logging
environment:
- GLOBAL_LOG_LEVEL=DEBUG
- VECTOR_DB=pinecone
- PINECONE_API_KEY=removed
- PINECONE_ENVIRONMENT=eastus2
- PINECONE_INDEX_NAME=ai4l
- PINECONE_DIMENSION=1536
- PINECONE_METRIC=cosine
- PINECONE_CLOUD=azure
- PORT=8080
- OAUTH_ENABLED=false
- WEBUI_ALLOW_REGISTRATION=false
- RAG_EMBEDDING_OPENAI_BATCH_SIZE=16
- RAG_EMBEDDING_BATCH_SIZE=16
volumes:
- /mnt_openwebui_data/data:/app/backend/data
- /mnt_openwebui_data/config:/app/backend/config
# Add net-tools installation
command: >
sh -c "
apt-get update && apt-get install -y net-tools &&
cd /app/backend && exec bash start.sh
"
restart: unless-stopped
networks:
- openwebui-network
nginx:
image: nginx:latest
container_name: openwebui-nginx
logging: *default-logging
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- /etc/letsencrypt:/etc/letsencrypt:ro
ports:
- "80:80"
- "443:443"
restart: unless-stopped
networks:
- openwebui-network
depends_on:
- openwebui
WebSocket Monitoring Service
system-monitor:
image: alpine:latest
container_name: websocket-monitor
logging: *default-logging
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./app/watchstats.sh:/app/watchstats.sh:ro
- ./monitor_logs:/app/logs
command: >
sh -c "
apk add --no-cache docker &&
cd /app &&
exec sh ./watchstats.sh
"
restart: unless-stopped
networks:
- openwebui-network
depends_on:
- openwebui
networks:
openwebui-network:
driver: bridge
When the issue occurs : Logs show a “hang and then a restart”. Note the 3 minutes that pass before the restart. My watchdog detects the container is “unhealthy” and restarts it.
nd.openxmlformats-officedocument.wordprocessingml.document', 'size': 36166, 'data': {}, 'collection_name': '19c13be5-12f0-41cb-a109-f2826d8f6382'}, 'created_at': 1749841911, 'updated_at': 1749841911}], 'type': 'collection'}]}, 'access_control': None, 'is_active': True, 'updated_at': 1750898365, 'created_at': 1750898365}, 'preset': True, 'actions': [], 'filters': [], 'tags': []}, 'direct': False}} - {}
ai4l-openwebui | 2025-07-30T00:06:14.538244099Z 2025-07-30 00:06:14.537 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 1 - {}
ai4l-openwebui | 2025-07-30T00:09:11.308691394Z Hit:1 http://deb.debian.org/debian bookworm InRelease
ai4l-openwebui | 2025-07-30T00:09:11.308721894Z Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
ai4l-openwebui | 2025-07-30T00:09:11.308726794Z Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
ai4l-openwebui | 2025-07-30T00:09:12.350788408Z Reading package lists...
ai4l-openwebui | 2025-07-30T00:09:13.121076718Z Reading package lists...
ai4l-openwebui | 2025-07-30T00:09:13.300521005Z Building dependency tree...
ai4l-openwebui | 2025-07-30T00:09:13.301012809Z Reading state information...
ai4l-openwebui |
The request that resulted in this “hang” occured at 2025-07-30T00:06:14.311. That request was made about 30 minutes AFTER the previous successful request/response. There was no openweb activity between the previous successful interaction and this one, so openweb was “idle” during this 30 minute gap. It seems like the “hangs” often occur when there is a gap in usage. While attempting to reproduce the issue, I use two clients - one on Mac OS, one from IOS (using Chrome), but it works for many 10s of minutes. If I stop testing, and then begin again (without a reset of the services), I seem to often get this hang right away (but NOT always).
imageCompressionSize': {'width': '', 'height': ''}, 'landingPageMode': '', 'showUsername': True, 'notifications': {'webhook_url': ''}, 'webSearch': None, 'params': {}, 'audio': {'stt': {}, 'tts': {'engineConfig': {}, 'playbackRate': 1, 'voice': 'EXAVITQu4vr4xnSDxMaL', 'defaultVoice': ''}}, 'memory': True}), info=None, oauth_sub=None))] 1 (0.0000)s - {}
ai4l-openwebui | 2025-07-29T23:27:04.101757591Z 2025-07-29 23:27:04.101 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/chat/completed HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:04.940614661Z 2025-07-29 23:27:04.940 | DEBUG | open_webui.socket.main:periodic_usage_pool_cleanup:174 - Cleaning up model 4l---our-company-kb from usage pool - {}
ai4l-openwebui | 2025-07-29T23:27:10.343281379Z 2025-07-29 23:27:10.342 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/v1/chats/700a4282-ac0f-4605-b350-734c5e55fecb HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:10.634581236Z 2025-07-29 23:27:10.634 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:10.711955110Z 2025-07-29 23:27:10.711 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/folders/ HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:13.918433448Z 2025-07-30 00:06:13.918 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/v1/chats/700a4282-ac0f-4605-b350-734c5e55fecb HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.135890915Z 2025-07-30 00:06:14.135 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.220920748Z 2025-07-30 00:06:14.220 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/folders/ HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.311436675Z 2025-07-30 00:06:14.310 | DEBUG | open_webui.utils.middleware:process_chat_payload:734 - form_data: {'stream': True, 'model': '4l---our-company-kb', 'messages': [{'role': 'user', 'content':
You can see from the WebSocket monitor that during the time of the hang, connections start to stack up.
websocket-monitor | 2025-07-30T00:06:14.334385862Z [2025-07-30 00:06:09] MEM: 457.7MiB / 31.35GiB | CPU: 9.36% | CONN: 2 | WS: 1 | PROC: 1 | FD: 35 | SIO: 0
websocket-monitor | 2025-07-30T00:06:14.334419262Z 0
websocket-monitor | 2025-07-30T00:06:23.385120638Z [2025-07-30 00:06:19] MEM: 457.9MiB / 31.35GiB | CPU: 0.13% | CONN: 3 | WS: 2 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:23.385166139Z 0
websocket-monitor | 2025-07-30T00:06:32.480488865Z [2025-07-30 00:06:28] MEM: 457.9MiB / 31.35GiB | CPU: 0.14% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:32.480527365Z 0
websocket-monitor | 2025-07-30T00:06:41.534318302Z [2025-07-30 00:06:37] MEM: 460.6MiB / 31.35GiB | CPU: 0.13% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:41.534351002Z 0
websocket-monitor | 2025-07-30T00:06:50.558386329Z [2025-07-30 00:06:46] MEM: 460.6MiB / 31.35GiB | CPU: 0.15% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:50.558423829Z 0
websocket-monitor | 2025-07-30T00:06:59.631845363Z [2025-07-30 00:06:55] MEM: 460.6MiB / 31.35GiB | CPU: 0.13% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:59.631912964Z 0
websocket-monitor | 2025-07-30T00:07:08.682524138Z [2025-07-30 00:07:04] MEM: 460.4MiB / 31.35GiB | CPU: 0.13% | CONN: 6 | WS: 5 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:07:08.682567539Z 0
websocket-monitor | 2025-07-30T00:07:17.751985180Z [2025-07-30 00:07:13] MEM: 460.4MiB / 31.35GiB | CPU: 0.14% | CONN: 6 | WS: 5 | PROC: 1 | FD: 36 | SIO: 0
Then near the ai4l-openwebui restart, the monitor shows resources being released
websocket-monitor | 2025-07-30T00:08:02.920865190Z [2025-07-30 00:07:58] MEM: 463.1MiB / 31.35GiB | CPU: 0.16% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:02.920909690Z 0
websocket-monitor | 2025-07-30T00:08:11.950835398Z [2025-07-30 00:08:07] MEM: 462.9MiB / 31.35GiB | CPU: 0.14% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:11.950935599Z 0
websocket-monitor | 2025-07-30T00:08:20.999781865Z [2025-07-30 00:08:16] MEM: 462.9MiB / 31.35GiB | CPU: 0.18% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:20.999821765Z 0
websocket-monitor | 2025-07-30T00:08:30.020799431Z [2025-07-30 00:08:26] MEM: 462.9MiB / 31.35GiB | CPU: 0.15% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:30.020842931Z 0
websocket-monitor | 2025-07-30T00:08:39.034728224Z [2025-07-30 00:08:35] MEM: 465.7MiB / 31.35GiB | CPU: 0.16% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:39.034764825Z 0
websocket-monitor | 2025-07-30T00:08:48.080635586Z [2025-07-30 00:08:44] MEM: 465.7MiB / 31.35GiB | CPU: 0.15% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:48.080666086Z 0
websocket-monitor | 2025-07-30T00:08:57.109347335Z [2025-07-30 00:08:53] MEM: 465.7MiB / 31.35GiB | CPU: 0.17% | CONN: 8 | WS: 7 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:57.109387735Z 0
websocket-monitor | 2025-07-30T00:09:06.127351081Z [2025-07-30 00:09:02] MEM: 465.5MiB / 31.35GiB | CPU: 0.16% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:09:06.127390781Z 0
websocket-monitor | 2025-07-30T00:09:15.310945779Z [2025-07-30 00:09:11] MEM: 39.66MiB / 31.35GiB | CPU: 100.50% | CONN: 0
websocket-monitor | 2025-07-30T00:09:15.310979079Z 0 | WS: 0
websocket-monitor | 2025-07-30T00:09:15.310984079Z 0 | PROC: 1 | FD: 23 | SIO: 0
websocket-monitor | 2025-07-30T00:09:15.310987879Z 0
websocket-monitor | 2025-07-30T00:09:24.314871368Z [2025-07-30 00:09:20] MEM: 243.2MiB / 31.35GiB | CPU: 100.11% | CONN: 1 | WS: 0
websocket-monitor | 2025-07-30T00:09:24.314898268Z 0 | PROC: 1 | FD: 27 | SIO: 0
websocket-monitor | 2025-07-30T00:09:24.314902968Z 0
websocket-monitor | 2025-07-30T00:09:33.360325695Z [2025-07-30 00:09:29] MEM: 324.3MiB / 31.35GiB | CPU: 3.91% | CONN: 2 | WS: 1 | PROC: 1 | FD: 30 | SIO: 0
Logs & Screenshots
The last successful log message is from open_webui.retrieval.utils:generate_openai_batch_embeddings
ai4l-openwebui | 2025-07-30T00:06:14.538244099Z 2025-07-30 00:06:14.537 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 1 - {}
Normal logs from successful calls done previously… (this is what I expect to see after each generate_openai_batch_embeddings call
ai4l-openwebui | 2025-07-29T23:26:56.671475035Z 2025-07-29 23:26:56.671 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 3 - {}
ai4l-openwebui | 2025-07-29T23:26:56.993248363Z 2025-07-29 23:26:56.992 | DEBUG | open_webui.retrieval.utils:query_collection:299 - query_collection: processing 3 queries across 1 collections - {}
ai4l-openwebui | 2025-07-29T23:26:56.993677564Z 2025-07-29 23:26:56.993 | DEBUG | open_webui.retrieval.utils:query_doc:85 - query_doc:doc 19c13be5-12f0-41cb-a109-f2826d8f6382 - {}
ai4l-openwebui | 2025-07-29T23:26:56.994781567Z 2025-07-29 23:26:56.994 | DEBUG | open_webui.retrieval.utils:query_doc:85 - query_doc:doc 19c13be5-12f0-41cb-a109-f2826d8f6382 - {}
ai4l-openwebui | 2025-07-29T23:26:56.995842570Z 2025-07-29 23:26:56.995 | DEBUG | open_webui.retrieval.utils:query_doc:85 - query_doc:doc 19c13be5-12f0-41cb-a109-f2826d8f6382 - {}
ai4l-openwebui | 2025-07-29T23:26:57.068472657Z 2025-07-29 23:26:57.067 | INFO | open_webui.retrieval.utils:query_doc:93 - query_doc:result [['76245fe7-22a8-4c31-a3ff-6b500dc48e9b', '0e04f732-264f-48a2-8cf3-860592f318e3', 'f65ed728-a131-4d10-873d-07e44641cede', 'a579a348-86fb-47ca-b00e-9448eb1fbc18', '43ab5288-a960-4da7-917b-4c934d63ddff', '01b21117-6960-4007-8bb6-63935075454a', '1e9e202a-0172-
When the issue occurs : Logs show a “hang and then a restart”. Note the 3 minutes that pass before the restart. My watchdog detects the container is “unhealthy” and restarts it.
nd.openxmlformats-officedocument.wordprocessingml.document', 'size': 36166, 'data': {}, 'collection_name': '19c13be5-12f0-41cb-a109-f2826d8f6382'}, 'created_at': 1749841911, 'updated_at': 1749841911}], 'type': 'collection'}]}, 'access_control': None, 'is_active': True, 'updated_at': 1750898365, 'created_at': 1750898365}, 'preset': True, 'actions': [], 'filters': [], 'tags': []}, 'direct': False}} - {}
ai4l-openwebui | 2025-07-30T00:06:14.538244099Z 2025-07-30 00:06:14.537 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 1 - {}
ai4l-openwebui | 2025-07-30T00:09:11.308691394Z Hit:1 http://deb.debian.org/debian bookworm InRelease
ai4l-openwebui | 2025-07-30T00:09:11.308721894Z Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
ai4l-openwebui | 2025-07-30T00:09:11.308726794Z Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
ai4l-openwebui | 2025-07-30T00:09:12.350788408Z Reading package lists...
ai4l-openwebui | 2025-07-30T00:09:13.121076718Z Reading package lists...
ai4l-openwebui | 2025-07-30T00:09:13.300521005Z Building dependency tree...
ai4l-openwebui | 2025-07-30T00:09:13.301012809Z Reading state information...
ai4l-openwebui |
The request that resulted in this “hang” occured at 2025-07-30T00:06:14.311. That request was made about 30 minutes AFTER the several previous successful request/responses. There was no openweb activity between the previous successful interaction and this one, so openweb was “idle” during this 30 minute gap. It seems like the “hangs” often occur when there is a gap in usage. While attempting to reproduce the issue, I use two clients - one on Mac OS, one from IOS (using Chrome), but it works for many 10s of minutes. If I stop testing, and then begin again (without a reset of the services), I seem to often get this hang right away (but NOT always).
imageCompressionSize': {'width': '', 'height': ''}, 'landingPageMode': '', 'showUsername': True, 'notifications': {'webhook_url': ''}, 'webSearch': None, 'params': {}, 'audio': {'stt': {}, 'tts': {'engineConfig': {}, 'playbackRate': 1, 'voice': 'EXAVITQu4vr4xnSDxMaL', 'defaultVoice': ''}}, 'memory': True}), info=None, oauth_sub=None))] 1 (0.0000)s - {}
ai4l-openwebui | 2025-07-29T23:27:04.101757591Z 2025-07-29 23:27:04.101 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/chat/completed HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:04.940614661Z 2025-07-29 23:27:04.940 | DEBUG | open_webui.socket.main:periodic_usage_pool_cleanup:174 - Cleaning up model 4l---our-company-kb from usage pool - {}
ai4l-openwebui | 2025-07-29T23:27:10.343281379Z 2025-07-29 23:27:10.342 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/v1/chats/700a4282-ac0f-4605-b350-734c5e55fecb HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:10.634581236Z 2025-07-29 23:27:10.634 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-29T23:27:10.711955110Z 2025-07-29 23:27:10.711 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/folders/ HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:13.918433448Z 2025-07-30 00:06:13.918 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "POST /api/v1/chats/700a4282-ac0f-4605-b350-734c5e55fecb HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.135890915Z 2025-07-30 00:06:14.135 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.220920748Z 2025-07-30 00:06:14.220 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 98.97.33.128:0 - "GET /api/v1/folders/ HTTP/1.1" 200 - {}
ai4l-openwebui | 2025-07-30T00:06:14.311436675Z 2025-07-30 00:06:14.310 | DEBUG | open_webui.utils.middleware:process_chat_payload:734 - form_data: {'stream': True, 'model': '4l---our-company-kb', 'messages': [{'role': 'user', 'content':
You can see from the WebSocket monitor that during the time of the hang, connections start to stack up (but NOT a large amount, so I don't see this as causation).
websocket-monitor | 2025-07-30T00:06:14.334385862Z [2025-07-30 00:06:09] MEM: 457.7MiB / 31.35GiB | CPU: 9.36% | CONN: 2 | WS: 1 | PROC: 1 | FD: 35 | SIO: 0
websocket-monitor | 2025-07-30T00:06:14.334419262Z 0
websocket-monitor | 2025-07-30T00:06:23.385120638Z [2025-07-30 00:06:19] MEM: 457.9MiB / 31.35GiB | CPU: 0.13% | CONN: 3 | WS: 2 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:23.385166139Z 0
websocket-monitor | 2025-07-30T00:06:32.480488865Z [2025-07-30 00:06:28] MEM: 457.9MiB / 31.35GiB | CPU: 0.14% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:32.480527365Z 0
websocket-monitor | 2025-07-30T00:06:41.534318302Z [2025-07-30 00:06:37] MEM: 460.6MiB / 31.35GiB | CPU: 0.13% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:41.534351002Z 0
websocket-monitor | 2025-07-30T00:06:50.558386329Z [2025-07-30 00:06:46] MEM: 460.6MiB / 31.35GiB | CPU: 0.15% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:50.558423829Z 0
websocket-monitor | 2025-07-30T00:06:59.631845363Z [2025-07-30 00:06:55] MEM: 460.6MiB / 31.35GiB | CPU: 0.13% | CONN: 5 | WS: 4 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:06:59.631912964Z 0
websocket-monitor | 2025-07-30T00:07:08.682524138Z [2025-07-30 00:07:04] MEM: 460.4MiB / 31.35GiB | CPU: 0.13% | CONN: 6 | WS: 5 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:07:08.682567539Z 0
websocket-monitor | 2025-07-30T00:07:17.751985180Z [2025-07-30 00:07:13] MEM: 460.4MiB / 31.35GiB | CPU: 0.14% | CONN: 6 | WS: 5 | PROC: 1 | FD: 36 | SIO: 0
Then near the ai4l-openwebui restart, the monitor shows resources being released
websocket-monitor | 2025-07-30T00:08:02.920865190Z [2025-07-30 00:07:58] MEM: 463.1MiB / 31.35GiB | CPU: 0.16% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:02.920909690Z 0
websocket-monitor | 2025-07-30T00:08:11.950835398Z [2025-07-30 00:08:07] MEM: 462.9MiB / 31.35GiB | CPU: 0.14% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:11.950935599Z 0
websocket-monitor | 2025-07-30T00:08:20.999781865Z [2025-07-30 00:08:16] MEM: 462.9MiB / 31.35GiB | CPU: 0.18% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:20.999821765Z 0
websocket-monitor | 2025-07-30T00:08:30.020799431Z [2025-07-30 00:08:26] MEM: 462.9MiB / 31.35GiB | CPU: 0.15% | CONN: 7 | WS: 6 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:30.020842931Z 0
websocket-monitor | 2025-07-30T00:08:39.034728224Z [2025-07-30 00:08:35] MEM: 465.7MiB / 31.35GiB | CPU: 0.16% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:39.034764825Z 0
websocket-monitor | 2025-07-30T00:08:48.080635586Z [2025-07-30 00:08:44] MEM: 465.7MiB / 31.35GiB | CPU: 0.15% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:48.080666086Z 0
websocket-monitor | 2025-07-30T00:08:57.109347335Z [2025-07-30 00:08:53] MEM: 465.7MiB / 31.35GiB | CPU: 0.17% | CONN: 8 | WS: 7 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:08:57.109387735Z 0
websocket-monitor | 2025-07-30T00:09:06.127351081Z [2025-07-30 00:09:02] MEM: 465.5MiB / 31.35GiB | CPU: 0.16% | CONN: 9 | WS: 8 | PROC: 1 | FD: 36 | SIO: 0
websocket-monitor | 2025-07-30T00:09:06.127390781Z 0
websocket-monitor | 2025-07-30T00:09:15.310945779Z [2025-07-30 00:09:11] MEM: 39.66MiB / 31.35GiB | CPU: 100.50% | CONN: 0
websocket-monitor | 2025-07-30T00:09:15.310979079Z 0 | WS: 0
websocket-monitor | 2025-07-30T00:09:15.310984079Z 0 | PROC: 1 | FD: 23 | SIO: 0
websocket-monitor | 2025-07-30T00:09:15.310987879Z 0
websocket-monitor | 2025-07-30T00:09:24.314871368Z [2025-07-30 00:09:20] MEM: 243.2MiB / 31.35GiB | CPU: 100.11% | CONN: 1 | WS: 0
websocket-monitor | 2025-07-30T00:09:24.314898268Z 0 | PROC: 1 | FD: 27 | SIO: 0
websocket-monitor | 2025-07-30T00:09:24.314902968Z 0
websocket-monitor | 2025-07-30T00:09:33.360325695Z [2025-07-30 00:09:29] MEM: 324.3MiB / 31.35GiB | CPU: 3.91% | CONN: 2 | WS: 1 | PROC: 1 | FD: 30 | SIO: 0
Additional Information
Here is my nginx configuration file.
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
@tjbck commented on GitHub (Jul 31, 2025):
#15023
@BAngelis commented on GitHub (Aug 1, 2025):
@tjbck thanks for looking at my issue. I added a AIOHTTP_CLIENT_TIMEOUT=30 to the environment (for a fail fast), but there is no change. Openweb UI doesnt log anything new (after the open_webui.retrieval.utils:generate_openai_batch_embeddings call) for 10+ minutes. It hangs up all other connected clients, nobody can log out or in. Then suddenly, after 17 minutes in this case, it starts responding to new requests.
I've been struggling with this issue for weeks. Where else can I get some help?!
Here's the logs after the AIOHTTP_CLIENT_TIMEOUT change..
@BAngelis commented on GitHub (Aug 1, 2025):
All I'm doing is chatting w/ a model, with a knowledge base in Pinecone. I'm not uploading files, or anything complicated (yet).
@rgaricano commented on GitHub (Aug 1, 2025):
Bruce, could you explain the whole setup & services that you are using?
(openwebui (install method), ollama or other llm provider, web proxy, database engine, embedd engine,...)
What it's your setup for pinecone (local or remote, starter or paid,..)...
I read about some timeouts error similar to yours...
https://community.pinecone.io/t/request-timeout/3750/2
https://community.n8n.io/t/pinecone-vector-store-timeout-with-large-whatsapp-chats-need-optimization-help/148202/4
https://community.pinecone.io/t/who-else-is-facing-504-gateway-timeouts-urgent/4064
...
I would try increasing the nginx timeouts, and since it seems most likely that the timeouts are coming from Pinecone, I would try another database engine to see if the same errors occur. That would help narrow down the problem.
@BAngelis commented on GitHub (Aug 8, 2025):
Hello @rgaricano,
Thanks for your response.
My setup here is
It seems to me that it stops performing normally between these two calls:
ai4l-openwebui | 2025-07-29T23:26:56.671475035Z 2025-07-29 23:26:56.671 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 3 - {}
ai4l-openwebui | 2025-07-29T23:26:56.993248363Z 2025-07-29 23:26:56.992 | DEBUG | open_webui.retrieval.utils:query_collection:299 - query_collection: processing 3 queries across 1 collections - {}
Because, everytime it blocks/hangs, the last DEBUG log entry is always "open_webui.retrieval.utils:generate_openai_batch_embeddings"
So adding the AIOHTTP_CLIENT_TIMEOUT seemed to make sense, but didn't change the behavior.
I'm experimenting with the nginx timeouts now, but not seeing any improvements yet.
@rgaricano commented on GitHub (Aug 8, 2025):
Not using redis for websockets?
@BAngelis commented on GitHub (Aug 8, 2025):
@rgaricano , No I'm not using redis, but happy to take any suggestions since this is totally blocking our progress.
After tweaking the timeouts i'm still able to get it to "hang" for 13 minutes between the embedding and query_collection calls. During that time, the clients are not able to stop their chats, log out, or nearly anything from the UI.
Here's a screen shot of the logs.
@BAngelis commented on GitHub (Aug 8, 2025):
@rgaricano, here is my new ngin.conf with better timeout settings.
@rgaricano commented on GitHub (Aug 8, 2025):
If you don't use any websocket manager then you have a bottleneck there. Without Redis, the task system falls back to local in-memory storage, which can cause enqueue failures.
@BAngelis commented on GitHub (Aug 8, 2025):
@rgaricano Good to know, sounds like whats happening (especially with multiple clients having chat sessions with several requests/responses). I'll dig into the docs on using Redis w/ Openweb. Thanks!
@rgaricano commented on GitHub (Aug 8, 2025):
https://docs.openwebui.com/tutorials/integrations/redis
@BAngelis commented on GitHub (Aug 9, 2025):
@rgaricano Okay, I've integrated Redis. But I can still create the hang/stall issue the same way. Two clients doing Q&A against models that use Pinecone as their Knowledge bases.
BUT I did get an additional clue. After 15 minutes the hang/stall is released and more log records are written (and clients error out simultaneously), I get these log ERRORs
`2025-08-08 23:20:52.254 | ERROR | open_webui.socket.main:periodic_usage_pool_cleanup:157 - Unable to renew cleanup lock. Exiting usage pool cleanup. - {}
2025-08-08 23:20:52.254 | DEBUG | open_webui.retrieval.utils:generate_openai_batch_embeddings:736 - generate_openai_batch_embeddings:model text-embedding-3-small batch size: 3 - {}
2025-08-08 23:20:52.255 | ERROR | asyncio.runners:run:118 - Task exception was never retrieved
future: <Task finished name='Task-4' coro=<periodic_usage_pool_cleanup() done, defined at /app/backend/open_webui/socket/main.py:133> exception=Exception('Unable to renew usage pool cleanup lock.')> - {}
Traceback (most recent call last):
Exception: Unable to renew usage pool cleanup lock.`
Here's a client side logs if needed
@rgaricano commented on GitHub (Aug 9, 2025):
yes, it's a connection interruption, of web and websockets,
did you check the Azure VM Load Balancer (maybe you have some of this kind of services enabled): https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout?source=recommendations&tabs=tcp-reset-idle-portal
(sorry but I don't not so much about azure services)
@BAngelis commented on GitHub (Aug 11, 2025):
@rgaricano Thanks for the feedback. I'm not yet using a load balancer, it's just a single VM instance running the entire configuration in Docker.
I'm going to remove my use of Pinecone as the Knowledge Base vector store. And re-run my tests.
@BAngelis commented on GitHub (Aug 11, 2025):
@rgaricano thanks for your support.
So far I've run the test suite three times using the default built in ChromaDB support, and no failures. So, the hang/stall on Openweb looks to be related to my use of Pinecone as the vector store.
According to the Openweb documentation, Pinecone is supported. But it doesn't work for me. Worked for basic chat, but not with multiple concurrent users doing typical research scenarios (long contexts, using many prompts). Its not a rate limit problem on the Pinceone side, but I am requesting the activity logs from them to see timing of API calls coming from my Openweb. To see what happens during that 13 to 15 minute OW stall after calling open_webui.retrieval.utils:generate_openai_batch_embeddings.
@theboyr commented on GitHub (Aug 12, 2025):
I ran into something similar after upgrading this weekend.
"Metadata value must be a string, number, boolean or list of strings" is an error that Pinecone throws that I'm getting a lot of lately. Pinecone does not support complex metadata.. but it appears OUI is now adding nested Metadata into Upsert to Pinecone.
I had planned to switch to S3 Vectors anyways so just a forcing function then i guess.
@BAngelis commented on GitHub (Aug 15, 2025):
@theboyr , @rgaricano , I had to fall back to ChromaDB (which does not meet our requirements), just to get it to work for now. Unfortunately the Pinecone support, and now I'm learning n8n integration are too buggy...
@theboyr commented on GitHub (Aug 16, 2025):
I moved everything to S3 Vectors when this happened, and works flawlessly for my use case.
I do think the Pinecone integration needs to be updated to remove any metadata that's nested, but we're an AWS shop to begin with and moving to S3 Vectors is better long term for our management. Plus, once we get out of free tier with Pinecone... it appears S3 Vectors will also be more cost effective.