github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-17 08:21:12 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #24751] bug: Critical memory leak in aiosqlite when editing multiple model/knowledge base permissions (OOM) #123706

New Issue
Closed
opened 2026-05-21 03:10:05 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-05-21 03:10:05 -05:00
Owner
Copy Link

Originally created by @jmc0x68 on GitHub (May 15, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24751

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.9.0-0.9.5

Ollama Version (if applicable)

none(didn't install)

Operating System

Debian 12 and Arch Linux(Linux 7.0.5-arch1-1)

Browser (if applicable)

Chrome 147.0

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

Permission modification completes quickly with minimal memory overhead.

Actual Behavior

When editing the access permissions of multiple models (or knowledge bases) in the Admin Panel, the open-webui container experiences a severe memory leak that quickly exhausts the allocated 1024MB of memory, causing the system's OOM killer to terminate the Python process. Profiling with py-spy and memray reveals that aiosqlite (specifically core.py) allocates over 3.5GB of memory during the operation, far exceeding the normal amount.

Steps to Reproduce

1.Use the attached docker-compose.yml to deploy Open WebUI (with remote Qdrant and offline mode).
(docker-compose.yml attached in Additional Information)

version: '3'

services:
 open-webui:
    image: ghcr.io/open-webui/open-webui:0.8-slim
    container_name: open-webui
    restart: unless-stopped
    cap_add:
      - SYS_PTRACE
    ports:
      - "3000:8080"
    volumes:
      - ./data/open-webui:/app/backend/data
    environment:
      - 'OFFLINE_MODE=true'
      - 'HF_HUB_OFFLINE=1'
      - 'RAG_EMBEDDING_MODEL_AUTO_UPDATE=false'
      - 'RAG_RERANKING_MODEL_AUTO_UPDATE=false'
      - 'WHISPER_MODEL_AUTO_UPDATE=false'
      - 'HF_ENDPOINT=https://hf-mirror.com'
      - 'ENABLE_MEMORIES=false'
      - 'ENABLE_WEBSOCKET_SUPPORT=true'
      - 'WEBHOOK_URL='
      - 'OPENAI_API_BASE_URLS=http://pipelines:9099'
      - 'OPENAI_API_KEY=${PIPELINES_API_KEY}'
      - 'VECTOR_DB=qdrant'
      - 'QDRANT_URI=${QDRANT_URI}'
      - 'QDRANT_API_KEY=${QDRANT_API_KEY}'
      - 'REQUEST_TIMEOUT=600'
      - 'AIORHTTP_TIMEOUT=600'
      - 'STREAMING_EMBEDDINGS=False'
    networks:
      - ai-network
    mem_limit: 1024m
    ulimits:    
      nofile:
        soft: 65535
        hard: 65535
networks:
  ai-network:
    driver: bridge

2.Ensure the environment contains a sufficient number of models (the leak is easier to trigger with a long model list).

3.Log in as an administrator.

4.Navigate to Admin Panel -> Settings -> Models.

5.Click on a model to edit, go to Access Control, add a user group, create another user, add one user into that user group, and click Save.
(Note: The model has Advanced Params configured: Function Calling → Native, Web Search, Image Generation, and Code Interpreter enabled.)

6.Repeat step 5 for multiple models.

7.Observe memory usage (docker stats) or wait for the container to be killed by OOM.

8.Affected Areas: The same issue can also be triggered (though less consistently) when editing knowledge base access permissions.

Logs & Screenshots

dmesg

Memory cgroup out of memory: Killed process 173382 (python3) total-vm:8839088kB, anon-rss:776180kB, file-rss:223312kB, shmem-rss:0kB, UID:0 pgtables:5256kB oom_score_adj:0

Docker logs(no Errors,only connection infos

openwebui_debug.log

Py-Spy Profilling

Image

Memray Analysis Flamegraph

memray-flamegraph.html

Summary: The flamegraph clearly shows that aiosqlite/core.py allocations dominate, accounting for over 3.5GB of memory during the permission modification process.

Additional Information

This bug was introduced after v0.8.12; all tested v0.9.x versions are affected.

The presence of other services (Web Search, DeepSeek Pipeline) does not appear to influence the leak; the trigger is strictly the permission editing workflow.

The size of the Qdrant database and the number of models both seem to correlate with the leak's severity.

I have attached all relevant profiling data to assist with debugging.

My System Special Configurations:

Remote Qdrant Cloud database (approx. 3.5GB used)

Offline mode enabled

No local Ollama service

Connected services: Web search engine, DeepSeek Pipeline

Model Advanced Params: Native Function Calling, Web Search, Image Generation, Code Interpreter enabled

Possible Root Cause (Preliminary Analysis)

Based on the memray flamegraph, aiosqlite is allocating a disproportionately large amount of memory during what should be a simple database operation (likely updating permission entries). This suggests a missing pagination, inefficient query, or object lifecycle bug in the ORM/database layer introduced in v0.9.0.

Originally created by @jmc0x68 on GitHub (May 15, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24751 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.9.0-0.9.5 ### Ollama Version (if applicable) none(didn't install) ### Operating System Debian 12 and Arch Linux(Linux 7.0.5-arch1-1) ### Browser (if applicable) Chrome 147.0 ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior Permission modification completes quickly with minimal memory overhead. ### Actual Behavior When editing the access permissions of multiple models (or knowledge bases) in the Admin Panel, the open-webui container experiences a severe memory leak that quickly exhausts the allocated 1024MB of memory, causing the system's OOM killer to terminate the Python process. Profiling with py-spy and memray reveals that aiosqlite (specifically core.py) allocates over 3.5GB of memory during the operation, far exceeding the normal amount. ### Steps to Reproduce 1.Use the attached docker-compose.yml to deploy Open WebUI (with remote Qdrant and offline mode). (docker-compose.yml attached in Additional Information) ``` version: '3' services: open-webui: image: ghcr.io/open-webui/open-webui:0.8-slim container_name: open-webui restart: unless-stopped cap_add: - SYS_PTRACE ports: - "3000:8080" volumes: - ./data/open-webui:/app/backend/data environment: - 'OFFLINE_MODE=true' - 'HF_HUB_OFFLINE=1' - 'RAG_EMBEDDING_MODEL_AUTO_UPDATE=false' - 'RAG_RERANKING_MODEL_AUTO_UPDATE=false' - 'WHISPER_MODEL_AUTO_UPDATE=false' - 'HF_ENDPOINT=https://hf-mirror.com' - 'ENABLE_MEMORIES=false' - 'ENABLE_WEBSOCKET_SUPPORT=true' - 'WEBHOOK_URL=' - 'OPENAI_API_BASE_URLS=http://pipelines:9099' - 'OPENAI_API_KEY=${PIPELINES_API_KEY}' - 'VECTOR_DB=qdrant' - 'QDRANT_URI=${QDRANT_URI}' - 'QDRANT_API_KEY=${QDRANT_API_KEY}' - 'REQUEST_TIMEOUT=600' - 'AIORHTTP_TIMEOUT=600' - 'STREAMING_EMBEDDINGS=False' networks: - ai-network mem_limit: 1024m ulimits: nofile: soft: 65535 hard: 65535 networks: ai-network: driver: bridge ``` 2.Ensure the environment contains a sufficient number of models (the leak is easier to trigger with a long model list). 3.Log in as an administrator. 4.Navigate to Admin Panel -> Settings -> Models. 5.Click on a model to edit, go to Access Control, add a user group, create another user, add one user into that user group, and click Save. (Note: The model has Advanced Params configured: Function Calling → Native, Web Search, Image Generation, and Code Interpreter enabled.) 6.Repeat step 5 for multiple models. 7.Observe memory usage (docker stats) or wait for the container to be killed by OOM. 8.Affected Areas: The same issue can also be triggered (though less consistently) when editing knowledge base access permissions. ### Logs & Screenshots ### dmesg `Memory cgroup out of memory: Killed process 173382 (python3) total-vm:8839088kB, anon-rss:776180kB, file-rss:223312kB, shmem-rss:0kB, UID:0 pgtables:5256kB oom_score_adj:0` ### Docker logs(no Errors,only connection infos [openwebui_debug.log](https://github.com/user-attachments/files/27795968/openwebui_debug.log) ### Py-Spy Profilling <img width="1200" height="714" alt="Image" src="https://github.com/user-attachments/assets/fa37bb7d-9860-4382-a7f0-3d21fdb4793c" /> ### Memray Analysis Flamegraph [memray-flamegraph.html](https://github.com/user-attachments/files/27795992/memray-flamegraph.html) **Summary**: The flamegraph clearly shows that aiosqlite/core.py allocations dominate, accounting for over 3.5GB of memory during the permission modification process. ### Additional Information This bug was introduced after v0.8.12; all tested v0.9.x versions are affected. The presence of other services (Web Search, DeepSeek Pipeline) does not appear to influence the leak; the trigger is strictly the permission editing workflow. The size of the Qdrant database and the number of models both seem to correlate with the leak's severity. I have attached all relevant profiling data to assist with debugging. ### My System Special Configurations: Remote Qdrant Cloud database (approx. 3.5GB used) Offline mode enabled No local Ollama service Connected services: Web search engine, DeepSeek Pipeline Model Advanced Params: Native Function Calling, Web Search, Image Generation, Code Interpreter enabled ### Possible Root Cause (Preliminary Analysis) Based on the memray flamegraph, aiosqlite is allocating a disproportionately large amount of memory during what should be a simple database operation (likely updating permission entries). This suggests a missing pagination, inefficient query, or object lifecycle bug in the ORM/database layer introduced in v0.9.0.
GiteaMirror added the bug label 2026-05-21 03:10:05 -05:00
GiteaMirror commented 2026-05-21 03:10:08 -05:00
Author
Owner
Copy Link

@owui-terminator[bot] commented on GitHub (May 15, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

  1. 🟣 #21377 issue: [object Object] error when changing model access control in Admin Panel (v0.8.0)
    This is directly about changing model access control in the Admin Panel, the same workflow as the new report. It shows a related permission-editing path and may point to the same model access control code path, even though the symptom there was a 404/UI error rather than OOM.
    by ahgray · bug

  2. 🟣 #8002 Group permissions for model not persistent
    This issue also concerns model group permissions not persisting when editing access settings. It is closely related to the same model permission editing feature, so it may share backend/UI/database code with the memory leak report.
    by Ivar-Nelson

  3. 🟣 #20229 issue: Knowledge Base not visible when shared with multiple groups (SQLite)
    This is a knowledge-base permission bug involving multi-group sharing and SQLite/access_control handling. Since the new issue also mentions knowledge base permission edits as a secondary trigger, this is a relevant adjacent issue in the same subsystem.
    by taka817123 · bug, confirmed issue

  4. 🟣 #11331 issue: Knowledge base permission issue
    This issue covers knowledge base permission/access problems and involves the same permission-management area. It is less specific than the new leak report, but still related because the new report says knowledge base access edits can also trigger the problem.
    by sleepyddl · bug

  5. 🟣 #11570 issue: Users can still view but cannot access knowledge bases/custom models after permissions are closed, and resources are deleted simultaneously when users delete them
    This report involves both knowledge bases and custom models after permission changes, which is the same general domain as the new issue. The overlap suggests the permission update logic for shared resources may be involved.
    by WYL-BruceLong · bug

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:4458838839 --> @owui-terminator[bot] commented on GitHub (May 15, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟣 [#21377](https://github.com/open-webui/open-webui/issues/21377) **issue: [object Object] error when changing model access control in Admin Panel (v0.8.0)** *This is directly about changing model access control in the Admin Panel, the same workflow as the new report. It shows a related permission-editing path and may point to the same model access control code path, even though the symptom there was a 404/UI error rather than OOM.* *by ahgray · `bug`* 2. 🟣 [#8002](https://github.com/open-webui/open-webui/issues/8002) **Group permissions for model not persistent** *This issue also concerns model group permissions not persisting when editing access settings. It is closely related to the same model permission editing feature, so it may share backend/UI/database code with the memory leak report.* *by Ivar-Nelson* 3. 🟣 [#20229](https://github.com/open-webui/open-webui/issues/20229) **issue: Knowledge Base not visible when shared with multiple groups (SQLite)** *This is a knowledge-base permission bug involving multi-group sharing and SQLite/access_control handling. Since the new issue also mentions knowledge base permission edits as a secondary trigger, this is a relevant adjacent issue in the same subsystem.* *by taka817123 · `bug`, `confirmed issue`* 4. 🟣 [#11331](https://github.com/open-webui/open-webui/issues/11331) **issue: Knowledge base permission issue** *This issue covers knowledge base permission/access problems and involves the same permission-management area. It is less specific than the new leak report, but still related because the new report says knowledge base access edits can also trigger the problem.* *by sleepyddl · `bug`* 5. 🟣 [#11570](https://github.com/open-webui/open-webui/issues/11570) **issue: Users can still view but cannot access knowledge bases/custom models after permissions are closed, and resources are deleted simultaneously when users delete them** *This report involves both knowledge bases and custom models after permission changes, which is the same general domain as the new issue. The overlap suggests the permission update logic for shared resources may be involved.* *by WYL-BruceLong · `bug`* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.
GiteaMirror commented 2026-05-21 03:10:08 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (May 15, 2026):

Thanks

This is not a leak inside aiosqlite, and it is unrelated to Qdrant. The 3.5 GB matching your Qdrant size is coincidental: aiosqlite is the local SQLite driver, Qdrant is a separate remote service, they share no memory path.

What you are hitting is the memory profile of the async SQLite layer introduced in v0.9.0 (you are right that v0.8.12 predates it and is unaffected), running under a 1024 MB cap:

On SQLite, with DATABASE_POOL_SIZE unset the engine falls back to a pool of up to 512 connections.
Each connection lazily grows its own SQLite page cache up to ~64 MB (PRAGMA cache_size=-65536) plus a 256 MB mmap window (PRAGMA mmap_size). These are per-connection, not shared.

Editing permissions and then reloading the model list opens many connections (more models means more connections), so peak memory scales with how many are active at once, each holding its own cache. That is exactly why memray attributes almost everything to aiosqlite/core.py: that frame is where each connection materialises rows and holds its cache. It is the signature of many connections, not a bug in aiosqlite. Your total-vm: 8.4 GB with RSS pinned near 1 GB (your mem_limit) is consistent with this.

Two things to check and adjust:

  • Where does the SQLite file physically live? Your compose bind-mounts ./data/open-webui. If ./data is on NFS or any network/remote storage, that is unsupported by Open WebUI and by SQLite itself (its file locking is unreliable there) and it makes this much worse: slow I/O keeps connections open longer, so the pool fills faster. It must be on local, directly-attached SSD/NVMe.
  • The defaults assume more headroom than 1024 MB. For a constrained container, set:
    • DATABASE_POOL_SIZE=8
    • DATABASE_SQLITE_PRAGMA_CACHE_SIZE=-2000 (≈ 2 MB per connection instead of 64 MB)
    • DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0
    • and raise mem_limit if you can; 1 GB is very low for a deployment.

Making the SQLite pool and per-connection cache defaults more conservative is on our radar, but the immediate resolution is the storage check plus the values above. Can you confirm where ./data is stored, and whether it still reproduces with those settings - if not we can change defaults?

<!-- gh-comment-id:4459139039 --> @Classic298 commented on GitHub (May 15, 2026): Thanks This is not a leak inside aiosqlite, and it is unrelated to Qdrant. The 3.5 GB matching your Qdrant size is coincidental: aiosqlite is the local SQLite driver, Qdrant is a separate remote service, they share no memory path. What you are hitting is the memory profile of the async SQLite layer introduced in v0.9.0 (you are right that v0.8.12 predates it and is unaffected), running under a 1024 MB cap: On SQLite, with DATABASE_POOL_SIZE unset the engine falls back to a pool of up to 512 connections. Each connection lazily grows its own SQLite page cache up to ~64 MB (PRAGMA cache_size=-65536) plus a 256 MB mmap window (PRAGMA mmap_size). These are per-connection, not shared. Editing permissions and then reloading the model list opens many connections (more models means more connections), so peak memory scales with how many are active at once, each holding its own cache. That is exactly why memray attributes almost everything to aiosqlite/core.py: that frame is where each connection materialises rows and holds its cache. It is the signature of many connections, not a bug in aiosqlite. Your total-vm: 8.4 GB with RSS pinned near 1 GB (your mem_limit) is consistent with this. Two things to check and adjust: - Where does the SQLite file physically live? Your compose bind-mounts ./data/open-webui. If ./data is on NFS or any network/remote storage, that is unsupported by Open WebUI and by SQLite itself (its file locking is unreliable there) and it makes this much worse: slow I/O keeps connections open longer, so the pool fills faster. It must be on local, directly-attached SSD/NVMe. - The defaults assume more headroom than 1024 MB. For a constrained container, set: - DATABASE_POOL_SIZE=8 - DATABASE_SQLITE_PRAGMA_CACHE_SIZE=-2000 (≈ 2 MB per connection instead of 64 MB) - DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0 - and raise mem_limit if you can; 1 GB is very low for a deployment. Making the SQLite pool and per-connection cache defaults more conservative is on our radar, but the immediate resolution is the storage check plus the values above. Can you confirm where ./data is stored, and whether it still reproduces with those settings - if not we can change defaults?
GiteaMirror commented 2026-05-21 03:10:09 -05:00
Author
Owner
Copy Link

@jmc0x68 commented on GitHub (May 15, 2026):

I tried adding the following environment variables:

DATABASE_POOL_SIZE=8
DATABASE_SQLITE_PRAGMA_CACHE_SIZE=-2000  # ~2 MB per connection instead of 64 MB
DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0

and also increased mem_limit to 1200M. The memory situation improved significantly. Thank you very much for the quick and helpful response!

<!-- gh-comment-id:4460654021 --> @jmc0x68 commented on GitHub (May 15, 2026): I tried adding the following environment variables: ``` DATABASE_POOL_SIZE=8 DATABASE_SQLITE_PRAGMA_CACHE_SIZE=-2000 # ~2 MB per connection instead of 64 MB DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0 ``` and also increased mem_limit to 1200M. The memory situation improved significantly. Thank you very much for the quick and helpful response!
GiteaMirror commented 2026-05-21 03:10:09 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (May 15, 2026):

thanks for confirming this helped in your case. we will add this to the docs

<!-- gh-comment-id:4461242993 --> @Classic298 commented on GitHub (May 15, 2026): thanks for confirming this helped in your case. we will add this to the docs
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#123706
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?