github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-11 10:34:13 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #19231] issue: Huge waste of disk space in vector_db #57482

New Issue
Closed
opened 2026-05-05 20:58:29 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-05-05 20:58:29 -05:00
Owner
Copy Link

Originally created by @andrew521 on GitHub (Nov 17, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/19231

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Pip Install

Open WebUI Version

v0.6.34

Ollama Version (if applicable)

No response

Operating System

wsl2 - ubuntu 24.04

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

122gb of storage for 4111 txt files, average size of 5kb. That is 6000 times more than the original documents.

Embedding Model : nomic-embed-text
Chunk Size : 1000
Chunk Overlap : 100

Actual Behavior

Each entry in vector_db includes a 32mb file, no matter how tini the original file is.

../venvs/webui-env/lib/python3.11/site-packages/open_webui/data/vector_db/*/data_level0.bin

Steps to Reproduce

ls - l ../venvs/webui-env/lib/python3.11/site-packages/open_webui/data/vector_db/*/data_level0.bin

Logs & Screenshots

(webui-env) ...user@RCT2PBK:~/sc_invest$ ls -l /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/*
-rw-rw-r-- 1 ...user ...user 32120000 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/data_level0.bin
-rw-rw-r-- 1 ...user ...user 100 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/header.bin
-rw-rw-r-- 1 ...user ...user 40000 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/length.bin
-rw-rw-r-- 1 ...user ...user 0 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/link_lists.bin

Additional Information

No response

Originally created by @andrew521 on GitHub (Nov 17, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/19231 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Pip Install ### Open WebUI Version v0.6.34 ### Ollama Version (if applicable) _No response_ ### Operating System wsl2 - ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior 122gb of storage for 4111 txt files, average size of 5kb. That is 6000 times more than the original documents. Embedding Model : nomic-embed-text Chunk Size : 1000 Chunk Overlap : 100 ### Actual Behavior Each entry in vector_db includes a 32mb file, no matter how tini the original file is. ../venvs/webui-env/lib/python3.11/site-packages/open_webui/data/vector_db/*/data_level0.bin ### Steps to Reproduce ls - l ../venvs/webui-env/lib/python3.11/site-packages/open_webui/data/vector_db/*/data_level0.bin ### Logs & Screenshots (webui-env) ...user@RCT2PBK:~/sc_invest$ ls -l /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/* -rw-rw-r-- 1 ...user ...user 32120000 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/data_level0.bin -rw-rw-r-- 1 ...user ...user 100 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/header.bin -rw-rw-r-- 1 ...user ...user 40000 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/length.bin -rw-rw-r-- 1 ...user ...user 0 Nov 12 19:42 /mnt/c/Data/AppData/open_webui/data/vector_db/031559de-3e63-4f46-8b67-4e78826de0c9/link_lists.bin ### Additional Information _No response_
GiteaMirror added the bug label 2026-05-05 20:58:29 -05:00
GiteaMirror commented 2026-05-05 20:58:31 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Nov 17, 2025):

We'll need a reliable way to reproduce your issue.

<!-- gh-comment-id:3543573231 --> @tjbck commented on GitHub (Nov 17, 2025): We'll need a reliable way to reproduce your issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#57482
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?