github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 14:39:31 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #5698] OpenAI Embedding Batch Size not working #133817

New Issue
Closed
opened 2026-05-24 23:15:36 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-05-24 23:15:36 -05:00
Owner
Copy Link

Originally created by @sp301415 on GitHub (Sep 25, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/5698

Bug Report

Installation Method

Custom Install

Environment

  • Open WebUI Version: [v0.3.22]

  • Operating System: [Ubuntu 22.04]

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

Batching queries to OpenAI compatible server working.

Actual Behavior:

Seperate queries get sent, no batching ocurrs.

Description

Regardless of embedding batch size of OpenAI endpoint (RAG_EMBEDDING_OPENAI_BATCH_SIZE), no batch queries are sent. This significantly slows down RAG for OpenAI endpoints.

A Quick git bisect shows commit 522afbb0a0 started this problem.

I also think this is the root cause of #5637.

Reproduction Details

Steps to Reproduce:

  1. Set OpenAI compatible embedding server.
  2. Set Batch Size to > 1.
  3. Upload PDF.
Originally created by @sp301415 on GitHub (Sep 25, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/5698 # Bug Report ## Installation Method Custom Install ## Environment - **Open WebUI Version:** [v0.3.22] - **Operating System:** [Ubuntu 22.04] **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [ ] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: Batching queries to OpenAI compatible server working. ## Actual Behavior: Seperate queries get sent, no batching ocurrs. ## Description Regardless of embedding batch size of OpenAI endpoint (`RAG_EMBEDDING_OPENAI_BATCH_SIZE`), no batch queries are sent. This significantly slows down RAG for OpenAI endpoints. A Quick git bisect shows commit 522afbb0a00aa3c688fe5560a48b53cff8f110c3 started this problem. I also think this is the root cause of #5637. ## Reproduction Details **Steps to Reproduce:** 1. Set OpenAI compatible embedding server. 2. Set Batch Size to > 1. 3. Upload PDF.
GiteaMirror commented 2026-05-24 23:15:44 -05:00
Author
Owner
Copy Link

@sp301415 commented on GitHub (Sep 25, 2024):

Hmm, upon further inspection, I think the issue is in this line: 82cda6e522/backend/open_webui/apps/rag/main.py (L1115-L1126)

Currently, something like [embedding_function(text[0]), embedding_function(text[1]), ... is passed to VECTOR_DB_CLIENT.insert, while it should be embedding_function[text[0], text[1], ...]. It seems to be working if it is fixed as

        embedding_texts = embedding_function(
            list(map(lambda x: x.replace("\n", " "), texts))
        )
        VECTOR_DB_CLIENT.insert(
            collection_name=collection_name,
            items=[
                {
                    "id": str(uuid.uuid4()),
                    "text": text,
                    "vector": embedding_texts[idx],
                    "metadata": metadatas[idx],
                }
                for idx, text in enumerate(texts)
            ],
        )

But I'm not really grasping all of the logic here. @tjbck Can you confirm this fix is enough? Thanks!

<!-- gh-comment-id:2374258602 --> @sp301415 commented on GitHub (Sep 25, 2024): Hmm, upon further inspection, I think the issue is in this line: https://github.com/open-webui/open-webui/blob/82cda6e52204f621882df696a4a26cc20ab482a0/backend/open_webui/apps/rag/main.py#L1115-L1126 Currently, something like `[embedding_function(text[0]), embedding_function(text[1]), ...` is passed to `VECTOR_DB_CLIENT.insert`, while it should be `embedding_function[text[0], text[1], ...]`. It seems to be working if it is fixed as ```py embedding_texts = embedding_function( list(map(lambda x: x.replace("\n", " "), texts)) ) VECTOR_DB_CLIENT.insert( collection_name=collection_name, items=[ { "id": str(uuid.uuid4()), "text": text, "vector": embedding_texts[idx], "metadata": metadatas[idx], } for idx, text in enumerate(texts) ], ) ``` But I'm not really grasping all of the logic here. @tjbck Can you confirm this fix is enough? Thanks!
GiteaMirror commented 2026-05-24 23:15:51 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Sep 26, 2024):

Looks reasonable to me, were you able to test the code yourself?

<!-- gh-comment-id:2375558112 --> @tjbck commented on GitHub (Sep 26, 2024): Looks reasonable to me, were you able to test the code yourself?
GiteaMirror commented 2026-05-24 23:15:56 -05:00
Author
Owner
Copy Link

@sp301415 commented on GitHub (Sep 26, 2024):

@tjbck Yes, I patched it myself and it seems like there is no issue right now. I'll test it a bit more and if it's fine, then I'll create a PR :)

<!-- gh-comment-id:2375638434 --> @sp301415 commented on GitHub (Sep 26, 2024): @tjbck Yes, I patched it myself and it seems like there is no issue right now. I'll test it a bit more and if it's fine, then I'll create a PR :)
GiteaMirror commented 2026-05-24 23:15:57 -05:00
Author
Owner
Copy Link

@ajikmr commented on GitHub (Sep 26, 2024):

Please check the same for ollama batch embedding as well.

<!-- gh-comment-id:2375806041 --> @ajikmr commented on GitHub (Sep 26, 2024): Please check the same for ollama batch embedding as well.
GiteaMirror commented 2026-05-24 23:15:59 -05:00
Author
Owner
Copy Link

@sp301415 commented on GitHub (Sep 26, 2024):

@ajikmr I don’t think open-webui supports batch embedding for ollama.

<!-- gh-comment-id:2375838895 --> @sp301415 commented on GitHub (Sep 26, 2024): @ajikmr I don’t think open-webui supports batch embedding for ollama.
GiteaMirror commented 2026-05-24 23:16:00 -05:00
Author
Owner
Copy Link

@bgeneto commented on GitHub (Sep 26, 2024):

@sp301415 You saved us! Totally related to #5637. No more need to rate limit embeddings api calls! Let's dive in...
I think you've tested only with Milvus, right? Because Chroma client has no create_collection method. I've got the following error while testing:

    VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChromaClient' object has no attribute 'create_collection'

I know nothing about Milvus, what's the advantage?
Anyway, it follows the code that worked for me (note the commented line):

        # apps/rag/main.py +1115
        embedding_texts = embedding_function(
            list(map(lambda x: x.replace("\n", " "), texts))
        )
        #VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)
        VECTOR_DB_CLIENT.insert(
            collection_name=collection_name,
            items=[
                {
                    "id": str(uuid.uuid4()),
                    "text": text,
                    "vector": embedding_texts[idx],
                    "metadata": metadatas[idx],
                }
                for idx, text in enumerate(texts)
            ],
        )

I've reseted the vector database and removed the contents of data/vector_db/ on every test I've done. Best results with OpenAI compatible API was with jina-embeddings-v3 and mxbai-rerank-xsmall-v1 reranker:

image

Best results with Ollama engine was with mxbai-embed-large and cross-encoder/ms-marco-MiniLM-L-6-v2. I don't have a GPU on my Open WebUI docker container, so I need to use a fast (and small) reranker that works until we can also use external API for reranking...

image

Important: We still require a means to alert the user when a document embed fails by displaying a push notification. Logs aren't reliable enough for this. The document will be there, but it will have no use.

<!-- gh-comment-id:2375874217 --> @bgeneto commented on GitHub (Sep 26, 2024): @sp301415 You saved us! Totally related to #5637. No more need to rate limit embeddings api calls! Let's dive in... I think you've tested only with Milvus, right? Because Chroma client has no `create_collection` method. I've got the following error while testing: ``` VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'ChromaClient' object has no attribute 'create_collection' ``` I know nothing about Milvus, what's the advantage? Anyway, it follows the code that worked for me (note the commented line): ```python # apps/rag/main.py +1115 embedding_texts = embedding_function( list(map(lambda x: x.replace("\n", " "), texts)) ) #VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) VECTOR_DB_CLIENT.insert( collection_name=collection_name, items=[ { "id": str(uuid.uuid4()), "text": text, "vector": embedding_texts[idx], "metadata": metadatas[idx], } for idx, text in enumerate(texts) ], ) ``` I've reseted the vector database and removed the contents of `data/vector_db/` on every test I've done. Best results with OpenAI compatible API was with `jina-embeddings-v3` and `mxbai-rerank-xsmall-v1` reranker: ![image](https://github.com/user-attachments/assets/471c2d57-90bb-4f7e-a6ff-1683e3344f25) Best results with Ollama engine was with `mxbai-embed-large` and `cross-encoder/ms-marco-MiniLM-L-6-v2`. I don't have a GPU on my Open WebUI docker container, so I need to use a fast (and small) reranker that works until we can also use **external API for reranking**... ![image](https://github.com/user-attachments/assets/8339126d-9030-41bf-b607-ddf41dcc1a6c) **Important**: We still require a means to alert the user when a document embed fails by displaying a push notification. Logs aren't reliable enough for this. The document will be there, but it will have no use.
GiteaMirror commented 2026-05-24 23:16:01 -05:00
Author
Owner
Copy Link

@sp301415 commented on GitHub (Sep 26, 2024):

@bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.

<!-- gh-comment-id:2375993475 --> @sp301415 commented on GitHub (Sep 26, 2024): @bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.
GiteaMirror commented 2026-05-24 23:16:02 -05:00
Author
Owner
Copy Link

@bgeneto commented on GitHub (Sep 26, 2024):

@bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.

I'm using the dev branch.

<!-- gh-comment-id:2376509932 --> @bgeneto commented on GitHub (Sep 26, 2024): > @bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version. I'm using the dev branch.
GiteaMirror commented 2026-05-24 23:16:03 -05:00
Author
Owner
Copy Link

@sp301415 commented on GitHub (Sep 26, 2024):

@bgeneto Intresting, it seems like that part of the code is gone after this commit: 4775fe43d8. Can you check your local repository is up-to-date?

Edit Oops, sorry, now I understand. The line

VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)

in my first comment was definitely a mistake. I was testing with an old version, and it slipped into my comment. I'll remove it!

<!-- gh-comment-id:2376684970 --> @sp301415 commented on GitHub (Sep 26, 2024): @bgeneto Intresting, it seems like that part of the code is gone after this commit: 4775fe43d8d7e1088e9e3730a82e2caf911622be. Can you check your local repository is up-to-date? **Edit** Oops, sorry, now I understand. The line ```py VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) ``` in my first comment was definitely a mistake. I was testing with an old version, and it slipped into my comment. I'll remove it!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#133817
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?