github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-08-30 17:25:30 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

OpenAI Embedding Batch Size not working #2196

New Issue
Closed
opened 2025-11-11 15:02:10 -06:00 by GiteaMirror · 9 comments
GiteaMirror commented 2025-11-11 15:02:10 -06:00
Owner
Copy Link
Copy Source

Originally created by @sp301415 on GitHub (Sep 25, 2024).

Bug Report

Installation Method

Custom Install

Environment

  • Open WebUI Version: [v0.3.22]

  • Operating System: [Ubuntu 22.04]

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

Batching queries to OpenAI compatible server working.

Actual Behavior:

Seperate queries get sent, no batching ocurrs.

Description

Regardless of embedding batch size of OpenAI endpoint (RAG_EMBEDDING_OPENAI_BATCH_SIZE), no batch queries are sent. This significantly slows down RAG for OpenAI endpoints.

A Quick git bisect shows commit 522afbb0a0 started this problem.

I also think this is the root cause of #5637.

Reproduction Details

Steps to Reproduce:

  1. Set OpenAI compatible embedding server.
  2. Set Batch Size to > 1.
  3. Upload PDF.
Originally created by @sp301415 on GitHub (Sep 25, 2024). # Bug Report ## Installation Method Custom Install ## Environment - **Open WebUI Version:** [v0.3.22] - **Operating System:** [Ubuntu 22.04] **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [ ] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: Batching queries to OpenAI compatible server working. ## Actual Behavior: Seperate queries get sent, no batching ocurrs. ## Description Regardless of embedding batch size of OpenAI endpoint (`RAG_EMBEDDING_OPENAI_BATCH_SIZE`), no batch queries are sent. This significantly slows down RAG for OpenAI endpoints. A Quick git bisect shows commit 522afbb0a00aa3c688fe5560a48b53cff8f110c3 started this problem. I also think this is the root cause of #5637. ## Reproduction Details **Steps to Reproduce:** 1. Set OpenAI compatible embedding server. 2. Set Batch Size to > 1. 3. Upload PDF.
GiteaMirror commented 2025-11-11 15:02:10 -06:00
Author
Owner
Copy Link
Copy Source

@sp301415 commented on GitHub (Sep 25, 2024):

Hmm, upon further inspection, I think the issue is in this line: https://github.com/open-webui/open-webui/blob/82cda6e52204f621882df696a4a26cc20ab482a0/backend/open_webui/apps/rag/main.py#L1115-L1126

Currently, something like [embedding_function(text[0]), embedding_function(text[1]), ... is passed to VECTOR_DB_CLIENT.insert, while it should be embedding_function[text[0], text[1], ...]. It seems to be working if it is fixed as

        embedding_texts = embedding_function(
            list(map(lambda x: x.replace("\n", " "), texts))
        )
        VECTOR_DB_CLIENT.insert(
            collection_name=collection_name,
            items=[
                {
                    "id": str(uuid.uuid4()),
                    "text": text,
                    "vector": embedding_texts[idx],
                    "metadata": metadatas[idx],
                }
                for idx, text in enumerate(texts)
            ],
        )

But I'm not really grasping all of the logic here. @tjbck Can you confirm this fix is enough? Thanks!

@sp301415 commented on GitHub (Sep 25, 2024): Hmm, upon further inspection, I think the issue is in this line: https://github.com/open-webui/open-webui/blob/82cda6e52204f621882df696a4a26cc20ab482a0/backend/open_webui/apps/rag/main.py#L1115-L1126 Currently, something like `[embedding_function(text[0]), embedding_function(text[1]), ...` is passed to `VECTOR_DB_CLIENT.insert`, while it should be `embedding_function[text[0], text[1], ...]`. It seems to be working if it is fixed as ```py embedding_texts = embedding_function( list(map(lambda x: x.replace("\n", " "), texts)) ) VECTOR_DB_CLIENT.insert( collection_name=collection_name, items=[ { "id": str(uuid.uuid4()), "text": text, "vector": embedding_texts[idx], "metadata": metadatas[idx], } for idx, text in enumerate(texts) ], ) ``` But I'm not really grasping all of the logic here. @tjbck Can you confirm this fix is enough? Thanks!
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@tjbck commented on GitHub (Sep 26, 2024):

Looks reasonable to me, were you able to test the code yourself?

@tjbck commented on GitHub (Sep 26, 2024): Looks reasonable to me, were you able to test the code yourself?
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@sp301415 commented on GitHub (Sep 26, 2024):

@tjbck Yes, I patched it myself and it seems like there is no issue right now. I'll test it a bit more and if it's fine, then I'll create a PR :)

@sp301415 commented on GitHub (Sep 26, 2024): @tjbck Yes, I patched it myself and it seems like there is no issue right now. I'll test it a bit more and if it's fine, then I'll create a PR :)
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@ajikmr commented on GitHub (Sep 26, 2024):

Please check the same for ollama batch embedding as well.

@ajikmr commented on GitHub (Sep 26, 2024): Please check the same for ollama batch embedding as well.
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@sp301415 commented on GitHub (Sep 26, 2024):

@ajikmr I don’t think open-webui supports batch embedding for ollama.

@sp301415 commented on GitHub (Sep 26, 2024): @ajikmr I don’t think open-webui supports batch embedding for ollama.
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@bgeneto commented on GitHub (Sep 26, 2024):

@sp301415 You saved us! Totally related to #5637. No more need to rate limit embeddings api calls! Let's dive in...
I think you've tested only with Milvus, right? Because Chroma client has no create_collection method. I've got the following error while testing:

    VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChromaClient' object has no attribute 'create_collection'

I know nothing about Milvus, what's the advantage?
Anyway, it follows the code that worked for me (note the commented line):

        # apps/rag/main.py +1115
        embedding_texts = embedding_function(
            list(map(lambda x: x.replace("\n", " "), texts))
        )
        #VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)
        VECTOR_DB_CLIENT.insert(
            collection_name=collection_name,
            items=[
                {
                    "id": str(uuid.uuid4()),
                    "text": text,
                    "vector": embedding_texts[idx],
                    "metadata": metadatas[idx],
                }
                for idx, text in enumerate(texts)
            ],
        )

I've reseted the vector database and removed the contents of data/vector_db/ on every test I've done. Best results with OpenAI compatible API was with jina-embeddings-v3 and mxbai-rerank-xsmall-v1 reranker:

image

Best results with Ollama engine was with mxbai-embed-large and cross-encoder/ms-marco-MiniLM-L-6-v2. I don't have a GPU on my Open WebUI docker container, so I need to use a fast (and small) reranker that works until we can also use external API for reranking...

image

Important: We still require a means to alert the user when a document embed fails by displaying a push notification. Logs aren't reliable enough for this. The document will be there, but it will have no use.

@bgeneto commented on GitHub (Sep 26, 2024): @sp301415 You saved us! Totally related to #5637. No more need to rate limit embeddings api calls! Let's dive in... I think you've tested only with Milvus, right? Because Chroma client has no `create_collection` method. I've got the following error while testing: ``` VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'ChromaClient' object has no attribute 'create_collection' ``` I know nothing about Milvus, what's the advantage? Anyway, it follows the code that worked for me (note the commented line): ```python # apps/rag/main.py +1115 embedding_texts = embedding_function( list(map(lambda x: x.replace("\n", " "), texts)) ) #VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) VECTOR_DB_CLIENT.insert( collection_name=collection_name, items=[ { "id": str(uuid.uuid4()), "text": text, "vector": embedding_texts[idx], "metadata": metadatas[idx], } for idx, text in enumerate(texts) ], ) ``` I've reseted the vector database and removed the contents of `data/vector_db/` on every test I've done. Best results with OpenAI compatible API was with `jina-embeddings-v3` and `mxbai-rerank-xsmall-v1` reranker: ![image](https://github.com/user-attachments/assets/471c2d57-90bb-4f7e-a6ff-1683e3344f25) Best results with Ollama engine was with `mxbai-embed-large` and `cross-encoder/ms-marco-MiniLM-L-6-v2`. I don't have a GPU on my Open WebUI docker container, so I need to use a fast (and small) reranker that works until we can also use **external API for reranking**... ![image](https://github.com/user-attachments/assets/8339126d-9030-41bf-b607-ddf41dcc1a6c) **Important**: We still require a means to alert the user when a document embed fails by displaying a push notification. Logs aren't reliable enough for this. The document will be there, but it will have no use.
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@sp301415 commented on GitHub (Sep 26, 2024):

@bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.

@sp301415 commented on GitHub (Sep 26, 2024): @bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.
GiteaMirror commented 2025-11-11 15:02:11 -06:00
Author
Owner
Copy Link
Copy Source

@bgeneto commented on GitHub (Sep 26, 2024):

@bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version.

I'm using the dev branch.

@bgeneto commented on GitHub (Sep 26, 2024): > @bgeneto That's great to hear! I'm also using ChromaDB, may I ask what version of open-webui are you using? I've seen that ChromaDB error in previous versions, but I believe it is gone in the latest version. I'm using the dev branch.
GiteaMirror commented 2025-11-11 15:02:12 -06:00
Author
Owner
Copy Link
Copy Source

@sp301415 commented on GitHub (Sep 26, 2024):

@bgeneto Intresting, it seems like that part of the code is gone after this commit: 4775fe43d8. Can you check your local repository is up-to-date?

Edit Oops, sorry, now I understand. The line

VECTOR_DB_CLIENT.create_collection(collection_name=collection_name)

in my first comment was definitely a mistake. I was testing with an old version, and it slipped into my comment. I'll remove it!

@sp301415 commented on GitHub (Sep 26, 2024): @bgeneto Intresting, it seems like that part of the code is gone after this commit: 4775fe43d8d7e1088e9e3730a82e2caf911622be. Can you check your local repository is up-to-date? **Edit** Oops, sorry, now I understand. The line ```py VECTOR_DB_CLIENT.create_collection(collection_name=collection_name) ``` in my first comment was definitely a mistake. I was testing with an old version, and it slipped into my comment. I'll remove it!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#2196