[GH-ISSUE #21315] issue: Title: [BUG/FIX] IndexError in save_docs_to_vector_db and lack of retry logic for embedding generation #106436

New Issue

GiteaMirror · 2026-05-18T04:47:15-05:00

GiteaMirror commented

2026-05-18 04:47:15 -05:00

Originally created by @DaMccRee on GitHub (Feb 12, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/21315

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

Open WebUI Version: Latest / Dev 0.7.2

Ollama Version (if applicable)

ollama latest

Operating System

Ubuntu 22.04

Browser (if applicable)

chrome latest

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

The system should automatically retry embedding generation when encountering transient service errors (e.g., HTTP 503 or network timeouts).
If embedding fails after all retries, the system should catch the error and provide a meaningful message to the user/logs instead of crashing with an IndexError.
All processed text chunks should have corresponding embeddings before attempting to save to the vector database.

Actual Behavior

When processing large files or under high load, open-webui can encounter an IndexError: list index out of range during the embedding saving process. This occurs because the embedding generation for certain batches might fail (e.g., due to a 503 Service Unavailable from Ollama or network timeouts), and the current implementation silently continues with a mismatched number of embeddings and text chunks.
Additionally, the asynchronous embedding functions lack a retry mechanism, making the RAG pipeline fragile when using external or local LLM services (like Ollama/OpenAI) that may experience transient failures.

Steps to Reproduce

Upload a large document that requires multiple batches for embedding.
Simulate a transient failure in the embedding engine (e.g., restart Ollama or trigger a rate limit).
The embedding function returns a list shorter than the input texts, leading to the IndexError when constructing the vector DB items.

Logs & Screenshots

File "/app/backend/open_webui/routers/retrieval.py", line 1750, in process_file
result = save_docs_to_vector_db(...)
File "/app/backend/open_webui/routers/retrieval.py", line 1555, in
"vector": embeddings[idx],
IndexError: list index out of range

Additional Information

Proposed Solution

1. Fix the Mismatch (IndexError Management)

In open_webui/routers/retrieval.py, a length check should be added before the list comprehension to ensure embeddings and texts are aligned. If a mismatch is detected, an explicit exception should be raised.

In open_webui/retrieval/utils.py -> get_embedding_function, ensure that if any batch fails to return a valid list, the entire process raises an exception instead of extending the list with None or skipping.

2. Implement Retry Logic with Exponential Backoff

The async embedding functions (agenerate_ollama_batch_embeddings, agenerate_openai_batch_embeddings, and agenerate_azure_openai_batch_embeddings) should be wrapped with a retry mechanism.

Key Changes Recommended:

Add a max_retries loop (e.g., 3-5 attempts).
Implement exponential backoff using asyncio.sleep between retries.
Specific handling for HTTP 503 (Service Unavailable) and HTTP 429 (Too Many Requests).

Implementation Example:

async def agenerate_ollama_batch_embeddings(...):
    max_retries = 3
    for attempt in range(max_retries):
        try:
            async with aiohttp.ClientSession(...) as session:
                async with session.post(...) as r:
                    if r.status == 503:
                        raise Exception("503 Service Unavailable")
                    r.raise_for_status()
                    data = await r.json()
                    return data["embeddings"]
        except Exception as e:
            if attempt < max_retries - 1:
                await asyncio.sleep(2 * (attempt + 1))
            else:
                log.exception(f"Failed after {max_retries} attempts")
                return None

Benefit

Robustness: The system can recover from transient network or service issues without failing the user's entire file upload.
Traceability: Instead of a generic IndexError, administrators can see "Failed to generate embeddings after X attempts" in the logs, pointing directly to the embedding engine issue.

Originally created by @DaMccRee on GitHub (Feb 12, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/21315 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version Open WebUI Version: Latest / Dev 0.7.2 ### Ollama Version (if applicable) ollama latest ### Operating System Ubuntu 22.04 ### Browser (if applicable) chrome latest ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior - The system should automatically retry embedding generation when encountering transient service errors (e.g., HTTP 503 or network timeouts). - If embedding fails after all retries, the system should catch the error and provide a meaningful message to the user/logs instead of crashing with an `IndexError`. - All processed text chunks should have corresponding embeddings before attempting to save to the vector database. ### Actual Behavior When processing large files or under high load, `open-webui` can encounter an `IndexError: list index out of range` during the embedding saving process. This occurs because the embedding generation for certain batches might fail (e.g., due to a 503 Service Unavailable from Ollama or network timeouts), and the current implementation silently continues with a mismatched number of embeddings and text chunks. Additionally, the asynchronous embedding functions lack a retry mechanism, making the RAG pipeline fragile when using external or local LLM services (like Ollama/OpenAI) that may experience transient failures. ### Steps to Reproduce 1. Upload a large document that requires multiple batches for embedding. 2. Simulate a transient failure in the embedding engine (e.g., restart Ollama or trigger a rate limit). 3. The embedding function returns a list shorter than the input `texts`, leading to the `IndexError` when constructing the vector DB items. ### Logs & Screenshots File "/app/backend/open_webui/routers/retrieval.py", line 1750, in process_file result = save_docs_to_vector_db(...) File "/app/backend/open_webui/routers/retrieval.py", line 1555, in <listcomp> "vector": embeddings[idx], IndexError: list index out of range ### Additional Information Proposed Solution ### 1. Fix the Mismatch (IndexError Management) In `open_webui/routers/retrieval.py`, a length check should be added before the list comprehension to ensure `embeddings` and `texts` are aligned. If a mismatch is detected, an explicit exception should be raised. In `open_webui/retrieval/utils.py` -> `get_embedding_function`, ensure that if any batch fails to return a valid list, the entire process raises an exception instead of extending the list with `None` or skipping. ### 2. Implement Retry Logic with Exponential Backoff The async embedding functions (`agenerate_ollama_batch_embeddings`, `agenerate_openai_batch_embeddings`, and `agenerate_azure_openai_batch_embeddings`) should be wrapped with a retry mechanism. **Key Changes Recommended:** - Add a `max_retries` loop (e.g., 3-5 attempts). - Implement exponential backoff using `asyncio.sleep` between retries. - Specific handling for HTTP 503 (Service Unavailable) and HTTP 429 (Too Many Requests). **Implementation Example:** ```python async def agenerate_ollama_batch_embeddings(...): max_retries = 3 for attempt in range(max_retries): try: async with aiohttp.ClientSession(...) as session: async with session.post(...) as r: if r.status == 503: raise Exception("503 Service Unavailable") r.raise_for_status() data = await r.json() return data["embeddings"] except Exception as e: if attempt < max_retries - 1: await asyncio.sleep(2 * (attempt + 1)) else: log.exception(f"Failed after {max_retries} attempts") return None ``` ## Benefit - **Robustness**: The system can recover from transient network or service issues without failing the user's entire file upload. - **Traceability**: Instead of a generic `IndexError`, administrators can see "Failed to generate embeddings after X attempts" in the logs, pointing directly to the embedding engine issue.

GiteaMirror added the bug label 2026-05-18 04:47:15 -05:00

GiteaMirror closed this issue

2026-05-18 04:47:16 -05:00

GiteaMirror commented

2026-05-18 04:47:17 -05:00

@DaMccRee commented on GitHub (Feb 12, 2026):

Technical Deep Dive & Validation (Supplemental)

Root Cause Analysis

The IndexError is a secondary symptom. The primary cause is that get_embedding_function (in retrieval/utils.py) processes embeddings in batches. Currently, if any batch fails (e.g., Ollama returning 503 during model loading), the system may continue with a mismatched number of embeddings versus text chunks.
Later, save_docs_to_vector_db iterates over the original texts length, but since the embeddings list is shorter, it triggers the IndexError.

Effectiveness of the Solution

Defensive Layer: Adds an explicit length check in retrieval.py to prevent the IndexError and provide a meaningful error message.
Resilience Layer: Introduces Exponential Backoff retries in utils.py. This ensures that transient network issues or service overloads (like Ollama loading a model) don't crash the entire document processing pipeline.

Validation Results

In testing, the system now successfully handles transient Ollama 503 errors:

DEBUG | agenerate_ollama_batch_embeddings: model=nomic-embed-text (attempt 1/3)
WARNING | Error generating ollama batch embeddings: 503 Service Unavailable
DEBUG | Retrying in 2 seconds...
DEBUG | agenerate_ollama_batch_embeddings: model=nomic-embed-text (attempt 2/3)
INFO | embeddings generated 4102 for 4102 items (Success after retry)

Readiness for Pull Request

The proposed changes have been verified in a production-like Docker environment. This fix maintains data consistency and significantly improves the robustness of the RAG pipeline.

@DaMccRee commented on GitHub (Feb 12, 2026): ## Technical Deep Dive & Validation (Supplemental) ### Root Cause Analysis The `IndexError` is a secondary symptom. The primary cause is that `get_embedding_function` (in `retrieval/utils.py`) processes embeddings in batches. Currently, if any batch fails (e.g., Ollama returning 503 during model loading), the system may continue with a mismatched number of embeddings versus text chunks. Later, `save_docs_to_vector_db` iterates over the original `texts` length, but since the `embeddings` list is shorter, it triggers the `IndexError`. ### Effectiveness of the Solution - **Defensive Layer**: Adds an explicit length check in `retrieval.py` to prevent the `IndexError` and provide a meaningful error message. - **Resilience Layer**: Introduces **Exponential Backoff** retries in `utils.py`. This ensures that transient network issues or service overloads (like Ollama loading a model) don't crash the entire document processing pipeline. ### Validation Results In testing, the system now successfully handles transient Ollama 503 errors: ```text DEBUG | agenerate_ollama_batch_embeddings: model=nomic-embed-text (attempt 1/3) WARNING | Error generating ollama batch embeddings: 503 Service Unavailable DEBUG | Retrying in 2 seconds... DEBUG | agenerate_ollama_batch_embeddings: model=nomic-embed-text (attempt 2/3) INFO | embeddings generated 4102 for 4102 items (Success after retry) ``` ### Readiness for Pull Request The proposed changes have been verified in a production-like Docker environment. This fix maintains data consistency and significantly improves the robustness of the RAG pipeline.

GiteaMirror commented

2026-05-18 04:47:17 -05:00

@DaMccRee commented on GitHub (Feb 13, 2026):

Ok
发自我的 iPhone

在 2026年2月13日，上午5:38，Tim Baek @.***> 写道：

Closed #21315 https://github.com/open-webui/open-webui/issues/21315 as not planned.

—
Reply to this email directly, view it on GitHubhttps://github.com/open-webui/open-webui/issues/21315#event-22748772503, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRS6UWYUGMDUO47HG4XVK34LTXFTAVCNFSM6AAAAACU2QEJBGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRSG42DQNZXGI2TAMY.
You are receiving this because you authored the thread.Message ID: @.***>

@DaMccRee commented on GitHub (Feb 13, 2026): Ok 发自我的 iPhone 在 2026年2月13日，上午5:38，Tim Baek ***@***.***> 写道： Closed #21315<https://github.com/open-webui/open-webui/issues/21315> as not planned. — Reply to this email directly, view it on GitHub<https://github.com/open-webui/open-webui/issues/21315#event-22748772503>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZRS6UWYUGMDUO47HG4XVK34LTXFTAVCNFSM6AAAAACU2QEJBGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRSG42DQNZXGI2TAMY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#106436