[GH-ISSUE #23390] issue: Errors in embedding function with some models & transformers>=5.0 #123278

Closed
opened 2026-05-21 02:33:51 -05:00 by GiteaMirror · 1 comment
Owner

Originally created by @rgaricano on GitHub (Apr 3, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23390

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Git Clone

Open WebUI Version

0.8.12

Ollama Version (if applicable)

No response

Operating System

ubuntu 24.04

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

.

Actual Behavior

.

Steps to Reproduce

.

Logs & Screenshots

.

Additional Information

More than an issue this is an advise of incompatibility of some embedd models and transformers>=5.0.

In my case the error arise using 'Alibaba-NLP/gte-multilingual-base` and hybrid search, giving me this kind of errors:

... embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION(
                                                               │         │           │                 └ <property object at 0x7cb7b1c588b0>
                                                               │         │           └ <starlette.requests.Request object at 0x7cb5c0820260> 
                                                               │         └ None
                                                               └ ['extrae los datos de las analiticas']
   TypeError: 'NoneType' object is not callable

also when trying to update the embedd model (clicking in the download arrow):

     token_type_ids = position_ids.mul(0)
                     │            └ <method 'mul' of 'torch._C.TensorBase' objects>
                      └ <unprintable Tensor object>
 RuntimeError: CUDA error: device-side assert triggered

This happend because Transformers v5 now creates models on the meta device (shape-only, no real memory) before loading checkpoint weights. Non-persistent buffers like position_ids aren't in the checkpoint, so they never get real values, ends up containing uninitialized garbage data, causing IndexError crashes or silently wrong embeddings.

Models that don't re-initialize those buffers in the _init_weights method script have that problem (this wasn't needed before v5).

In my case, instead of simply changing the model, I patched it because, right now, I won't change the embedd model and have to re-embedd the vector database.

FIX:
In backend/open_webui/routers/retrieval.py

def get_ef(
    engine: str,
    embedding_model: str,
    auto_update: bool = RAG_EMBEDDING_MODEL_AUTO_UPDATE,
):
    ef = None
    if embedding_model and engine == "":
        from sentence_transformers import SentenceTransformer
        import torch

        try:
            ef = SentenceTransformer(
                get_model_path(embedding_model, auto_update),
                device=DEVICE_TYPE,
                trust_remote_code=RAG_EMBEDDING_MODEL_TRUST_REMOTE_CODE,
                backend=SENTENCE_TRANSFORMERS_BACKEND,
                model_kwargs=SENTENCE_TRANSFORMERS_MODEL_KWARGS,
            )

            # Fix for Alibaba-NLP/gte-multilingual-base with transformers>=5.0
            if "Alibaba-NLP/gte-multilingual-base" in embedding_model:
                model = ef._first_module().auto_model
                model.eval()
                embeddings = model.embeddings
                max_pos = embeddings.position_ids.size(0)
                embeddings.register_buffer("position_ids", torch.arange(max_pos), persistent=True)
                log.debug(f"Applied position_ids fix for {embedding_model}")

        except Exception as e:
            log.debug(f"Error loading SentenceTransformer: {e}")

    return ef

In backend/open_webui/main.py (where the embedding function is initialized)

try:
    app.state.ef = get_ef(
        app.state.config.RAG_EMBEDDING_ENGINE, app.state.config.RAG_EMBEDDING_MODEL
    )

    # Apply fix for Alibaba-NLP/gte-multilingual-base if needed
    if (app.state.ef and 
        "Alibaba-NLP/gte-multilingual-base" in app.state.config.RAG_EMBEDDING_MODEL):
        import torch
        model = app.state.ef._first_module().auto_model
        model.eval()
        embeddings = model.embeddings
        max_pos = embeddings.position_ids.size(0)
        embeddings.register_buffer("position_ids", torch.arange(max_pos), persistent=True)
        log.debug(f"Applied position_ids fix for {app.state.config.RAG_EMBEDDING_MODEL}")

Note:
I have a couple of possible generic solutions but as this it's not a Open-WebUI bug and the solution can be problematic in some scenaries maybe is better just to know and only patch it on own development if neccessary.

I hope this can be helpful if you have similar issues.

Originally created by @rgaricano on GitHub (Apr 3, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23390 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Git Clone ### Open WebUI Version 0.8.12 ### Ollama Version (if applicable) _No response_ ### Operating System ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior . ### Actual Behavior . ### Steps to Reproduce . ### Logs & Screenshots . ### Additional Information More than an issue this is an advise of incompatibility of some embedd models and `transformers>=5.0.` In my case the error arise using 'Alibaba-NLP/gte-multilingual-base` and hybrid search, giving me this kind of errors: ``` ... embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION( │ │ │ └ <property object at 0x7cb7b1c588b0> │ │ └ <starlette.requests.Request object at 0x7cb5c0820260> │ └ None └ ['extrae los datos de las analiticas'] TypeError: 'NoneType' object is not callable ``` also when trying to update the embedd model (clicking in the download arrow): ``` token_type_ids = position_ids.mul(0) │ └ <method 'mul' of 'torch._C.TensorBase' objects> └ <unprintable Tensor object> RuntimeError: CUDA error: device-side assert triggered ``` This happend because Transformers v5 now creates models on the meta device (shape-only, no real memory) before loading checkpoint weights. Non-persistent buffers like` position_ids` aren't in the checkpoint, so they never get real values, ends up containing uninitialized garbage data, causing IndexError crashes or silently wrong embeddings. Models that don't re-initialize those buffers in the _init_weights method script have that problem (this wasn't needed before v5). In my case, instead of simply changing the model, I patched it because, right now, I won't change the embedd model and have to re-embedd the vector database. FIX: In `backend/open_webui/routers/retrieval.py` ``` def get_ef( engine: str, embedding_model: str, auto_update: bool = RAG_EMBEDDING_MODEL_AUTO_UPDATE, ): ef = None if embedding_model and engine == "": from sentence_transformers import SentenceTransformer import torch try: ef = SentenceTransformer( get_model_path(embedding_model, auto_update), device=DEVICE_TYPE, trust_remote_code=RAG_EMBEDDING_MODEL_TRUST_REMOTE_CODE, backend=SENTENCE_TRANSFORMERS_BACKEND, model_kwargs=SENTENCE_TRANSFORMERS_MODEL_KWARGS, ) # Fix for Alibaba-NLP/gte-multilingual-base with transformers>=5.0 if "Alibaba-NLP/gte-multilingual-base" in embedding_model: model = ef._first_module().auto_model model.eval() embeddings = model.embeddings max_pos = embeddings.position_ids.size(0) embeddings.register_buffer("position_ids", torch.arange(max_pos), persistent=True) log.debug(f"Applied position_ids fix for {embedding_model}") except Exception as e: log.debug(f"Error loading SentenceTransformer: {e}") return ef ``` In `backend/open_webui/main.py` (where the embedding function is initialized) ``` try: app.state.ef = get_ef( app.state.config.RAG_EMBEDDING_ENGINE, app.state.config.RAG_EMBEDDING_MODEL ) # Apply fix for Alibaba-NLP/gte-multilingual-base if needed if (app.state.ef and "Alibaba-NLP/gte-multilingual-base" in app.state.config.RAG_EMBEDDING_MODEL): import torch model = app.state.ef._first_module().auto_model model.eval() embeddings = model.embeddings max_pos = embeddings.position_ids.size(0) embeddings.register_buffer("position_ids", torch.arange(max_pos), persistent=True) log.debug(f"Applied position_ids fix for {app.state.config.RAG_EMBEDDING_MODEL}") ``` Note: I have a couple of possible generic solutions but as this it's not a Open-WebUI bug and the solution can be problematic in some scenaries maybe is better just to know and only patch it on own development if neccessary. I hope this can be helpful if you have similar issues.
GiteaMirror added the bug label 2026-05-21 02:33:51 -05:00
Author
Owner

@Classic298 commented on GitHub (Apr 15, 2026):

Hey @rgaricano

The dependency was updated on dev (transformers 5.5.4, sentence-transformers 5.4.0), but no model-specific workaround was added to Open WebUI itself. The get_ef function still does a straightforward SentenceTransformer load without any position_ids patching.

This means users of Alibaba-NLP/gte-multilingual-base (or any other model that registers position_ids as a non-persistent buffer without re-initializing it in _init_weights) will still hit this issue with transformers >= 5.0.

That said, this is ultimately a model-level bug. The model's own modeling.py needs to properly handle meta-device initialization. Adding hardcoded model-specific patches in Open WebUI wouldn't be a sustainable path forward. The proper fix should come from the model authors on HuggingFace - until then that model is not compatible and deprecated since it is not compatible with newer versions of the transformers library.

In the meantime, if you're stuck on this model and can't re-embed, pinning transformers < 5.0 in your environment is probably the most reliable workaround.

<!-- gh-comment-id:4254925250 --> @Classic298 commented on GitHub (Apr 15, 2026): Hey @rgaricano The dependency was updated on dev (transformers 5.5.4, sentence-transformers 5.4.0), but no model-specific workaround was added to Open WebUI itself. The get_ef function still does a straightforward SentenceTransformer load without any position_ids patching. This means users of Alibaba-NLP/gte-multilingual-base (or any other model that registers position_ids as a non-persistent buffer without re-initializing it in _init_weights) will still hit this issue with transformers >= 5.0. That said, this is ultimately a model-level bug. The model's own modeling.py needs to properly handle meta-device initialization. Adding hardcoded model-specific patches in Open WebUI wouldn't be a sustainable path forward. The proper fix should come from the model authors on HuggingFace - until then that model is not compatible and deprecated since it is not compatible with newer versions of the transformers library. In the meantime, if you're stuck on this model and can't re-embed, pinning transformers < 5.0 in your environment is probably the most reliable workaround.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#123278