Bug: the default sbert model was trained on tweets and shouldn't be used for RAG #2047

New Issue

GiteaMirror · 2025-11-11T14:59:21-06:00

GiteaMirror commented

2025-11-11 14:59:21 -06:00

Originally created by @thiswillbeyourgithub on GitHub (Sep 10, 2024).

Bug Report

Expected Behavior:

The default value for RAG should make sense: the chunk size and overlap should be compatible with the default model.

Actual Behavior:

The default model is sentence-transformers/all-MiniLM-L6v2. Despite what its reputation seems to be, IMO it's a pretty terrible model for today's standard.

To quote from their hf page :

Our model is intended to be used as a sentence and short paragraph encoder.
The sequence length was limited to 128 tokens.
By default, input text longer than 256 word pieces is truncated.
Also it's english only.

I have been saying in multiple places now that this shouldn't be used by default ever, as it was trained on basically short tweets and silently clips any text above their limit.

Now there are several ways to go about this:

You can simply drastically reduce the default chunk size. I don't think it's a good idea but at least it makes sense for the model.
if you want to keep an english only model I can suggest using bge-small-en-v1.5. It's 130Mb, compared to the 90Mb of all-MiniLM-L6. I think it has 512 token limit (and a more compact tokenizer) but still an improvement. Otherwise there's also another one from jina.ai
If you want to switch to a decent multilingual model there's bge-m3 but it's 2Go, or jina's colbert model just out of the over that's 1Go. Each has 8192 token length support and more modern tokenizers.

FWIW my prefered suggestion would be to use a multilingual model (they also are the only one with that long context AFAIK), jina's colbert might be a sane default as it's not so big.

AFAIK all those models can be used by the sentence-transformers package or as hugging face embeddings. As you can see here for example

Jina also makes good rerankers so you could set one up by default (even if it's disabled by default).

PS: I you use sentence transformer models, you can add checks to lookup the maximum supported embedding size of a given model and warn the user. Example can be found at my old broken script

Originally created by @thiswillbeyourgithub on GitHub (Sep 10, 2024). # Bug Report ## Expected Behavior: The default value for RAG should make sense: the chunk size and overlap should be compatible with the default model. ## Actual Behavior: The default model is `sentence-transformers/all-MiniLM-L6v2`. Despite what its reputation seems to be, IMO it's a pretty terrible model for today's standard. To quote from [their hf page](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) : - `Our model is intended to be used as a sentence and short paragraph encoder.` - `The sequence length was limited to 128 tokens.` - `By default, input text longer than 256 word pieces is truncated.` - Also it's english only. [I have](https://github.com/simonw/llm-sentence-transformers/issues/8#issue-1880545810) [been saying](https://github.com/langchain-ai/langchain/issues/2442#issuecomment-1601119941) [in multiple places](https://github.com/langchain-ai/langchain/issues/2026#issuecomment-1601331113) now that this shouldn't be used by default ever, as it was trained on basically short tweets and **silently clips any text above their limit**. Now there are several ways to go about this: 1. You can simply drastically reduce the default chunk size. I don't think it's a good idea but at least it makes sense for the model. 2. if you want to keep an english only model I can suggest using [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5). It's 130Mb, compared to the 90Mb of all-MiniLM-L6. I think it has 512 token limit (and a more compact tokenizer) but still an improvement. Otherwise there's also [another one from jina.ai](https://huggingface.co/jinaai/jina-embeddings-v2-base-en) 3. If you want to switch to a decent multilingual model there's [bge-m3](https://huggingface.co/BAAI/bge-m3/tree/main) but it's 2Go, or [jina's colbert model just out of the over](https://huggingface.co/jinaai/jina-colbert-v2/tree/main) that's 1Go. Each has 8192 token length support and more modern tokenizers. FWIW my prefered suggestion would be to use a multilingual model (they also are the only one with that long context AFAIK), jina's colbert might be a sane default as it's not so big. AFAIK all those models can be used by the sentence-transformers package or as hugging face embeddings. As you can see [here for example](https://www.sbert.net/examples/applications/computing-embeddings/README.html?highlight=bge) Jina also makes good rerankers so you could set one up by default (even if it's disabled by default). PS: I you use sentence transformer models, you can add checks to lookup the maximum supported embedding size of a given model and warn the user. Example can be found at [my old broken script](https://github.com/simonw/llm-sentence-transformers/issues/8#issuecomment-1732618592)

GiteaMirror closed this issue

2025-11-11 14:59:21 -06:00

GiteaMirror referenced this issue

2025-11-11 17:32:54 -06:00

[PR #2047] [CLOSED] chore(deps): bump the python-packages group across 1 directory with 11 updates #7671

GiteaMirror referenced this issue

2026-04-20 03:14:17 -05:00

[PR #2047] [CLOSED] chore(deps): bump the python-packages group across 1 directory with 11 updates #20875

GiteaMirror referenced this issue

2026-04-25 10:20:52 -05:00

[PR #2047] [CLOSED] chore(deps): bump the python-packages group across 1 directory with 11 updates #36505

GiteaMirror referenced this issue