mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 06:03:26 -05:00
[GH-ISSUE #6209] Default chunk size too big for default embedding model? #134012
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#134012
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @lbluett on GitHub (Oct 16, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6209
Hi all, this may be due to my own ignorance but isn't the default chunk size too big for the default embedding model?
The default embedding model is: sentence-transformers/all-MiniLM-L6-v2
The default chunk size is: 1500
Isn't the max sequence length for all-MiniLM-L6-v2 256 tokens? According to: https://www.sbert.net/docs/sentence_transformer/pretrained_models.html
I believe this would result in the 1500 token chunks being truncuated to 256 tokens which means a lot of context is dropped.
@thiswillbeyourgithub commented on GitHub (Oct 16, 2024):
I brought it up multiple times and got ignored each time. See #5323
I think it's counting letters instead of tokens, which is complete bonkers IMO