mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-18 02:04:06 -05:00
issue: hybrid search #3044
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#3044
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @AttaInsularis on GitHub (Dec 19, 2024).
Bug Report
Installation Method
pip (updated once)
Environment
Open WebUI Version: v0.4.8
Ollama (if applicable): ollama -v = 0.5.4
Operating System: Ubuntu 24.04.1 LTS
Browser (if applicable): LibreWolf (Firefox fork from stable latest)
Confirmation:
Expected Behavior:
When invoking RAG on the referenced library (uploaded folder) with # it goes always with some info returning to me in the chat flow.
But I can't know reliably if it does what is to be expected from that - go through the chunks from that #referenced_folder (with help of some RAG template) as > embed <> rerank > send it to LLM.
Can get some sense only after it does something (because of what is returned back from LLM).
But as the results were often unsatisfactory did 'dig in the process' and actually seems to me it failing silently - with different stages - to be described below.
Only clear error is if the request is too big for context window of LLM, otherwise the effort is wasted or incomplete response.
Actual Behavior:
There are 3 states of what hybrid search (as sourcing document from knowledge/RAG) ending after run with my set up:
INFO [open_webui.apps.retrieval.utils] query_doc_with_hybrid_search:result [[]] [[]]For this obtained answer from LLM usually stating 'Unfortunately, no document was provided for this task. However, I can provide some general information ..' because there is no info processed from knowledge base.
/Deleted quite a lot of lines - should that be here(?) - it's similar to the Traceback, just different .py files./
ending with this:
and last part
Similar to(?) #https://github.com/open-webui/open-webui/discussions/7437
INFO [open_webui.apps.retrieval.utils] query_doc_with_hybrid_search:result.Usually if it return matching {chunks0},{chunk1}..up to {chunkXX}, that the next after this first is:
a) less filled or even empty,
or
b) empty and 3th containing few chunks
Obtained so far maximum of 3 returned if it does not fail.
Don't know really how many returned
query_doc_with_hybrid_search:result [[]] [[]]there should be.Description
Bug Summary:
It will be great to let me know there is some state of returned RAG and ask me if I would like to:
or
or
Reproduction Details
Steps to Reproduce:
Not sure if it can be reproduce, it just does happen, don't know why or how to trigger that with certainty.
My installation is, maybe that help:
starting service with env variable (
export USE_CUDA_DOCKER=true) in ~/.bashrc(Info from #https://github.com/open-webui/open-webui/discussions/6530)
source open-webui/venv/bin/activate
open-webui serve --port some_port
OWUI does start with CUDA info:
and only warning is
CORS_ALLOW_ORIGIN IS SET TO '*' - NOT RECOMMENDED FOR PRODUCTION DEPLOYMENTS.Ollama is running as well - from
watch -n 10 'journalctl -e -u ollama'I can't see any error during the RAG.Tika, Ollama and the Open AI API are OK - confirming connection in the settings.
Have Nvidia only with 4GB VRAM - yet models does fit in afaik.
(NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4)
GPU utilization spike up (when first attempt made after start of OWIU, it fill up VRAM or load again if there has been a release after no use for awhile) but despite that no returned data in array.
In case of Error it seems to me that there is attempt, but small spike just as quick sharp angle to be seen on graph.
But as I do not know for sure it happening until LLM response - do not catch that up yet to be sure.
Usually after one or occasionally more attempts there is meaningful outcome - if the empty one or error happened.
Also sometimes the referenced files are in line below response (if can't fit in, with drop down to list them all) and sometimes they are referenced directly in the response.
Not sure how can that be controlled so far - but definitely best LLM response is with direct link to the doc from rather than the below option.
Logs and Screenshots
Docker Container Logs:
Can only see those and the time does not match the behavior/attempts for RAG - it does match the document processing (embedding).
Example:
WARN [qtp47918293-17] 19:39:03,238 org.apache.pdfbox.pdmodel.font.PDTrueTypeFont Using fallback font LiberationSans for ArialMT
INFO [qtp47918293-21] 19:39:08,333 org.apache.tika.server.core.resource.TikaResource /tika (application/pdf)
Additional Information
feels a bit like happening often as with the first attempt to RAG the referenced library, or after starting the OWUI
do not know what that error is
that is somehow correct response from the hybrid search - just it's odd if multiple of them is returned with one empty
What is correct finish of this procedure anyway?
Overall - the very next invoked repeat from UI (so no change in prompt - just repeating the same again) are usually OK, rarely it fail twice in a row.
So far more than 200 questions in chat with RAG, multiple times did reset Upload and Vector storage.
Fiddling with Chunk params, Query params (Minimum score helps a lot to have at least 'some').
Changing the Embed/Rerank models (but mostly to fit it in VRAM and not having to wait for load while also can work RAG'ing through files - so can't compare much as bigger ones having problems as GPU memory insufficient - thus not anyhow related to this).
Found out this #https://github.com/open-webui/open-webui/discussions/3868#discussioncomment-10782274 - but unsure if that is the same case. My prompt is not above 2048 words/tokens counting for with RAG template, and my chunk is half size to be processed.
I'm missing something here maybe?
Did set the Context length to 4K at least for my chosen Embedding Model: nomic-embed-text:latest - run some RAG several times and still having for example the empty HS response case.
Can't set up that for reranker - it's downloaded (that can be seen in cli after pressing download button) but not in the Models list.
I do not import documents into chat but rather reference them after mass importing into knowledge base to have it more on one place - thematically. It's not big numbers - usually 20 or so books or up to 50 .pdf to keep the context of the files concentrated.
And also to ease the load on machine.
RAM definitely with some free space at the moment and the CPU doesn't spike more than +-2/3 utilization while normally processing from RAG with good outcome.
Info below this - outdated: this #https://github.com/open-webui/open-webui/discussions/7821#discussioncomment-11641870 does solve the VRAM problem for me so far. Retained for context.
@mahenning commented on GitHub (Dec 19, 2024):
Please add the full traceback for 2.
@mahenning commented on GitHub (Dec 19, 2024):
I can re-create 1., and I think the problem is the minimum score of 0.2, at least it is for me. My tested knowledge base is 4 documents of varying size and research topics, and if I ask a general question like
The LLM (gemma2:2b) can answer me, but the relevance scores are at max at 0.01 (you can click on a source below the answer to see the relevance scores). If I set the min score to 0.2, the model get none of the sources because of the low relevance score.
When I ask a question about e.g. swin transformers (one of the papers is about that), I get a lot of chunks that far exceeds the 0.2 threshold and are therefore used.
One other thing, when I use sentence transformers as embedding model, I get the following error:
open-webui version: 0.4.8, docker with CUDA
Model: gemma2:2b, Ollama (GPU)
Embedding settings:
@AttaInsularis commented on GitHub (Dec 22, 2024):
Yay, find it, there it is:
INFO [open_webui.apps.ollama.main] get_all_models() INFO [open_webui.apps.ollama.main] url: http://localhost:11434 ERROR [open_webui.apps.retrieval.utils] Error when querying the collection with hybrid_search: 'dict' object has no attribute 'split' Traceback (most recent call last): File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 216, in query_collection_with_hybrid_search result = query_doc_with_hybrid_search( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 130, in query_doc_with_hybrid_search raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 117, in query_doc_with_hybrid_search result = compression_retriever.invoke(query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_core/retrievers.py", line 266, in invoke raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_core/retrievers.py", line 259, in invoke result = self._get_relevant_documents( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents docs = self.base_retriever.invoke( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain/retrievers/ensemble.py", line 118, in invoke raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain/retrievers/ensemble.py", line 115, in invoke result = self.rank_fusion(input, run_manager=run_manager, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain/retrievers/ensemble.py", line 222, in rank_fusion retriever_docs = [ ^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain/retrievers/ensemble.py", line 223, in <listcomp> retriever.invoke( File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_core/retrievers.py", line 266, in invoke raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_core/retrievers.py", line 259, in invoke result = self._get_relevant_documents( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_community/retrievers/bm25.py", line 101, in _get_relevant_documents processed_query = self.preprocess_func(query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/langchain_community/retrievers/bm25.py", line 12, in default_preprocessing_func return text.split() ^^^^^^^^^^ AttributeError: 'dict' object has no attribute 'split' 400 Client Error: Bad Request for url: http://localhost:11434/api/embed 'NoneType' object has no attribute 'ids' ERROR [open_webui.apps.retrieval.utils] Error when querying the collection: 'NoneType' object has no attribute 'ids' Traceback (most recent call last): File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 188, in query_collection result = query_doc( ^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 77, in query_doc raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 73, in query_doc log.info(f"query_doc:result {result.ids} {result.metadatas}") ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'ids' 400 Client Error: Bad Request for url: http://localhost:11434/api/embed 'NoneType' object has no attribute 'ids' ERROR [open_webui.apps.retrieval.utils] Error when querying the collection: 'NoneType' object has no attribute 'ids' Traceback (most recent call last): File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 188, in query_collection result = query_doc( ^^^^^^^^^^ File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 77, in query_doc raise e File "/home/home/open-webui/venv/lib/python3.11/site-packages/open_webui/apps/retrieval/utils.py", line 73, in query_doc log.info(f"query_doc:result {result.ids} {result.metadatas}") ^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'ids'@AttaInsularis commented on GitHub (Dec 22, 2024):
You are right, me having 30 books per one # referenced folder never accounted for that process being able to pull.. nothing out of such wast amount of text.
Despite being asking more precise and not that general questions.
That is sort of misunderstanding as 'normal' noRAG question into LLM's will usually return some hallucination at least one expecting it here as well.
Also did set for Nomic the Context Length for max. 8192.
But unsure how and if the rest of params for model are applied even for embedding/RAG or it is just here in general but shouldn't be use in this case(?)
So far the top_k and min. score had to be adjusted manually per texts to be retrieved and also counting for context window of the receiving LLM. And that balancing is some tedious job.
So the warning from UI could help, then this can be tackled immediately as the warning will eventually mark such occurrence.
Or a 'Tip' below prompt window:
If too much is retrieved (and send) > API error as it does not fit in context w. + if it does may it be also full of garbage and/or double-dealing with having 'same content' chunks occupying the precious space despite re-ranking.
(Good test for this is using 1st end 2nd edition of the same, lets say, a manual - it's to be seen immediately as LLM return duplicate statement.)
If too little then
a) if prompt (and/or RAG template) restrictive, no meaningful outcome
b) it just return some blatant not much target related text
Did have a lot of b) options not even know that what being returned is not much relevant to the content of actual files.
So as countermeasure my now used prompt is kinda restrictive:
Do not add or infer anything into what is provided in the first place. Be as precise as possible using information available within the documents. Print the name of the source. Only if nothing else can be obtained from source add as last paragraph some external content relevant to previous informationand such.That does work quite well so far.
But it's yes or no option and mainly only one or two facts summary can be obtained from the documents for me.
No middle ground.
Basically one of biggest help is that have, after years, my files somehow in folders - related by topic - and they are similar but diverse enough to be usable almost instantly.
But when started with this went the way 'lets take a pile of not-yet-sorted books and sort them with no reading'.
No way yet, we aren't here.
@AttaInsularis commented on GitHub (Dec 26, 2024):
It's happening to me that if only one (or very few) document are presented from the # option, with hybrid search on, I'm unable to pull simple summary of file.
If the case is I do not know what it is about, had to switch the HS off.
Only thereafter the first summary is completed, then can try search again (now knowing some semantics context from the source).
basically getting back with HS = on:
WARNI [chromadb.segment.impl.vector.local_persistent_hnsw] Number of requested results 65 is greater than number of elements in index 5, updating n_results = 5 INFO [open_webui.retrieval.utils] query_doc_with_hybrid_search:result [[]] [[]]That's ok, those are my settings (top_k=65) for entire folder with dozen or so documents. So that's easy to 'hit' something through re-ranking.
But with only few doc and 0 knowledge about that, it doesn't work straightforward.
(Say for making summary of a show I didn't see yet.)
Maybe manual trigger 'Summary/Unknown content' (doing HS on/of) for resolving the status of discrepancy:
the knowledge the person thinks he have as asking the question vs the documents content.
Triggered by the presence of 'RAG' (# of Knowledge or a manual import of files).
With title such as: Is the content known to me yet?
@gabriel-ecegi commented on GitHub (Feb 3, 2025):
It is fixed in 0.5.6 by moving hybrid_search process to background.
@ivanbaldo commented on GitHub (Mar 4, 2025):
If it's fixed then maybe this issue can be closed?
@AttaInsularis can you confirm?
@AttaInsularis commented on GitHub (Mar 6, 2025):
Hi, sorry for the delay.
As of now:
have no access for API that can handle a prompt containing that much information obtained from the RAG.
Also not atm near the machine that can handle RAG the way I did that (local database + local re-ranking with GPU).
So can't confirm any changes - 'solved by moving hybrid_search process to background.' - don't know what that really changed.
But as none contributed with any issues and the RAG is complicated in general, so my way of doing it probably hitting several different areas, I'm closing this.
Thank you all for whatever done! Really.