Bug Summary:
After fetching and processing files, the reranking model does not function as expected. Weights from the DebertaV2ForSequenceClassification are not properly initialized, leading to ineffective reranking outcomes. Additionally, there seems to be a lack of interaction from the reranker with the documents, suggesting it cannot interpret or process document content effectively.
Steps to Reproduce:
Run start_windows.bat with the following environment variables set:
Ensure that the RAG_RERANKING_MODEL variable points to a functional model.
Observe the system logs for errors or warnings related to model initialization and embedding generation.
Note the behavior of the reranking model during document processing.
Expected Behavior:
The reranking model should successfully initialize with pre-trained weights and effectively interpret and process documents to enhance search and retrieval outcomes.
Actual Behavior:
Initialization errors occur due to some weights not being loaded from the checkpoint.
The reranker does not interact effectively with document content, leading to no meaningful enhancements in search or document retrieval functionality as no results used.
Environment
Open WebUI Version: v0.1.124
Operating System: Windows 10, manual install, CUDA support
Browser: Chrome
Reproduction Details
Confirmation:
I have read and followed all the instructions provided in the README.md.
I am on the latest version of Open WebUI.
I have included the browser console logs.
I have included the CMD logs.
Additional Notes:
The issues occur consistently across multiple attempts with varying document types and contents.
Potential compatibility issues with the mightbe/Better-PairRM model could be a contributing factor to the initialization problems.
Other Reranking models with same issue.
Logs and Screenshots
WARNING:models.DebertaV2:Some weights of DebertaV2ForSequenceClassification were not initialized from the model checkpoint at C:\Users\Chris\.cache\huggingface\hub\models--mightbe--Better-PairRM\snapshots\c26058437c327e56e878163f74b345b4fb75ce98 and are newly initialized
INFO:apps.rag.main:file.content_type: application/pdf
INFO:apps.rag.main:store_data_in_vector_db
INFO:apps.ollama.main:generate_ollama_embeddings model='snowflake-arctic-embed:latest' prompt='[details of prompt not fully captured]'
INFO: 192.168.1.224:61610 - "GET /_app/immutable/nodes/5.9117d953.js HTTP/1.1" 304 Not Modified
INFO:apps.ollama.main:generate_ollama_embeddings
and then does INFO:apps.ollama.main:generate_ollama_embeddings
where is reranking seen there?
Originally created by @chrisoutwright on GitHub (May 17, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/2354
## Bug Report
## Description
**Bug Summary:**
After fetching and processing files, the reranking model does not function as expected. Weights from the DebertaV2ForSequenceClassification are not properly initialized, leading to ineffective reranking outcomes. Additionally, there seems to be a lack of interaction from the reranker with the documents, suggesting it cannot interpret or process document content effectively.
**Steps to Reproduce:**
1. Run `start_windows.bat` with the following environment variables set:
```
ENABLE_RAG_HYBRID_SEARCH=True
RAG_EMBEDDING_MODEL=snowflake-arctic-embed:latest
RAG_EMBEDDING_ENGINE=ollama
OLLAMA_BASE_URLS=http://host.docker.internal:11434;http://host.docker.internal:11435;http://192.168.1.225:11439;http://192.168.1.225:11437;http://192.168.1.225:11436
CHUNK_SIZE=500
CHUNK_OVERLAP=100
RAG_RERANKING_MODEL_AUTO_UPDATE=True
RAG_RERANKING_MODEL=mightbe/Better-PairRM
```
2. Ensure that the `RAG_RERANKING_MODEL` variable points to a functional model.
3. Observe the system logs for errors or warnings related to model initialization and embedding generation.
4. Note the behavior of the reranking model during document processing.
**Expected Behavior:**
The reranking model should successfully initialize with pre-trained weights and effectively interpret and process documents to enhance search and retrieval outcomes.
**Actual Behavior:**
- Initialization errors occur due to some weights not being loaded from the checkpoint.
- The reranker does not interact effectively with document content, leading to no meaningful enhancements in search or document retrieval functionality as no results used.
## Environment
- **Open WebUI Version:** v0.1.124
- **Operating System:** Windows 10, manual install, CUDA support
- **Browser:** Chrome
## Reproduction Details
**Confirmation:**
- [x] I have read and followed all the instructions provided in the README.md.
- [x] I am on the latest version of Open WebUI.
- [ ] I have included the browser console logs.
- [ ] I have included the CMD logs.
**Additional Notes:**
- The issues occur consistently across multiple attempts with varying document types and contents.
- Potential compatibility issues with the `mightbe/Better-PairRM` model could be a contributing factor to the initialization problems.
- Other Reranking models with same issue.
## Logs and Screenshots
```
WARNING:models.DebertaV2:Some weights of DebertaV2ForSequenceClassification were not initialized from the model checkpoint at C:\Users\Chris\.cache\huggingface\hub\models--mightbe--Better-PairRM\snapshots\c26058437c327e56e878163f74b345b4fb75ce98 and are newly initialized
INFO:apps.rag.main:file.content_type: application/pdf
INFO:apps.rag.main:store_data_in_vector_db
INFO:apps.ollama.main:generate_ollama_embeddings model='snowflake-arctic-embed:latest' prompt='[details of prompt not fully captured]'
INFO: 192.168.1.224:61610 - "GET /_app/immutable/nodes/5.9117d953.js HTTP/1.1" 304 Not Modified
INFO:apps.ollama.main:generate_ollama_embeddings
```
Effectively it uses:
```
INFO:apps.ollama.main:generate_ollama_embeddings model='snowflake-arctic-embed:latest' prompt='who involved' options=None keep_alive=None
INFO:apps.ollama.main:url: http://localhost:11435
```
and then does
`INFO:apps.ollama.main:generate_ollama_embeddings `
where is reranking seen there?
@chrisoutwright commented on GitHub (May 18, 2024):
With hybrid search one would expect RerankCompressor being involved and then being logged with the query_doc_with_hybrid_search:result
I will try again in the Win10 setup, as for the logs. We need to understand the challenge better first, or are there known ones?
<!-- gh-comment-id:2118894491 -->
@chrisoutwright commented on GitHub (May 18, 2024):
With hybrid search one would expect [RerankCompressor ](https://github.com/open-webui/open-webui/blob/90503be2edef1a1f7ce2074286b6316d5cb8868a/backend/apps/rag/utils.py#L78) being involved and then being logged with the `query_doc_with_hybrid_search:result`
I will try again in the Win10 setup, as for the logs. We need to understand the challenge better first, or are there known ones?
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @chrisoutwright on GitHub (May 17, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/2354
Bug Report
Description
Bug Summary:
After fetching and processing files, the reranking model does not function as expected. Weights from the DebertaV2ForSequenceClassification are not properly initialized, leading to ineffective reranking outcomes. Additionally, there seems to be a lack of interaction from the reranker with the documents, suggesting it cannot interpret or process document content effectively.
Steps to Reproduce:
start_windows.batwith the following environment variables set:RAG_RERANKING_MODELvariable points to a functional model.Expected Behavior:
The reranking model should successfully initialize with pre-trained weights and effectively interpret and process documents to enhance search and retrieval outcomes.
Actual Behavior:
Environment
Reproduction Details
Confirmation:
Additional Notes:
mightbe/Better-PairRMmodel could be a contributing factor to the initialization problems.Logs and Screenshots
Effectively it uses:
and then does
INFO:apps.ollama.main:generate_ollama_embeddingswhere is reranking seen there?
@chrisoutwright commented on GitHub (May 17, 2024):
with reranking:
without reranking
@tjbck commented on GitHub (May 18, 2024):
PR welcome!
@chrisoutwright commented on GitHub (May 18, 2024):
With hybrid search one would expect RerankCompressor being involved and then being logged with the
query_doc_with_hybrid_search:resultI will try again in the Win10 setup, as for the logs. We need to understand the challenge better first, or are there known ones?