[GH-ISSUE #12347] issue: Hybrid search fails since version 0.6.0 #16566

Closed
opened 2026-04-19 22:27:34 -05:00 by GiteaMirror · 9 comments
Owner

Originally created by @DrZoidberg09 on GitHub (Apr 2, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12347

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.0

Ollama Version (if applicable)

No response

Operating System

Ubuntu 24.04

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have listed steps to reproduce the bug in detail.

Expected Behavior

I use Openweb-ui for hybrid RAG with ROCm and Flash-Attention based on Triton. This worked until the newest version. Please see docker log.

Actual Behavior

Hybrid search does not work anymore

Steps to Reproduce

  1. Have the described setup
  2. Use hybrid RAG search with JinaAI embedding and reranker
  3. Search in your Vector store
  4. Get error message as shown in logs

Logs & Screenshots

open-webui-rocm | 2025-04-02 13:41:29.465 | ERROR | open_webui.retrieval.utils:process_query:320 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'shape' - {}
open-webui-rocm | Traceback (most recent call last):
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1032, in _bootstrap
open-webui-rocm | self._bootstrap_inner()
open-webui-rocm | │ └ <function Thread._bootstrap_inner at 0x7f009fd8d580>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
open-webui-rocm | self.run()
open-webui-rocm | │ └ <function Thread.run at 0x7f009fd8d260>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1012, in run
open-webui-rocm | self._target(*self._args, **self._kwargs)
open-webui-rocm | │ │ │ │ │ └ {}
open-webui-rocm | │ │ │ │ └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)>
open-webui-rocm | │ │ │ └ (<weakref at 0x7efc42c29f80; to 'ThreadPoolExecutor' at 0x7efd5bc89e80>, <_queue.SimpleQueue object at 0x7efc42c29a80>, None,...
open-webui-rocm | │ │ └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)>
open-webui-rocm | │ └ <function _worker at 0x7f009ee21300>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
open-webui-rocm | work_item.run()
open-webui-rocm | │ └ <function _WorkItem.run at 0x7f009ee21440>
open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 59, in run
open-webui-rocm | result = self.fn(*self.args, **self.kwargs)
open-webui-rocm | │ │ │ │ │ └ {}
open-webui-rocm | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10>
open-webui-rocm | │ │ │ └ ('ab37fb86-7213-422c-b37d-01c83b58e6e5', 'nonwoven projects')
open-webui-rocm | │ │ └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10>
open-webui-rocm | │ └ <function query_collection_with_hybrid_search..process_query at 0x7efc9c5b6c00>
open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10>
open-webui-rocm |
open-webui-rocm | > File "/app/backend/open_webui/retrieval/utils.py", line 308, in process_query
open-webui-rocm | result = query_doc_with_hybrid_search(
open-webui-rocm | └ <function query_doc_with_hybrid_search at 0x7efd5c75c040>
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search
open-webui-rocm | raise e
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 145, in query_doc_with_hybrid_search
open-webui-rocm | result = compression_retriever.invoke(query)
open-webui-rocm | │ │ └ 'nonwoven projects'
open-webui-rocm | │ └ <function BaseRetriever.invoke at 0x7efd5cc97b00>
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke
open-webui-rocm | result = self._get_relevant_documents(
open-webui-rocm | │ └ <function ContextualCompressionRetriever._get_relevant_documents at 0x7efd5cc97740>
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents
open-webui-rocm | docs = self.base_retriever.invoke(
open-webui-rocm | │ │ └ <function EnsembleRetriever.invoke at 0x7efd5c9a80e0>
open-webui-rocm | │ └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7efd2e580500>, k=30), VectorSearchRet...
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 118, in invoke
open-webui-rocm | raise e
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 115, in invoke
open-webui-rocm | result = self.rank_fusion(input, run_manager=run_manager, config=config)
open-webui-rocm | │ │ │ │ └ {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x7efd2d0875c0>, 'recur...
open-webui-rocm | │ │ │ └ <langchain_core.callbacks.manager.CallbackManagerForRetrieverRun object at 0x7f009ed8c500>
open-webui-rocm | │ │ └ 'nonwoven projects'
open-webui-rocm | │ └ <function EnsembleRetriever.rank_fusion at 0x7efd5c9a8400>
open-webui-rocm | └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7efd2e580500>, k=30), VectorSearchRet...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 223, in rank_fusion
open-webui-rocm | retriever.invoke(
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke
open-webui-rocm | result = self._get_relevant_documents(
open-webui-rocm | │ └ <function VectorSearchRetriever._get_relevant_documents at 0x7efd5c78bec0>
open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi...
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 57, in _get_relevant_documents
open-webui-rocm | vectors=[self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)],
open-webui-rocm | │ │ │ └ None
open-webui-rocm | │ │ └ 'nonwoven projects'
open-webui-rocm | │ └ <function chat_completion_files_handler.... at 0x7efc169f9260>
open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi...
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/utils/middleware.py", line 598, in
open-webui-rocm | embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION(
open-webui-rocm | │ │ │ └ <property object at 0x7f009c5791c0>
open-webui-rocm | │ │ └ <starlette.requests.Request object at 0x7efc753e1880>
open-webui-rocm | │ └ None
open-webui-rocm | └ 'nonwoven projects'
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 356, in
open-webui-rocm | return lambda query, prefix=None, user=None: embedding_function.encode(
open-webui-rocm | │ │ └ <function SentenceTransformer.encode at 0x7efd2fb54d60>
open-webui-rocm | │ └ SentenceTransformer(
open-webui-rocm | │ (transformer): Transformer(
open-webui-rocm | │ (auto_model): XLMRobertaLoRA(
open-webui-rocm | │ (roberta): XLMRobertaModel(
open-webui-rocm | │ ...
open-webui-rocm | └ 'nonwoven projects'
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 557, in encode
open-webui-rocm | out_features = self.forward(features, **kwargs)
open-webui-rocm | │ │ │ └ {}
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | │ └ <function SentenceTransformer.forward at 0x7efd2fb54e00>
open-webui-rocm | └ SentenceTransformer(
open-webui-rocm | (transformer): Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 624, in forward
open-webui-rocm | input = module(input, **module_kwargs)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {}
open-webui-rocm | │ │ └ ({'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, ...
open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40>
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ ({'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, ...
open-webui-rocm | └ <bound method Transformer.forward of Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embe...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/f1944de8402dcd5f2b03f822a4bc22a7f2de2eb9/custom_st.py", line 143, in forward
open-webui-rocm | output_states = self.auto_model.forward(**features, **lora_arguments, return_dict=False)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_lora.py", line 372, in forward
open-webui-rocm | return self.roberta(*args, **kwargs)
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | │ └ ()
open-webui-rocm | └ XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEm...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | │ │ └ ()
open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40>
open-webui-rocm | └ XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEmbedding(
open-webui-rocm | 250002, 1024, padd...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 77635, 2, device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1...
open-webui-rocm | │ └ ()
open-webui-rocm | └ <bound method XLMRobertaModel.forward of XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): Parame...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 734, in forward
open-webui-rocm | sequence_output = self.encoder(
open-webui-rocm | └ XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEmbedding(
open-webui-rocm | 250002, 1024, padd...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'key_padding_mask': tensor(1, 1, 1, 1, 1, 1, device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_hidden...
open-webui-rocm | │ │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ...
open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40>
open-webui-rocm | └ XLMRobertaEncoder(
open-webui-rocm | (layers): ModuleList(
open-webui-rocm | (0-23): 24 x Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding(...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'key_padding_mask': tensor(1, 1, 1, 1, 1, 1, device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_hidden...
open-webui-rocm | │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ...
open-webui-rocm | └ <bound method XLMRobertaEncoder.forward of XLMRobertaEncoder(
open-webui-rocm | (layers): ModuleList(
open-webui-rocm | (0-23): 24 x Block(
open-webui-rocm | (mixer): ...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 258, in forward
open-webui-rocm | hidden_states = layer(hidden_states, mixer_kwargs=mixer_kwargs)
open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}
open-webui-rocm | │ └ tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02,
open-webui-rocm | │ 8.1177e-03, -5.4932e-02],
open-webui-rocm | │ [ 2.1094e-01, ...
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}}
open-webui-rocm | │ │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02,
open-webui-rocm | │ │ 8.1177e-03, -5.4932e-02],
open-webui-rocm | │ │ [ 2.1094e-01,...
open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40>
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}}
open-webui-rocm | │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02,
open-webui-rocm | │ 8.1177e-03, -5.4932e-02],
open-webui-rocm | │ [ 2.1094e-01,...
open-webui-rocm | └ <bound method Block.forward of Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResid...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/block.py", line 201, in forward
open-webui-rocm | mixer_out = self.mixer(
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}
open-webui-rocm | │ │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02,
open-webui-rocm | │ │ 8.1177e-03, -5.4932e-02],
open-webui-rocm | │ │ [ 2.1094e-01,...
open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40>
open-webui-rocm | └ MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, out_features=3072, bias=Tr...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}
open-webui-rocm | │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02,
open-webui-rocm | │ 8.1177e-03, -5.4932e-02],
open-webui-rocm | │ [ 2.1094e-01,...
open-webui-rocm | └ <bound method MHA.forward of MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=102...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/mha.py", line 732, in forward
open-webui-rocm | qkv = self.rotary_emb(
open-webui-rocm | └ MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, out_features=3072, bias=Tr...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in wrapped_call_impl
open-webui-rocm | return self.call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6}
open-webui-rocm | │ │ └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01,
open-webui-rocm | │ │ 6.8750e-01, 9.4141e-01],
open-webui-rocm | │ │ [ 1.845...
open-webui-rocm | │ └ <function Module.call_impl at 0x7efe70110f40>
open-webui-rocm | └ RotaryEmbedding()
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6}
open-webui-rocm | │ └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01,
open-webui-rocm | │ 6.8750e-01, 9.4141e-01],
open-webui-rocm | │ [ 1.845...
open-webui-rocm | └ <bound method RotaryEmbedding.forward of RotaryEmbedding()>
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 604, in forward
open-webui-rocm | return apply_rotary_emb_qkv
(
open-webui-rocm | └ <function apply_rotary_emb_qkv
at 0x7efd2e5d23e0>
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 327, in apply_rotary_emb_qkv

open-webui-rocm | return ApplyRotaryEmbQKV
.apply(
open-webui-rocm | │ └ <classmethod(<function Function.apply at 0x7efe6fe8d760>)>
open-webui-rocm | └ <class 'transformers_modules.jinaai.xlm-roberta-flash-implementation.2b6bc3f30750b3a9648fe9b63448c09920efe9be.rotary.ApplyRot...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply
open-webui-rocm | return super().apply(*args, **kwargs) # type: ignore[misc]
open-webui-rocm | │ └ {}
open-webui-rocm | └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01,
open-webui-rocm | 6.8750e-01, 9.4141e-01],
open-webui-rocm | [ 1.845...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 186, in forward
open-webui-rocm | apply_rotary(
open-webui-rocm | └ <function apply_rotary at 0x7efd2e5d1ee0>
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary
open-webui-rocm | assert sin.shape == cos.shape
open-webui-rocm | │ │ └ <attribute 'shape' of 'torch._C.TensorBase' objects>
open-webui-rocm | │ └ tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
open-webui-rocm | │ 1.0000, 1.0000, 1.0000, 1.0000,...
open-webui-rocm | └ None

Additional Information

No response

Originally created by @DrZoidberg09 on GitHub (Apr 2, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/12347 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.0 ### Ollama Version (if applicable) _No response_ ### Operating System Ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior I use Openweb-ui for hybrid RAG with ROCm and Flash-Attention based on Triton. This worked until the newest version. Please see docker log. ### Actual Behavior Hybrid search does not work anymore ### Steps to Reproduce 1. Have the described setup 2. Use hybrid RAG search with JinaAI embedding and reranker 3. Search in your Vector store 4. Get error message as shown in logs ### Logs & Screenshots open-webui-rocm | 2025-04-02 13:41:29.465 | ERROR | open_webui.retrieval.utils:process_query:320 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'shape' - {} open-webui-rocm | Traceback (most recent call last): open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1032, in _bootstrap open-webui-rocm | self._bootstrap_inner() open-webui-rocm | │ └ <function Thread._bootstrap_inner at 0x7f009fd8d580> open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1075, in _bootstrap_inner open-webui-rocm | self.run() open-webui-rocm | │ └ <function Thread.run at 0x7f009fd8d260> open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1012, in run open-webui-rocm | self._target(*self._args, **self._kwargs) open-webui-rocm | │ │ │ │ │ └ {} open-webui-rocm | │ │ │ │ └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)> open-webui-rocm | │ │ │ └ (<weakref at 0x7efc42c29f80; to 'ThreadPoolExecutor' at 0x7efd5bc89e80>, <_queue.SimpleQueue object at 0x7efc42c29a80>, None,... open-webui-rocm | │ │ └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)> open-webui-rocm | │ └ <function _worker at 0x7f009ee21300> open-webui-rocm | └ <Thread(ThreadPoolExecutor-9_0, started 139616139085504)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker open-webui-rocm | work_item.run() open-webui-rocm | │ └ <function _WorkItem.run at 0x7f009ee21440> open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 59, in run open-webui-rocm | result = self.fn(*self.args, **self.kwargs) open-webui-rocm | │ │ │ │ │ └ {} open-webui-rocm | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10> open-webui-rocm | │ │ │ └ ('ab37fb86-7213-422c-b37d-01c83b58e6e5', 'nonwoven projects') open-webui-rocm | │ │ └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10> open-webui-rocm | │ └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7efc9c5b6c00> open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7efd2dcacd10> open-webui-rocm | open-webui-rocm | > File "/app/backend/open_webui/retrieval/utils.py", line 308, in process_query open-webui-rocm | result = query_doc_with_hybrid_search( open-webui-rocm | └ <function query_doc_with_hybrid_search at 0x7efd5c75c040> open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search open-webui-rocm | raise e open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 145, in query_doc_with_hybrid_search open-webui-rocm | result = compression_retriever.invoke(query) open-webui-rocm | │ │ └ 'nonwoven projects' open-webui-rocm | │ └ <function BaseRetriever.invoke at 0x7efd5cc97b00> open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke open-webui-rocm | result = self._get_relevant_documents( open-webui-rocm | │ └ <function ContextualCompressionRetriever._get_relevant_documents at 0x7efd5cc97740> open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents open-webui-rocm | docs = self.base_retriever.invoke( open-webui-rocm | │ │ └ <function EnsembleRetriever.invoke at 0x7efd5c9a80e0> open-webui-rocm | │ └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7efd2e580500>, k=30), VectorSearchRet... open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 118, in invoke open-webui-rocm | raise e open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 115, in invoke open-webui-rocm | result = self.rank_fusion(input, run_manager=run_manager, config=config) open-webui-rocm | │ │ │ │ └ {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x7efd2d0875c0>, 'recur... open-webui-rocm | │ │ │ └ <langchain_core.callbacks.manager.CallbackManagerForRetrieverRun object at 0x7f009ed8c500> open-webui-rocm | │ │ └ 'nonwoven projects' open-webui-rocm | │ └ <function EnsembleRetriever.rank_fusion at 0x7efd5c9a8400> open-webui-rocm | └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7efd2e580500>, k=30), VectorSearchRet... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 223, in rank_fusion open-webui-rocm | retriever.invoke( open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke open-webui-rocm | result = self._get_relevant_documents( open-webui-rocm | │ └ <function VectorSearchRetriever._get_relevant_documents at 0x7efd5c78bec0> open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi... open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 57, in _get_relevant_documents open-webui-rocm | vectors=[self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)], open-webui-rocm | │ │ │ └ None open-webui-rocm | │ │ └ 'nonwoven projects' open-webui-rocm | │ └ <function chat_completion_files_handler.<locals>.<lambda>.<locals>.<lambda> at 0x7efc169f9260> open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi... open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/utils/middleware.py", line 598, in <lambda> open-webui-rocm | embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION( open-webui-rocm | │ │ │ └ <property object at 0x7f009c5791c0> open-webui-rocm | │ │ └ <starlette.requests.Request object at 0x7efc753e1880> open-webui-rocm | │ └ None open-webui-rocm | └ 'nonwoven projects' open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 356, in <lambda> open-webui-rocm | return lambda query, prefix=None, user=None: embedding_function.encode( open-webui-rocm | │ │ └ <function SentenceTransformer.encode at 0x7efd2fb54d60> open-webui-rocm | │ └ SentenceTransformer( open-webui-rocm | │ (transformer): Transformer( open-webui-rocm | │ (auto_model): XLMRobertaLoRA( open-webui-rocm | │ (roberta): XLMRobertaModel( open-webui-rocm | │ ... open-webui-rocm | └ 'nonwoven projects' open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 557, in encode open-webui-rocm | out_features = self.forward(features, **kwargs) open-webui-rocm | │ │ │ └ {} open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | │ └ <function SentenceTransformer.forward at 0x7efd2fb54e00> open-webui-rocm | └ SentenceTransformer( open-webui-rocm | (transformer): Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 624, in forward open-webui-rocm | input = module(input, **module_kwargs) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {} open-webui-rocm | │ │ └ ({'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, ... open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ ({'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, ... open-webui-rocm | └ <bound method Transformer.forward of Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embe... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/f1944de8402dcd5f2b03f822a4bc22a7f2de2eb9/custom_st.py", line 143, in forward open-webui-rocm | output_states = self.auto_model.forward(**features, **lora_arguments, return_dict=False) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_lora.py", line 372, in forward open-webui-rocm | return self.roberta(*args, **kwargs) open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | │ └ () open-webui-rocm | └ XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEm... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | │ │ └ () open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEmbedding( open-webui-rocm | 250002, 1024, padd... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 77635, 2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1... open-webui-rocm | │ └ () open-webui-rocm | └ <bound method XLMRobertaModel.forward of XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): Parame... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 734, in forward open-webui-rocm | sequence_output = self.encoder( open-webui-rocm | └ XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEmbedding( open-webui-rocm | 250002, 1024, padd... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'key_padding_mask': tensor([[1, 1, 1, 1, 1, 1]], device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_hidden... open-webui-rocm | │ │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ... open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ XLMRobertaEncoder( open-webui-rocm | (layers): ModuleList( open-webui-rocm | (0-23): 24 x Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding(... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'key_padding_mask': tensor([[1, 1, 1, 1, 1, 1]], device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_hidden... open-webui-rocm | │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ... open-webui-rocm | └ <bound method XLMRobertaEncoder.forward of XLMRobertaEncoder( open-webui-rocm | (layers): ModuleList( open-webui-rocm | (0-23): 24 x Block( open-webui-rocm | (mixer): ... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 258, in forward open-webui-rocm | hidden_states = layer(hidden_states, mixer_kwargs=mixer_kwargs) open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None} open-webui-rocm | │ └ tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02, open-webui-rocm | │ 8.1177e-03, -5.4932e-02], open-webui-rocm | │ [ 2.1094e-01, ... open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}} open-webui-rocm | │ │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02, open-webui-rocm | │ │ 8.1177e-03, -5.4932e-02], open-webui-rocm | │ │ [ 2.1094e-01,... open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None}} open-webui-rocm | │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02, open-webui-rocm | │ 8.1177e-03, -5.4932e-02], open-webui-rocm | │ [ 2.1094e-01,... open-webui-rocm | └ <bound method Block.forward of Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResid... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/block.py", line 201, in forward open-webui-rocm | mixer_out = self.mixer( open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None} open-webui-rocm | │ │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02, open-webui-rocm | │ │ 8.1177e-03, -5.4932e-02], open-webui-rocm | │ │ [ 2.1094e-01,... open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, out_features=3072, bias=Tr... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6, 'adapter_mask': None} open-webui-rocm | │ └ (tensor([[ 2.2583e-02, 8.3618e-03, -3.0670e-03, ..., 3.9062e-02, open-webui-rocm | │ 8.1177e-03, -5.4932e-02], open-webui-rocm | │ [ 2.1094e-01,... open-webui-rocm | └ <bound method MHA.forward of MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=102... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/mha.py", line 732, in forward open-webui-rocm | qkv = self.rotary_emb( open-webui-rocm | └ MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, out_features=3072, bias=Tr... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6} open-webui-rocm | │ │ └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01, open-webui-rocm | │ │ 6.8750e-01, 9.4141e-01], open-webui-rocm | │ │ [ 1.845... open-webui-rocm | │ └ <function Module._call_impl at 0x7efe70110f40> open-webui-rocm | └ RotaryEmbedding() open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 6], device='cuda:0', dtype=torch.int32), 'max_seqlen': 6} open-webui-rocm | │ └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01, open-webui-rocm | │ 6.8750e-01, 9.4141e-01], open-webui-rocm | │ [ 1.845... open-webui-rocm | └ <bound method RotaryEmbedding.forward of RotaryEmbedding()> open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 604, in forward open-webui-rocm | return apply_rotary_emb_qkv_( open-webui-rocm | └ <function apply_rotary_emb_qkv_ at 0x7efd2e5d23e0> open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 327, in apply_rotary_emb_qkv_ open-webui-rocm | return ApplyRotaryEmbQKV_.apply( open-webui-rocm | │ └ <classmethod(<function Function.apply at 0x7efe6fe8d760>)> open-webui-rocm | └ <class 'transformers_modules.jinaai.xlm-roberta-flash-implementation.2b6bc3f30750b3a9648fe9b63448c09920efe9be.rotary.ApplyRot... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply open-webui-rocm | return super().apply(*args, **kwargs) # type: ignore[misc] open-webui-rocm | │ └ {} open-webui-rocm | └ (tensor([[[[ 1.8555e-01, 1.9531e-01, 2.7344e-01, ..., -4.1992e-01, open-webui-rocm | 6.8750e-01, 9.4141e-01], open-webui-rocm | [ 1.845... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 186, in forward open-webui-rocm | apply_rotary( open-webui-rocm | └ <function apply_rotary at 0x7efd2e5d1ee0> open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary open-webui-rocm | assert sin.shape == cos.shape open-webui-rocm | │ │ └ <attribute 'shape' of 'torch._C.TensorBase' objects> open-webui-rocm | │ └ tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, open-webui-rocm | │ 1.0000, 1.0000, 1.0000, 1.0000,... open-webui-rocm | └ None ### Additional Information _No response_
GiteaMirror added the bug label 2026-04-19 22:27:35 -05:00
Author
Owner

@tjbck commented on GitHub (Apr 3, 2025):

@Phlogi

<!-- gh-comment-id:2774023694 --> @tjbck commented on GitHub (Apr 3, 2025): @Phlogi
Author
Owner

@Phlogi commented on GitHub (Apr 3, 2025):

I can't reproduce yet, is the collection empty? @DrZoidberg09 Please provide the log earlier, the error does not match the code in my opinion.

There we have an issue @tjbck This is the output when querying a collection without any documents in it. I think this be handled early, maybe before even calling query_doc_with_hybrid_search()?

open-webui        | 2025-04-03 07:16:43.989 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
open-webui        | 2025-04-03 07:16:46.120 | ERROR    | open_webui.retrieval.utils:query_collection_with_hybrid_search:299 - Failed to fetch collection 26268380-14ee-4c1a-bd25-c790c8fd3e16: Collection 26268380-14ee-4c1a-bd25-c790c8fd3e16 does not exist. - {}
open-webui        | Traceback (most recent call last):
open-webui        | 
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
open-webui        |     self._bootstrap_inner()
open-webui        |     │    └ <function Thread._bootstrap_inner at 0x767101794860>
open-webui        |     └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
open-webui        |     self.run()
open-webui        |     │    └ <function Thread.run at 0x767101794540>
open-webui        |     └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 982, in run
open-webui        |     self._target(*self._args, **self._kwargs)
open-webui        |     │    │        │    │        │    └ {}
open-webui        |     │    │        │    │        └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)>
open-webui        |     │    │        │    └ (<weakref at 0x76700dde31a0; to 'ThreadPoolExecutor' at 0x76700dd206d0>, <_queue.SimpleQueue object at 0x76700f041df0>, None,...
open-webui        |     │    │        └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)>
open-webui        |     │    └ <function _worker at 0x7671006b00e0>
open-webui        |     └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
open-webui        |     work_item.run()
open-webui        |     │         └ <function _WorkItem.run at 0x7671006b0220>
open-webui        |     └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
open-webui        |     result = self.fn(*self.args, **self.kwargs)
open-webui        |              │    │   │    │       │    └ {}
open-webui        |              │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0>
open-webui        |              │    │   │    └ ()
open-webui        |              │    │   └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0>
open-webui        |              │    └ <function chat_completion_files_handler.<locals>.<lambda> at 0x76700e76e2a0>
open-webui        |              └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0>
open-webui        | 
open-webui        |   File "/app/backend/open_webui/utils/middleware.py", line 594, in <lambda>
open-webui        |     lambda: get_sources_from_files(
open-webui        |             └ <function get_sources_from_files at 0x7670838236a0>
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 501, in get_sources_from_files
open-webui        |     context = query_collection_with_hybrid_search(
open-webui        |               └ <function query_collection_with_hybrid_search at 0x767083822ac0>
open-webui        | 
open-webui        | > File "/app/backend/open_webui/retrieval/utils.py", line 295, in query_collection_with_hybrid_search
open-webui        |     collection_results[collection_name] = VECTOR_DB_CLIENT.get(
open-webui        |     │                  │                  │                └ <function ChromaClient.get at 0x76708dfd8ea0>
open-webui        |     │                  │                  └ <open_webui.retrieval.vector.dbs.chroma.ChromaClient object at 0x76708dfce950>
open-webui        |     │                  └ '26268380-14ee-4c1a-bd25-c790c8fd3e16'
open-webui        |     └ {}
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/vector/dbs/chroma.py", line 121, in get
open-webui        |     collection = self.client.get_collection(name=collection_name)
open-webui        |                  │    │      │                   └ '26268380-14ee-4c1a-bd25-c790c8fd3e16'
open-webui        |                  │    │      └ <function Client.get_collection at 0x76708ebac2c0>
open-webui        |                  │    └ <chromadb.api.client.Client object at 0x76708df6c750>
open-webui        |                  └ <open_webui.retrieval.vector.dbs.chroma.ChromaClient object at 0x76708dfce950>
open-webui        | 
open-webui        |   File "/usr/local/lib/python3.11/site-packages/chromadb/api/client.py", line 171, in get_collection
open-webui        |     model = self._server.get_collection(
open-webui        |             │    │       └ <function SegmentAPI.get_collection at 0x76708dbb40e0>
open-webui        |             │    └ <chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>
open-webui        |             └ <chromadb.api.client.Client object at 0x76708df6c750>
open-webui        |   File "/usr/local/lib/python3.11/site-packages/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper
open-webui        |     return f(*args, **kwargs)
open-webui        |            │  │       └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'}
open-webui        |            │  └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,)
open-webui        |            └ <function SegmentAPI.get_collection at 0x76708dbb4040>
open-webui        |   File "/usr/local/lib/python3.11/site-packages/chromadb/api/segment.py", line 103, in wrapper
open-webui        |     return self._rate_limit_enforcer.rate_limit(func)(*args, **kwargs)
open-webui        |            │    │                    │          │      │       └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'}
open-webui        |            │    │                    │          │      └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,)
open-webui        |            │    │                    │          └ <function SegmentAPI.get_collection at 0x76708dba3f60>
open-webui        |            │    │                    └ <function SimpleRateLimitEnforcer.rate_limit at 0x76708d99d300>
open-webui        |            │    └ <chromadb.rate_limit.simple_rate_limit.SimpleRateLimitEnforcer object at 0x76708e0ee310>
open-webui        |            └ <chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>
open-webui        |   File "/usr/local/lib/python3.11/site-packages/chromadb/rate_limit/simple_rate_limit/__init__.py", line 23, in wrapper
open-webui        |     return func(*args, **kwargs)
open-webui        |            │     │       └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'}
open-webui        |            │     └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,)
open-webui        |            └ <function SegmentAPI.get_collection at 0x76708dba3f60>
open-webui        |   File "/usr/local/lib/python3.11/site-packages/chromadb/api/segment.py", line 300, in get_collection
open-webui        |     raise InvalidCollectionException(f"Collection {name} does not exist.")
open-webui        |           └ <class 'chromadb.errors.InvalidCollectionException'>
open-webui        | 
open-webui        | chromadb.errors.InvalidCollectionException: Collection 26268380-14ee-4c1a-bd25-c790c8fd3e16 does not exist.
open-webui        | 2025-04-03 07:16:46.123 | INFO     | open_webui.retrieval.utils:query_collection_with_hybrid_search:302 - Starting hybrid search for 3 queries in 1 collections... - {}
open-webui        | 2025-04-03 07:16:46.124 | ERROR    | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {}
open-webui        | Traceback (most recent call last):
open-webui        | 
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
open-webui        |     self._bootstrap_inner()
open-webui        |     │    └ <function Thread._bootstrap_inner at 0x767101794860>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
open-webui        |     self.run()
open-webui        |     │    └ <function Thread.run at 0x767101794540>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 982, in run
open-webui        |     self._target(*self._args, **self._kwargs)
open-webui        |     │    │        │    │        │    └ {}
open-webui        |     │    │        │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    │        │    └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,...
open-webui        |     │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    └ <function _worker at 0x7671006b00e0>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
open-webui        |     work_item.run()
open-webui        |     │         └ <function _WorkItem.run at 0x7671006b0220>
open-webui        |     └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
open-webui        |     result = self.fn(*self.args, **self.kwargs)
open-webui        |              │    │   │    │       │    └ {}
open-webui        |              │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0>
open-webui        |              │    │   │    └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Was steht drin?')
open-webui        |              │    │   └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0>
open-webui        |              │    └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400>
open-webui        |              └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0>
open-webui        | 
open-webui        | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query
open-webui        |     result = query_doc_with_hybrid_search(
open-webui        |              └ <function query_doc_with_hybrid_search at 0x767083823880>
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search
open-webui        |     raise e
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search
open-webui        |     texts=collection_result.documents[0],
open-webui        |           └ None
open-webui        | 
open-webui        | AttributeError: 'NoneType' object has no attribute 'documents'
open-webui        | 2025-04-03 07:16:46.125 | ERROR    | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {}
open-webui        | Traceback (most recent call last):
open-webui        | 
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
open-webui        |     self._bootstrap_inner()
open-webui        |     │    └ <function Thread._bootstrap_inner at 0x767101794860>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
open-webui        |     self.run()
open-webui        |     │    └ <function Thread.run at 0x767101794540>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 982, in run
open-webui        |     self._target(*self._args, **self._kwargs)
open-webui        |     │    │        │    │        │    └ {}
open-webui        |     │    │        │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    │        │    └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,...
open-webui        |     │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    └ <function _worker at 0x7671006b00e0>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
open-webui        |     work_item.run()
open-webui        |     │         └ <function _WorkItem.run at 0x7671006b0220>
open-webui        |     └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
open-webui        |     result = self.fn(*self.args, **self.kwargs)
open-webui        |              │    │   │    │       │    └ {}
open-webui        |              │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490>
open-webui        |              │    │   │    └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Aktuelle Informationen zu Was steht drin?')
open-webui        |              │    │   └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490>
open-webui        |              │    └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400>
open-webui        |              └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490>
open-webui        | 
open-webui        | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query
open-webui        |     result = query_doc_with_hybrid_search(
open-webui        |              └ <function query_doc_with_hybrid_search at 0x767083823880>
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search
open-webui        |     raise e
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search
open-webui        |     texts=collection_result.documents[0],
open-webui        |           └ None
open-webui        | 
open-webui        | AttributeError: 'NoneType' object has no attribute 'documents'
open-webui        | 2025-04-03 07:16:46.126 | ERROR    | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {}
open-webui        | Traceback (most recent call last):
open-webui        | 
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
open-webui        |     self._bootstrap_inner()
open-webui        |     │    └ <function Thread._bootstrap_inner at 0x767101794860>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
open-webui        |     self.run()
open-webui        |     │    └ <function Thread.run at 0x767101794540>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/threading.py", line 982, in run
open-webui        |     self._target(*self._args, **self._kwargs)
open-webui        |     │    │        │    │        │    └ {}
open-webui        |     │    │        │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    │        │    └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,...
open-webui        |     │    │        └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |     │    └ <function _worker at 0x7671006b00e0>
open-webui        |     └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
open-webui        |     work_item.run()
open-webui        |     │         └ <function _WorkItem.run at 0x7671006b0220>
open-webui        |     └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50>
open-webui        |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
open-webui        |     result = self.fn(*self.args, **self.kwargs)
open-webui        |              │    │   │    │       │    └ {}
open-webui        |              │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50>
open-webui        |              │    │   │    └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Was steht drin? 2025')
open-webui        |              │    │   └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50>
open-webui        |              │    └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400>
open-webui        |              └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50>
open-webui        | 
open-webui        | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query
open-webui        |     result = query_doc_with_hybrid_search(
open-webui        |              └ <function query_doc_with_hybrid_search at 0x767083823880>
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search
open-webui        |     raise e
open-webui        | 
open-webui        |   File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search
open-webui        |     texts=collection_result.documents[0],
open-webui        |           └ None
open-webui        | 
open-webui        | AttributeError: 'NoneType' object has no attribute 'documents'
Batches: 100%|██████████| 1/1 [00:00<00:00, 77.42it/s]
Batches: 100%|██████████| 1/1 [00:00<00:00, 77.52it/s]
Batches: 100%|██████████| 1/1 [00:00<00:00, 92.58it/s]
open-webui        | 2025-04-03 07:16:46.187 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "POST /api/chat/completions HTTP/1.1" 200 - {}
open-webui        | 2025-04-03 07:16:46.226 | INFO     | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {}
<!-- gh-comment-id:2774723614 --> @Phlogi commented on GitHub (Apr 3, 2025): I can't reproduce yet, is the collection empty? @DrZoidberg09 Please provide the log earlier, the error does not match the code in my opinion. There we have an issue @tjbck This is the output when querying a collection without any documents in it. I think this be handled early, maybe before even calling query_doc_with_hybrid_search()? ``` open-webui | 2025-04-03 07:16:43.989 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {} open-webui | 2025-04-03 07:16:46.120 | ERROR | open_webui.retrieval.utils:query_collection_with_hybrid_search:299 - Failed to fetch collection 26268380-14ee-4c1a-bd25-c790c8fd3e16: Collection 26268380-14ee-4c1a-bd25-c790c8fd3e16 does not exist. - {} open-webui | Traceback (most recent call last): open-webui | open-webui | File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap open-webui | self._bootstrap_inner() open-webui | │ └ <function Thread._bootstrap_inner at 0x767101794860> open-webui | └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner open-webui | self.run() open-webui | │ └ <function Thread.run at 0x767101794540> open-webui | └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 982, in run open-webui | self._target(*self._args, **self._kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)> open-webui | │ │ │ └ (<weakref at 0x76700dde31a0; to 'ThreadPoolExecutor' at 0x76700dd206d0>, <_queue.SimpleQueue object at 0x76700f041df0>, None,... open-webui | │ │ └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)> open-webui | │ └ <function _worker at 0x7671006b00e0> open-webui | └ <Thread(ThreadPoolExecutor-5_0, started 130223286781632)> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker open-webui | work_item.run() open-webui | │ └ <function _WorkItem.run at 0x7671006b0220> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run open-webui | result = self.fn(*self.args, **self.kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0> open-webui | │ │ │ └ () open-webui | │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0> open-webui | │ └ <function chat_completion_files_handler.<locals>.<lambda> at 0x76700e76e2a0> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700e745bd0> open-webui | open-webui | File "/app/backend/open_webui/utils/middleware.py", line 594, in <lambda> open-webui | lambda: get_sources_from_files( open-webui | └ <function get_sources_from_files at 0x7670838236a0> open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 501, in get_sources_from_files open-webui | context = query_collection_with_hybrid_search( open-webui | └ <function query_collection_with_hybrid_search at 0x767083822ac0> open-webui | open-webui | > File "/app/backend/open_webui/retrieval/utils.py", line 295, in query_collection_with_hybrid_search open-webui | collection_results[collection_name] = VECTOR_DB_CLIENT.get( open-webui | │ │ │ └ <function ChromaClient.get at 0x76708dfd8ea0> open-webui | │ │ └ <open_webui.retrieval.vector.dbs.chroma.ChromaClient object at 0x76708dfce950> open-webui | │ └ '26268380-14ee-4c1a-bd25-c790c8fd3e16' open-webui | └ {} open-webui | open-webui | File "/app/backend/open_webui/retrieval/vector/dbs/chroma.py", line 121, in get open-webui | collection = self.client.get_collection(name=collection_name) open-webui | │ │ │ └ '26268380-14ee-4c1a-bd25-c790c8fd3e16' open-webui | │ │ └ <function Client.get_collection at 0x76708ebac2c0> open-webui | │ └ <chromadb.api.client.Client object at 0x76708df6c750> open-webui | └ <open_webui.retrieval.vector.dbs.chroma.ChromaClient object at 0x76708dfce950> open-webui | open-webui | File "/usr/local/lib/python3.11/site-packages/chromadb/api/client.py", line 171, in get_collection open-webui | model = self._server.get_collection( open-webui | │ │ └ <function SegmentAPI.get_collection at 0x76708dbb40e0> open-webui | │ └ <chromadb.api.segment.SegmentAPI object at 0x76708dfcf110> open-webui | └ <chromadb.api.client.Client object at 0x76708df6c750> open-webui | File "/usr/local/lib/python3.11/site-packages/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper open-webui | return f(*args, **kwargs) open-webui | │ │ └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'} open-webui | │ └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,) open-webui | └ <function SegmentAPI.get_collection at 0x76708dbb4040> open-webui | File "/usr/local/lib/python3.11/site-packages/chromadb/api/segment.py", line 103, in wrapper open-webui | return self._rate_limit_enforcer.rate_limit(func)(*args, **kwargs) open-webui | │ │ │ │ │ └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'} open-webui | │ │ │ │ └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,) open-webui | │ │ │ └ <function SegmentAPI.get_collection at 0x76708dba3f60> open-webui | │ │ └ <function SimpleRateLimitEnforcer.rate_limit at 0x76708d99d300> open-webui | │ └ <chromadb.rate_limit.simple_rate_limit.SimpleRateLimitEnforcer object at 0x76708e0ee310> open-webui | └ <chromadb.api.segment.SegmentAPI object at 0x76708dfcf110> open-webui | File "/usr/local/lib/python3.11/site-packages/chromadb/rate_limit/simple_rate_limit/__init__.py", line 23, in wrapper open-webui | return func(*args, **kwargs) open-webui | │ │ └ {'name': '26268380-14ee-4c1a-bd25-c790c8fd3e16', 'tenant': 'default_tenant', 'database': 'default_database'} open-webui | │ └ (<chromadb.api.segment.SegmentAPI object at 0x76708dfcf110>,) open-webui | └ <function SegmentAPI.get_collection at 0x76708dba3f60> open-webui | File "/usr/local/lib/python3.11/site-packages/chromadb/api/segment.py", line 300, in get_collection open-webui | raise InvalidCollectionException(f"Collection {name} does not exist.") open-webui | └ <class 'chromadb.errors.InvalidCollectionException'> open-webui | open-webui | chromadb.errors.InvalidCollectionException: Collection 26268380-14ee-4c1a-bd25-c790c8fd3e16 does not exist. open-webui | 2025-04-03 07:16:46.123 | INFO | open_webui.retrieval.utils:query_collection_with_hybrid_search:302 - Starting hybrid search for 3 queries in 1 collections... - {} open-webui | 2025-04-03 07:16:46.124 | ERROR | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {} open-webui | Traceback (most recent call last): open-webui | open-webui | File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap open-webui | self._bootstrap_inner() open-webui | │ └ <function Thread._bootstrap_inner at 0x767101794860> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner open-webui | self.run() open-webui | │ └ <function Thread.run at 0x767101794540> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 982, in run open-webui | self._target(*self._args, **self._kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ │ │ └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,... open-webui | │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ └ <function _worker at 0x7671006b00e0> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker open-webui | work_item.run() open-webui | │ └ <function _WorkItem.run at 0x7671006b0220> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run open-webui | result = self.fn(*self.args, **self.kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0> open-webui | │ │ │ └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Was steht drin?') open-webui | │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0> open-webui | │ └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700e7455d0> open-webui | open-webui | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query open-webui | result = query_doc_with_hybrid_search( open-webui | └ <function query_doc_with_hybrid_search at 0x767083823880> open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search open-webui | raise e open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search open-webui | texts=collection_result.documents[0], open-webui | └ None open-webui | open-webui | AttributeError: 'NoneType' object has no attribute 'documents' open-webui | 2025-04-03 07:16:46.125 | ERROR | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {} open-webui | Traceback (most recent call last): open-webui | open-webui | File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap open-webui | self._bootstrap_inner() open-webui | │ └ <function Thread._bootstrap_inner at 0x767101794860> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner open-webui | self.run() open-webui | │ └ <function Thread.run at 0x767101794540> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 982, in run open-webui | self._target(*self._args, **self._kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ │ │ └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,... open-webui | │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ └ <function _worker at 0x7671006b00e0> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker open-webui | work_item.run() open-webui | │ └ <function _WorkItem.run at 0x7671006b0220> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run open-webui | result = self.fn(*self.args, **self.kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490> open-webui | │ │ │ └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Aktuelle Informationen zu Was steht drin?') open-webui | │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490> open-webui | │ └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700c4e8490> open-webui | open-webui | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query open-webui | result = query_doc_with_hybrid_search( open-webui | └ <function query_doc_with_hybrid_search at 0x767083823880> open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search open-webui | raise e open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search open-webui | texts=collection_result.documents[0], open-webui | └ None open-webui | open-webui | AttributeError: 'NoneType' object has no attribute 'documents' open-webui | 2025-04-03 07:16:46.126 | ERROR | open_webui.retrieval.utils:process_query:317 - Error when querying the collection with hybrid_search: 'NoneType' object has no attribute 'documents' - {} open-webui | Traceback (most recent call last): open-webui | open-webui | File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap open-webui | self._bootstrap_inner() open-webui | │ └ <function Thread._bootstrap_inner at 0x767101794860> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner open-webui | self.run() open-webui | │ └ <function Thread.run at 0x767101794540> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/threading.py", line 982, in run open-webui | self._target(*self._args, **self._kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ │ │ └ (<weakref at 0x76700e7b31a0; to 'ThreadPoolExecutor' at 0x76700dd75090>, <_queue.SimpleQueue object at 0x76700fbe7290>, None,... open-webui | │ │ └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | │ └ <function _worker at 0x7671006b00e0> open-webui | └ <Thread(ThreadPoolExecutor-6_0, started 130222502446784)> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker open-webui | work_item.run() open-webui | │ └ <function _WorkItem.run at 0x7671006b0220> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50> open-webui | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run open-webui | result = self.fn(*self.args, **self.kwargs) open-webui | │ │ │ │ │ └ {} open-webui | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50> open-webui | │ │ │ └ ('26268380-14ee-4c1a-bd25-c790c8fd3e16', 'Was steht drin? 2025') open-webui | │ │ └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50> open-webui | │ └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7670fdbe4400> open-webui | └ <concurrent.futures.thread._WorkItem object at 0x76700c4eaa50> open-webui | open-webui | > File "/app/backend/open_webui/retrieval/utils.py", line 305, in process_query open-webui | result = query_doc_with_hybrid_search( open-webui | └ <function query_doc_with_hybrid_search at 0x767083823880> open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search open-webui | raise e open-webui | open-webui | File "/app/backend/open_webui/retrieval/utils.py", line 120, in query_doc_with_hybrid_search open-webui | texts=collection_result.documents[0], open-webui | └ None open-webui | open-webui | AttributeError: 'NoneType' object has no attribute 'documents' Batches: 100%|██████████| 1/1 [00:00<00:00, 77.42it/s] Batches: 100%|██████████| 1/1 [00:00<00:00, 77.52it/s] Batches: 100%|██████████| 1/1 [00:00<00:00, 92.58it/s] open-webui | 2025-04-03 07:16:46.187 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "POST /api/chat/completions HTTP/1.1" 200 - {} open-webui | 2025-04-03 07:16:46.226 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 213.200.228.112:0 - "GET /api/v1/chats/?page=1 HTTP/1.1" 200 - {} ```
Author
Owner

@DrZoidberg09 commented on GitHub (Apr 3, 2025):

The collection is totally fine. It works perfectly fine without reranking / hybrid RAG.

Here are the complete logs, but I don't think they give any extra information:

open-webui-rocm | Loading WEBUI_SECRET_KEY from file, not provided as an environment variable.
open-webui-rocm | Generating WEBUI_SECRET_KEY
open-webui-rocm | Loading WEBUI_SECRET_KEY from .webui_secret_key
open-webui-rocm | USE_OLLAMA is set to true, starting ollama serve.
open-webui-rocm | CUDA is enabled, appending LD_LIBRARY_PATH to include torch/cudnn & cublas libraries.
open-webui-rocm | 2025/04/03 09:16:56 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:11.0.0 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0 http_proxy: https_proxy: no_proxy:]"
open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=images.go:432 msg="total blobs: 17"
open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.2)"
open-webui-rocm | time=2025-04-03T09:16:56.951Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
open-webui-rocm | time=2025-04-03T09:16:56.961Z level=INFO source=amd_linux.go:389 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0
open-webui-rocm | time=2025-04-03T09:16:56.965Z level=INFO source=types.go:130 msg="inference compute" id=GPU-4fe313f8c48c7ba2 library=rocm variant="" compute=gfx1100 driver=6.10 name=1002:744c total="24.0 GiB" available="24.0 GiB"
open-webui-rocm | /app/backend/open_webui
open-webui-rocm | /app/backend
open-webui-rocm | /app
open-webui-rocm | INFO [alembic.runtime.migration] Context impl SQLiteImpl.
open-webui-rocm | INFO [alembic.runtime.migration] Will assume non-transactional DDL.
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:300: SyntaxWarning: invalid escape sequence '('
open-webui-rocm | m = re.match('(sup?) (([0-9]{1,2}) bit)$', token)
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:301: SyntaxWarning: invalid escape sequence '('
open-webui-rocm | m2 = re.match('(sup?)( (default))?$', token)
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:310: SyntaxWarning: invalid escape sequence '('
open-webui-rocm | elif re.match('(flt)p?( (default))?$', token):
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:314: SyntaxWarning: invalid escape sequence '('
open-webui-rocm | elif re.match('(dbl)p?( (default))?$', token):
open-webui-rocm | WARNI [langchain_community.utils.user_agent] USER_AGENT environment variable not set, consider setting it to identify your requests.
Fetching 15 files: 100%|██████████| 15/15 [00:00<00:00, 44026.98it/s]
open-webui-rocm |
open-webui-rocm | ██████╗ ██████╗ ███████���███╗ ██╗ ██╗ ██╗███████╗██████╗ ██╗ ██╗██╗
open-webui-rocm | ██╔═══██╗██╔══██╗██╔════╝████╗ ██║ ██║ ██║██╔════╝██╔══██╗██║ ██║██║
open-webui-rocm | ██║ ██║██████╔╝█████╗ ██╔██╗ ██║ ██║ █╗ ██║█████╗ ██████╔╝██║ ██║██║
open-webui-rocm | ██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║ ██║███╗██║██╔══╝ ██╔══██╗██║ ██║██║
open-webui-rocm | ╚██████╔╝██║ ███████╗██║ ╚████║ ╚███╔███╔╝███████╗██████╔╝╚██████╔╝██║
open-webui-rocm | ╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝ ╚══╝╚══╝ ╚══════╝╚═════╝ ╚═════╝ ╚═╝
open-webui-rocm |
open-webui-rocm |
open-webui-rocm | v0.6.0 - building the best open-source AI user interface.
open-webui-rocm |
open-webui-rocm | https://github.com/open-webui/open-webui
open-webui-rocm |
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/layer_norm.py:984: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
open-webui-rocm | @custom_fwd
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/layer_norm.py:1043: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
open-webui-rocm | @custom_bwd
Fetching 24 files: 100%|██████████| 24/24 [00:00<00:00, 17788.18it/s]
open-webui-rocm | INFO: Started server process [1]
open-webui-rocm | INFO: Waiting for application startup.
open-webui-rocm | [GIN] 2025/04/03 - 09:18:29 | 200 | 2.629874ms | 127.0.0.1 | GET "/api/tags"
open-webui-rocm | [GIN] 2025/04/03 - 09:18:29 | 200 | 85.765µs | 127.0.0.1 | GET "/api/version"
Fetching 24 files: 100%|██████████| 24/24 [00:00<00:00, 106409.40it/s]
open-webui-rocm | [GIN] 2025/04/03 - 09:19:29 | 200 | 106.579µs | 127.0.0.1 | GET "/api/version"
open-webui-rocm | time=2025-04-03T09:19:35.876Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0
open-webui-rocm | time=2025-04-03T09:19:35.877Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0
open-webui-rocm | time=2025-04-03T09:19:35.878Z level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f gpu=GPU-4fe313f8c48c7ba2 parallel=1 available=21986557952 required="17.4 GiB"
open-webui-rocm | time=2025-04-03T09:19:35.878Z level=INFO source=server.go:105 msg="system memory" total="15.5 GiB" free="11.5 GiB" free_swap="4.0 GiB"
open-webui-rocm | time=2025-04-03T09:19:35.878Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0
open-webui-rocm | time=2025-04-03T09:19:35.879Z level=INFO source=server.go:138 msg=offload library=rocm layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[20.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.4 GiB" memory.required.partial="17.4 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[17.4 GiB]" memory.weights.total="12.5 GiB" memory.weights.repeating="12.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB"
open-webui-rocm | time=2025-04-03T09:19:35.879Z level=INFO source=server.go:185 msg="enabling flash attention"
open-webui-rocm | time=2025-04-03T09:19:35.879Z level=WARN source=server.go:193 msg="kv cache type not supported by model" type=""
open-webui-rocm | llama_model_loader: loaded meta data with 40 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f (version GGUF V3 (latest))
open-webui-rocm | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
open-webui-rocm | llama_model_loader: - kv 0: general.architecture str = llama
open-webui-rocm | llama_model_loader: - kv 1: general.type str = model
open-webui-rocm | llama_model_loader: - kv 2: general.name str = Mistral Small 24B Instruct 2501
open-webui-rocm | llama_model_loader: - kv 3: general.version str = 2501
open-webui-rocm | llama_model_loader: - kv 4: general.finetune str = Instruct
open-webui-rocm | llama_model_loader: - kv 5: general.basename str = Mistral-Small
open-webui-rocm | llama_model_loader: - kv 6: general.size_label str = 24B
open-webui-rocm | llama_model_loader: - kv 7: general.license str = apache-2.0
open-webui-rocm | llama_model_loader: - kv 8: general.base_model.count u32 = 1
open-webui-rocm | llama_model_loader: - kv 9: general.base_model.0.name str = Mistral Small Base 2501
open-webui-rocm | llama_model_loader: - kv 10: general.base_model.0.version str = 2501
open-webui-rocm | llama_model_loader: - kv 11: general.base_model.0.organization str = Mistralai
open-webui-rocm | llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/mistralai/Mist...
open-webui-rocm | llama_model_loader: - kv 13: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ...
open-webui-rocm | llama_model_loader: - kv 14: llama.block_count u32 = 40
open-webui-rocm | llama_model_loader: - kv 15: llama.context_length u32 = 32768
open-webui-rocm | llama_model_loader: - kv 16: llama.embedding_length u32 = 5120
open-webui-rocm | llama_model_loader: - kv 17: llama.feed_forward_length u32 = 32768
open-webui-rocm | llama_model_loader: - kv 18: llama.attention.head_count u32 = 32
open-webui-rocm | llama_model_loader: - kv 19: llama.attention.head_count_kv u32 = 8
open-webui-rocm | llama_model_loader: - kv 20: llama.rope.freq_base f32 = 100000000.000000
open-webui-rocm | llama_model_loader: - kv 21: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
open-webui-rocm | llama_model_loader: - kv 22: llama.attention.key_length u32 = 128
open-webui-rocm | llama_model_loader: - kv 23: llama.attention.value_length u32 = 128
open-webui-rocm | llama_model_loader: - kv 24: general.file_type u32 = 15
open-webui-rocm | llama_model_loader: - kv 25: llama.vocab_size u32 = 131072
open-webui-rocm | llama_model_loader: - kv 26: llama.rope.dimension_count u32 = 128
open-webui-rocm | llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2
open-webui-rocm | llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen2
open-webui-rocm | llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
open-webui-rocm | llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
open-webui-rocm | llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ ▒...
open-webui-rocm | llama_model_loader: - kv 32: tokenizer.ggml.bos_token_id u32 = 1
open-webui-rocm | llama_model_loader: - kv 33: tokenizer.ggml.eos_token_id u32 = 2
open-webui-rocm | llama_model_loader: - kv 34: tokenizer.ggml.unknown_token_id u32 = 0
open-webui-rocm | llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = true
open-webui-rocm | llama_model_loader: - kv 36: tokenizer.ggml.add_eos_token bool = false
open-webui-rocm | llama_model_loader: - kv 37: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
open-webui-rocm | llama_model_loader: - kv 38: tokenizer.ggml.add_space_prefix bool = false
open-webui-rocm | llama_model_loader: - kv 39: general.quantization_version u32 = 2
open-webui-rocm | llama_model_loader: - type f32: 81 tensors
open-webui-rocm | llama_model_loader: - type q4_K: 241 tensors
open-webui-rocm | llama_model_loader: - type q6_K: 41 tensors
open-webui-rocm | print_info: file format = GGUF V3 (latest)
open-webui-rocm | print_info: file type = Q4_K - Medium
open-webui-rocm | print_info: file size = 13.34 GiB (4.86 BPW)
open-webui-rocm | load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
open-webui-rocm | load: special tokens cache size = 1000
open-webui-rocm | load: token to piece cache size = 0.8498 MB
open-webui-rocm | print_info: arch = llama
open-webui-rocm | print_info: vocab_only = 1
open-webui-rocm | print_info: model type = ?B
open-webui-rocm | print_info: model params = 23.57 B
open-webui-rocm | print_info: general.name = Mistral Small 24B Instruct 2501
open-webui-rocm | print_info: vocab type = BPE
open-webui-rocm | print_info: n_vocab = 131072
open-webui-rocm | print_info: n_merges = 269443
open-webui-rocm | print_info: BOS token = 1 ''
open-webui-rocm | print_info: EOS token = 2 '
'
open-webui-rocm | print_info: UNK token = 0 ''
open-webui-rocm | print_info: LF token = 1010 'Ċ'
open-webui-rocm | print_info: EOG token = 2 ''
open-webui-rocm | print_info: max token length = 150
open-webui-rocm | llama_model_load: vocab only - skipping tensors
open-webui-rocm | time=2025-04-03T09:19:36.275Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f --ctx-size 16384 --batch-size 512 --n-gpu-layers 41 --threads 4 --flash-attn --no-mmap --parallel 1 --port 42673"
open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=sched.go:450 msg="loaded runners" count=1
open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
open-webui-rocm | time=2025-04-03T09:19:36.288Z level=INFO source=runner.go:846 msg="starting go runner"
open-webui-rocm | ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
open-webui-rocm | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
open-webui-rocm | ggml_cuda_init: found 1 ROCm devices:
open-webui-rocm | Device 0: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
open-webui-rocm | load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so
open-webui-rocm | load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
open-webui-rocm | time=2025-04-03T09:19:38.088Z level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
open-webui-rocm | time=2025-04-03T09:19:38.088Z level=INFO source=runner.go:906 msg="Server listening on 127.0.0.1:42673"
open-webui-rocm | llama_model_load_from_file_impl: using device ROCm0 (Radeon RX 7900 XTX) - 21086 MiB free
open-webui-rocm | llama_model_loader: loaded meta data with 40 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f (version GGUF V3 (latest))
open-webui-rocm | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
open-webui-rocm | llama_model_loader: - kv 0: general.architecture str = llama
open-webui-rocm | llama_model_loader: - kv 1: general.type str = model
open-webui-rocm | llama_model_loader: - kv 2: general.name str = Mistral Small 24B Instruct 2501
open-webui-rocm | llama_model_loader: - kv 3: general.version str = 2501
open-webui-rocm | llama_model_loader: - kv 4: general.finetune str = Instruct
open-webui-rocm | llama_model_loader: - kv 5: general.basename str = Mistral-Small
open-webui-rocm | llama_model_loader: - kv 6: general.size_label str = 24B
open-webui-rocm | llama_model_loader: - kv 7: general.license str = apache-2.0
open-webui-rocm | llama_model_loader: - kv 8: general.base_model.count u32 = 1
open-webui-rocm | llama_model_loader: - kv 9: general.base_model.0.name str = Mistral Small Base 2501
open-webui-rocm | llama_model_loader: - kv 10: general.base_model.0.version str = 2501
open-webui-rocm | llama_model_loader: - kv 11: general.base_model.0.organization str = Mistralai
open-webui-rocm | llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/mistralai/Mist...
open-webui-rocm | llama_model_loader: - kv 13: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ...
open-webui-rocm | llama_model_loader: - kv 14: llama.block_count u32 = 40
open-webui-rocm | llama_model_loader: - kv 15: llama.context_length u32 = 32768
open-webui-rocm | llama_model_loader: - kv 16: llama.embedding_length u32 = 5120
open-webui-rocm | llama_model_loader: - kv 17: llama.feed_forward_length u32 = 32768
open-webui-rocm | llama_model_loader: - kv 18: llama.attention.head_count u32 = 32
open-webui-rocm | llama_model_loader: - kv 19: llama.attention.head_count_kv u32 = 8
open-webui-rocm | llama_model_loader: - kv 20: llama.rope.freq_base f32 = 100000000.000000
open-webui-rocm | llama_model_loader: - kv 21: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
open-webui-rocm | llama_model_loader: - kv 22: llama.attention.key_length u32 = 128
open-webui-rocm | llama_model_loader: - kv 23: llama.attention.value_length u32 = 128
open-webui-rocm | llama_model_loader: - kv 24: general.file_type u32 = 15
open-webui-rocm | llama_model_loader: - kv 25: llama.vocab_size u32 = 131072
open-webui-rocm | llama_model_loader: - kv 26: llama.rope.dimension_count u32 = 128
open-webui-rocm | llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2
open-webui-rocm | llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen2
open-webui-rocm | llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
open-webui-rocm | llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
open-webui-rocm | llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ ▒...
open-webui-rocm | llama_model_loader: - kv 32: tokenizer.ggml.bos_token_id u32 = 1
open-webui-rocm | llama_model_loader: - kv 33: tokenizer.ggml.eos_token_id u32 = 2
open-webui-rocm | llama_model_loader: - kv 34: tokenizer.ggml.unknown_token_id u32 = 0
open-webui-rocm | llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = true
open-webui-rocm | llama_model_loader: - kv 36: tokenizer.ggml.add_eos_token bool = false
open-webui-rocm | llama_model_loader: - kv 37: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
open-webui-rocm | llama_model_loader: - kv 38: tokenizer.ggml.add_space_prefix bool = false
open-webui-rocm | llama_model_loader: - kv 39: general.quantization_version u32 = 2
open-webui-rocm | llama_model_loader: - type f32: 81 tensors
open-webui-rocm | llama_model_loader: - type q4_K: 241 tensors
open-webui-rocm | llama_model_loader: - type q6_K: 41 tensors
open-webui-rocm | print_info: file format = GGUF V3 (latest)
open-webui-rocm | print_info: file type = Q4_K - Medium
open-webui-rocm | print_info: file size = 13.34 GiB (4.86 BPW)
open-webui-rocm | time=2025-04-03T09:19:38.283Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
open-webui-rocm | load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
open-webui-rocm | load: special tokens cache size = 1000
open-webui-rocm | load: token to piece cache size = 0.8498 MB
open-webui-rocm | print_info: arch = llama
open-webui-rocm | print_info: vocab_only = 0
open-webui-rocm | print_info: n_ctx_train = 32768
open-webui-rocm | print_info: n_embd = 5120
open-webui-rocm | print_info: n_layer = 40
open-webui-rocm | print_info: n_head = 32
open-webui-rocm | print_info: n_head_kv = 8
open-webui-rocm | print_info: n_rot = 128
open-webui-rocm | print_info: n_swa = 0
open-webui-rocm | print_info: n_embd_head_k = 128
open-webui-rocm | print_info: n_embd_head_v = 128
open-webui-rocm | print_info: n_gqa = 4
open-webui-rocm | print_info: n_embd_k_gqa = 1024
open-webui-rocm | print_info: n_embd_v_gqa = 1024
open-webui-rocm | print_info: f_norm_eps = 0.0e+00
open-webui-rocm | print_info: f_norm_rms_eps = 1.0e-05
open-webui-rocm | print_info: f_clamp_kqv = 0.0e+00
open-webui-rocm | print_info: f_max_alibi_bias = 0.0e+00
open-webui-rocm | print_info: f_logit_scale = 0.0e+00
open-webui-rocm | print_info: n_ff = 32768
open-webui-rocm | print_info: n_expert = 0
open-webui-rocm | print_info: n_expert_used = 0
open-webui-rocm | print_info: causal attn = 1
open-webui-rocm | print_info: pooling type = 0
open-webui-rocm | print_info: rope type = 0
open-webui-rocm | print_info: rope scaling = linear
open-webui-rocm | print_info: freq_base_train = 100000000.0
open-webui-rocm | print_info: freq_scale_train = 1
open-webui-rocm | print_info: n_ctx_orig_yarn = 32768
open-webui-rocm | print_info: rope_finetuned = unknown
open-webui-rocm | print_info: ssm_d_conv = 0
open-webui-rocm | print_info: ssm_d_inner = 0
open-webui-rocm | print_info: ssm_d_state = 0
open-webui-rocm | print_info: ssm_dt_rank = 0
open-webui-rocm | print_info: ssm_dt_b_c_rms = 0
open-webui-rocm | print_info: model type = 13B
open-webui-rocm | print_info: model params = 23.57 B
open-webui-rocm | print_info: general.name = Mistral Small 24B Instruct 2501
open-webui-rocm | print_info: vocab type = BPE
open-webui-rocm | print_info: n_vocab = 131072
open-webui-rocm | print_info: n_merges = 269443
open-webui-rocm | print_info: BOS token = 1 ''
open-webui-rocm | print_info: EOS token = 2 '
'
open-webui-rocm | print_info: UNK token = 0 ''
open-webui-rocm | print_info: LF token = 1010 'Ċ'
open-webui-rocm | print_info: EOG token = 2 ''
open-webui-rocm | print_info: max token length = 150
open-webui-rocm | load_tensors: loading model tensors, this can take a while... (mmap = false)
open-webui-rocm | load_tensors: offloading 40 repeating layers to GPU
open-webui-rocm | load_tensors: offloading output layer to GPU
open-webui-rocm | load_tensors: offloaded 41/41 layers to GPU
open-webui-rocm | load_tensors: ROCm0 model buffer size = 13302.36 MiB
open-webui-rocm | load_tensors: CPU model buffer size = 360.00 MiB
open-webui-rocm | llama_init_from_model: n_seq_max = 1
open-webui-rocm | llama_init_from_model: n_ctx = 16384
open-webui-rocm | llama_init_from_model: n_ctx_per_seq = 16384
open-webui-rocm | llama_init_from_model: n_batch = 512
open-webui-rocm | llama_init_from_model: n_ubatch = 512
open-webui-rocm | llama_init_from_model: flash_attn = 1
open-webui-rocm | llama_init_from_model: freq_base = 100000000.0
open-webui-rocm | llama_init_from_model: freq_scale = 1
open-webui-rocm | llama_init_from_model: n_ctx_per_seq (16384) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
open-webui-rocm | llama_kv_cache_init: kv_size = 16384, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1
open-webui-rocm | llama_kv_cache_init: ROCm0 KV buffer size = 2560.00 MiB
open-webui-rocm | llama_init_from_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB
open-webui-rocm | llama_init_from_model: ROCm_Host output buffer size = 0.52 MiB
open-webui-rocm | llama_init_from_model: ROCm0 compute buffer size = 266.00 MiB
open-webui-rocm | llama_init_from_model: ROCm_Host compute buffer size = 42.01 MiB
open-webui-rocm | llama_init_from_model: graph nodes = 1127
open-webui-rocm | llama_init_from_model: graph splits = 2
open-webui-rocm | time=2025-04-03T09:20:12.462Z level=INFO source=server.go:619 msg="llama runner started in 36.19 seconds"
open-webui-rocm | [GIN] 2025/04/03 - 09:20:13 | 200 | 38.184212593s | 127.0.0.1 | POST "/api/chat"
open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/__init__.py:30: UserWarning: Importing debug from langchain root module is no longer supported. Please use langchain.globals.set_debug() / langchain.globals.get_debug() instead.
open-webui-rocm | warnings.warn(
open-webui-rocm | 2025-04-03 09:20:56.059 | ERROR | open_webui.retrieval.utils:process_query:320 - Error when querying the collection with hybrid_search: - {}
open-webui-rocm | Traceback (most recent call last):
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1032, in _bootstrap
open-webui-rocm | self._bootstrap_inner()
open-webui-rocm | │ └ <function Thread._bootstrap_inner at 0x7e10af439580>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
open-webui-rocm | self.run()
open-webui-rocm | │ └ <function Thread.run at 0x7e10af439260>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1012, in run
open-webui-rocm | self._target(*self._args, **self._kwargs)
open-webui-rocm | │ │ │ │ │ └ {}
open-webui-rocm | │ │ │ │ └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)>
open-webui-rocm | │ │ │ └ (<weakref at 0x7e0d745370b0; to 'ThreadPoolExecutor' at 0x7e0d3dbe34a0>, <_queue.SimpleQueue object at 0x7e0d745e3f10>, None,...
open-webui-rocm | │ │ └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)>
open-webui-rocm | │ └ <function _worker at 0x7e10ae2f1300>
open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker
open-webui-rocm | work_item.run()
open-webui-rocm | │ └ <function _WorkItem.run at 0x7e10ae2f1440>
open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30>
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 59, in run
open-webui-rocm | result = self.fn(*self.args, **self.kwargs)
open-webui-rocm | │ │ │ │ │ └ {}
open-webui-rocm | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30>
open-webui-rocm | │ │ │ └ ('ab37fb86-7213-422c-b37d-01c83b58e6e5', 'nonwoven research summaries')
open-webui-rocm | │ │ └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30>
open-webui-rocm | │ └ <function query_collection_with_hybrid_search..process_query at 0x7e0d363b0400>
open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30>
open-webui-rocm |
open-webui-rocm | > File "/app/backend/open_webui/retrieval/utils.py", line 308, in process_query
open-webui-rocm | result = query_doc_with_hybrid_search(
open-webui-rocm | └ <function query_doc_with_hybrid_search at 0x7e0d6bd54040>
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search
open-webui-rocm | raise e
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 145, in query_doc_with_hybrid_search
open-webui-rocm | result = compression_retriever.invoke(query)
open-webui-rocm | │ │ └ 'nonwoven research summaries'
open-webui-rocm | │ └ <function BaseRetriever.invoke at 0x7e0d6c297b00>
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke
open-webui-rocm | result = self._get_relevant_documents(
open-webui-rocm | │ └ <function ContextualCompressionRetriever._get_relevant_documents at 0x7e0d6c297740>
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents
open-webui-rocm | docs = self.base_retriever.invoke(
open-webui-rocm | │ │ └ <function EnsembleRetriever.invoke at 0x7e0d6bfb00e0>
open-webui-rocm | │ └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7e0d3497ab70>, k=30), VectorSearchRet...
open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 118, in invoke
open-webui-rocm | raise e
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 115, in invoke
open-webui-rocm | result = self.rank_fusion(input, run_manager=run_manager, config=config)
open-webui-rocm | │ │ │ │ └ {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x7e0ccb0bc200>, 'recur...
open-webui-rocm | │ │ │ └ <langchain_core.callbacks.manager.CallbackManagerForRetrieverRun object at 0x7e0ccb0bcf80>
open-webui-rocm | │ │ └ 'nonwoven research summaries'
open-webui-rocm | │ └ <function EnsembleRetriever.rank_fusion at 0x7e0d6bfb0400>
open-webui-rocm | └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7e0d3497ab70>, k=30), VectorSearchRet...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 223, in rank_fusion
open-webui-rocm | retriever.invoke(
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke
open-webui-rocm | result = self._get_relevant_documents(
open-webui-rocm | │ └ <function VectorSearchRetriever._get_relevant_documents at 0x7e0d6bd83ec0>
open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi...
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 57, in _get_relevant_documents
open-webui-rocm | vectors=[self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)],
open-webui-rocm | │ │ │ └ None
open-webui-rocm | │ │ └ 'nonwoven research summaries'
open-webui-rocm | │ └ <function chat_completion_files_handler.... at 0x7e0d3457a700>
open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi...
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/utils/middleware.py", line 598, in
open-webui-rocm | embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION(
open-webui-rocm | │ │ │ └ <property object at 0x7e10abba62a0>
open-webui-rocm | │ │ └ <starlette.requests.Request object at 0x7e0d3418b5c0>
open-webui-rocm | │ └ None
open-webui-rocm | └ 'nonwoven research summaries'
open-webui-rocm |
open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 356, in
open-webui-rocm | return lambda query, prefix=None, user=None: embedding_function.encode(
open-webui-rocm | │ │ └ <function SentenceTransformer.encode at 0x7e0d3f260d60>
open-webui-rocm | │ └ SentenceTransformer(
open-webui-rocm | │ (transformer): Transformer(
open-webui-rocm | │ (auto_model): XLMRobertaLoRA(
open-webui-rocm | │ (roberta): XLMRobertaModel(
open-webui-rocm | │ ...
open-webui-rocm | └ 'nonwoven research summaries'
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 557, in encode
open-webui-rocm | out_features = self.forward(features, **kwargs)
open-webui-rocm | │ │ │ └ {}
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | │ └ <function SentenceTransformer.forward at 0x7e0d3f260e00>
open-webui-rocm | └ SentenceTransformer(
open-webui-rocm | (transformer): Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 624, in forward
open-webui-rocm | input = module(input, **module_kwargs)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {}
open-webui-rocm | │ │ └ ({'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': ...
open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ ({'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ device='cuda:0'), 'attention_mask': ...
open-webui-rocm | └ <bound method Transformer.forward of Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embe...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/f1944de8402dcd5f2b03f822a4bc22a7f2de2eb9/custom_st.py", line 143, in forward
open-webui-rocm | output_states = self.auto_model.forward(**features, **lora_arguments, return_dict=False)
open-webui-rocm | │ │ └ {}
open-webui-rocm | │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | └ Transformer(
open-webui-rocm | (auto_model): XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | ...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_lora.py", line 372, in forward
open-webui-rocm | return self.roberta(*args, **kwargs)
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | │ └ ()
open-webui-rocm | └ XLMRobertaLoRA(
open-webui-rocm | (roberta): XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEm...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ │ │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | │ │ └ ()
open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEmbedding(
open-webui-rocm | 250002, 1024, padd...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'input_ids': tensor( 0, 351, 3613, 1353, 25188, 29334, 10484, 2,
open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t...
open-webui-rocm | │ └ ()
open-webui-rocm | └ <bound method XLMRobertaModel.forward of XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): Parame...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 734, in forward
open-webui-rocm | sequence_output = self.encoder(
open-webui-rocm | └ XLMRobertaModel(
open-webui-rocm | (embeddings): XLMRobertaEmbeddings(
open-webui-rocm | (word_embeddings): ParametrizedEmbedding(
open-webui-rocm | 250002, 1024, padd...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self.call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'key_padding_mask': tensor(1, 1, 1, 1, 1, 1, 1, 1, device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output
...
open-webui-rocm | │ │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ...
open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ XLMRobertaEncoder(
open-webui-rocm | (layers): ModuleList(
open-webui-rocm | (0-23): 24 x Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding(...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'key_padding_mask': tensor(1, 1, 1, 1, 1, 1, 1, 1, device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output
...
open-webui-rocm | │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ...
open-webui-rocm | └ <bound method XLMRobertaEncoder.forward of XLMRobertaEncoder(
open-webui-rocm | (layers): ModuleList(
open-webui-rocm | (0-23): 24 x Block(
open-webui-rocm | (mixer): ...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 258, in forward
open-webui-rocm | hidden_states = layer(hidden_states, mixer_kwargs=mixer_kwargs)
open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}
open-webui-rocm | │ └ tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.0...
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}}
open-webui-rocm | │ │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0....
open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}}
open-webui-rocm | │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0....
open-webui-rocm | └ <bound method Block.forward of Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResid...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/block.py", line 201, in forward
open-webui-rocm | mixer_out = self.mixer(
open-webui-rocm | └ Block(
open-webui-rocm | (mixer): MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, ou...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
open-webui-rocm | return self._call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}
open-webui-rocm | │ │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0....
open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, out_features=3072, bias=Tr...
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}
open-webui-rocm | │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352],
open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0....
open-webui-rocm | └ <bound method MHA.forward of MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=102...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/mha.py", line 732, in forward
open-webui-rocm | qkv = self.rotary_emb(
open-webui-rocm | └ MHA(
open-webui-rocm | (rotary_emb): RotaryEmbedding()
open-webui-rocm | (Wqkv): ParametrizedLinearResidual(
open-webui-rocm | in_features=1024, out_features=3072, bias=Tr...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in wrapped_call_impl
open-webui-rocm | return self.call_impl(*args, **kwargs)
open-webui-rocm | │ │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8}
open-webui-rocm | │ │ └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01,
open-webui-rocm | │ │ 1.3125e+00, 1.0781e+00],
open-webui-rocm | │ │ [-6.796...
open-webui-rocm | │ └ <function Module.call_impl at 0x7e0e7f720f40>
open-webui-rocm | └ RotaryEmbedding()
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in call_impl
open-webui-rocm | return forward_call(*args, **kwargs)
open-webui-rocm | │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8}
open-webui-rocm | │ └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01,
open-webui-rocm | │ 1.3125e+00, 1.0781e+00],
open-webui-rocm | │ [-6.796...
open-webui-rocm | └ <bound method RotaryEmbedding.forward of RotaryEmbedding()>
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 604, in forward
open-webui-rocm | return apply_rotary_emb_qkv
(
open-webui-rocm | └ <function apply_rotary_emb_qkv
at 0x7e0d3da023e0>
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 327, in apply_rotary_emb_qkv

open-webui-rocm | return ApplyRotaryEmbQKV
.apply(
open-webui-rocm | │ └ <classmethod(<function Function.apply at 0x7e0e7f499760>)>
open-webui-rocm | └ <class 'transformers_modules.jinaai.xlm-roberta-flash-implementation.2b6bc3f30750b3a9648fe9b63448c09920efe9be.rotary.ApplyRot...
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply
open-webui-rocm | return super().apply(*args, **kwargs) # type: ignore[misc]
open-webui-rocm | │ └ {}
open-webui-rocm | └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01,
open-webui-rocm | 1.3125e+00, 1.0781e+00],
open-webui-rocm | [-6.796...
open-webui-rocm |
open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 186, in forward
open-webui-rocm | apply_rotary(
open-webui-rocm | └ <function apply_rotary at 0x7e0d3da01ee0>
open-webui-rocm |
open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary
open-webui-rocm | assert sin.shape == cos.shape
open-webui-rocm | │ │ │ └ <attribute 'shape' of 'torch._C.TensorBase' objects>
open-webui-rocm | │ │ └ tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000,
open-webui-rocm | │ │ 1.0000, 1.0000, 1.0000, 1.0000,...
open-webui-rocm | │ └ <attribute 'shape' of 'torch._C.TensorBase' objects>
open-webui-rocm | └ tensor([[ 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
open-webui-rocm | 0.0000e+00, 0.0000e+00, 0.0000e+00, 0....
open-webui-rocm |
open-webui-rocm | AssertionError: assert sin.shape == cos.shape
open-webui-rocm | [GIN] 2025/04/03 - 09:21:11 | 200 | 12.946911352s | 127.0.0.1 | POST "/api/chat"
open-webui-rocm | [GIN] 2025/04/03 - 09:21:13 | 200 | 1.468138859s | 127.0.0.1 | POST "/api/chat"
open-webui-rocm | [GIN] 2025/04/03 - 09:21:15 | 200 | 1.365315684s | 127.0.0.1 | POST "/api/chat"

<!-- gh-comment-id:2775053127 --> @DrZoidberg09 commented on GitHub (Apr 3, 2025): The collection is totally fine. It works perfectly fine without reranking / hybrid RAG. Here are the complete logs, but I don't think they give any extra information: > open-webui-rocm | Loading WEBUI_SECRET_KEY from file, not provided as an environment variable. open-webui-rocm | Generating WEBUI_SECRET_KEY open-webui-rocm | Loading WEBUI_SECRET_KEY from .webui_secret_key open-webui-rocm | USE_OLLAMA is set to true, starting ollama serve. open-webui-rocm | CUDA is enabled, appending LD_LIBRARY_PATH to include torch/cudnn & cublas libraries. open-webui-rocm | 2025/04/03 09:16:56 routes.go:1230: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:11.0.0 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:0 http_proxy: https_proxy: no_proxy:]" open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=images.go:432 msg="total blobs: 17" open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" open-webui-rocm | time=2025-04-03T09:16:56.950Z level=INFO source=routes.go:1297 msg="Listening on 127.0.0.1:11434 (version 0.6.2)" open-webui-rocm | time=2025-04-03T09:16:56.951Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" open-webui-rocm | time=2025-04-03T09:16:56.961Z level=INFO source=amd_linux.go:389 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=11.0.0 open-webui-rocm | time=2025-04-03T09:16:56.965Z level=INFO source=types.go:130 msg="inference compute" id=GPU-4fe313f8c48c7ba2 library=rocm variant="" compute=gfx1100 driver=6.10 name=1002:744c total="24.0 GiB" available="24.0 GiB" open-webui-rocm | /app/backend/open_webui open-webui-rocm | /app/backend open-webui-rocm | /app open-webui-rocm | INFO [alembic.runtime.migration] Context impl SQLiteImpl. open-webui-rocm | INFO [alembic.runtime.migration] Will assume non-transactional DDL. open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:300: SyntaxWarning: invalid escape sequence '\(' open-webui-rocm | m = re.match('([su]([0-9]{1,2})p?) \(([0-9]{1,2}) bit\)$', token) open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:301: SyntaxWarning: invalid escape sequence '\(' open-webui-rocm | m2 = re.match('([su]([0-9]{1,2})p?)( \(default\))?$', token) open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:310: SyntaxWarning: invalid escape sequence '\(' open-webui-rocm | elif re.match('(flt)p?( \(default\))?$', token): open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/pydub/utils.py:314: SyntaxWarning: invalid escape sequence '\(' open-webui-rocm | elif re.match('(dbl)p?( \(default\))?$', token): open-webui-rocm | WARNI [langchain_community.utils.user_agent] USER_AGENT environment variable not set, consider setting it to identify your requests. Fetching 15 files: 100%|██████████| 15/15 [00:00<00:00, 44026.98it/s] open-webui-rocm | open-webui-rocm | ██████╗ ██████╗ ███████���███╗ ██╗ ██╗ ██╗███████╗██████╗ ██╗ ██╗██╗ open-webui-rocm | ██╔═══██╗██╔══██╗██╔════╝████╗ ██║ ██║ ██║██╔════╝██╔══██╗██║ ██║██║ open-webui-rocm | ██║ ██║██████╔╝█████╗ ██╔██╗ ██║ ██║ █╗ ██║█████╗ ██████╔╝██║ ██║██║ open-webui-rocm | ██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║ ██║███╗██║██╔══╝ ██╔══██╗██║ ██║██║ open-webui-rocm | ╚██████╔╝██║ ███████╗██║ ╚████║ ╚███╔███╔╝███████╗██████╔╝╚██████╔╝██║ open-webui-rocm | ╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝ ╚══╝╚══╝ ╚══════╝╚═════╝ ╚═════╝ ╚═╝ open-webui-rocm | open-webui-rocm | open-webui-rocm | v0.6.0 - building the best open-source AI user interface. open-webui-rocm | open-webui-rocm | https://github.com/open-webui/open-webui open-webui-rocm | open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/layer_norm.py:984: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. open-webui-rocm | @custom_fwd open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/layer_norm.py:1043: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead. open-webui-rocm | @custom_bwd Fetching 24 files: 100%|██████████| 24/24 [00:00<00:00, 17788.18it/s] open-webui-rocm | INFO: Started server process [1] open-webui-rocm | INFO: Waiting for application startup. open-webui-rocm | [GIN] 2025/04/03 - 09:18:29 | 200 | 2.629874ms | 127.0.0.1 | GET "/api/tags" open-webui-rocm | [GIN] 2025/04/03 - 09:18:29 | 200 | 85.765µs | 127.0.0.1 | GET "/api/version" Fetching 24 files: 100%|██████████| 24/24 [00:00<00:00, 106409.40it/s] open-webui-rocm | [GIN] 2025/04/03 - 09:19:29 | 200 | 106.579µs | 127.0.0.1 | GET "/api/version" open-webui-rocm | time=2025-04-03T09:19:35.876Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0 open-webui-rocm | time=2025-04-03T09:19:35.877Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0 open-webui-rocm | time=2025-04-03T09:19:35.878Z level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f gpu=GPU-4fe313f8c48c7ba2 parallel=1 available=21986557952 required="17.4 GiB" open-webui-rocm | time=2025-04-03T09:19:35.878Z level=INFO source=server.go:105 msg="system memory" total="15.5 GiB" free="11.5 GiB" free_swap="4.0 GiB" open-webui-rocm | time=2025-04-03T09:19:35.878Z level=WARN source=ggml.go:149 msg="key not found" key=llama.vision.block_count default=0 open-webui-rocm | time=2025-04-03T09:19:35.879Z level=INFO source=server.go:138 msg=offload library=rocm layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[20.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.4 GiB" memory.required.partial="17.4 GiB" memory.required.kv="2.5 GiB" memory.required.allocations="[17.4 GiB]" memory.weights.total="12.5 GiB" memory.weights.repeating="12.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB" open-webui-rocm | time=2025-04-03T09:19:35.879Z level=INFO source=server.go:185 msg="enabling flash attention" open-webui-rocm | time=2025-04-03T09:19:35.879Z level=WARN source=server.go:193 msg="kv cache type not supported by model" type="" open-webui-rocm | llama_model_loader: loaded meta data with 40 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f (version GGUF V3 (latest)) open-webui-rocm | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. open-webui-rocm | llama_model_loader: - kv 0: general.architecture str = llama open-webui-rocm | llama_model_loader: - kv 1: general.type str = model open-webui-rocm | llama_model_loader: - kv 2: general.name str = Mistral Small 24B Instruct 2501 open-webui-rocm | llama_model_loader: - kv 3: general.version str = 2501 open-webui-rocm | llama_model_loader: - kv 4: general.finetune str = Instruct open-webui-rocm | llama_model_loader: - kv 5: general.basename str = Mistral-Small open-webui-rocm | llama_model_loader: - kv 6: general.size_label str = 24B open-webui-rocm | llama_model_loader: - kv 7: general.license str = apache-2.0 open-webui-rocm | llama_model_loader: - kv 8: general.base_model.count u32 = 1 open-webui-rocm | llama_model_loader: - kv 9: general.base_model.0.name str = Mistral Small Base 2501 open-webui-rocm | llama_model_loader: - kv 10: general.base_model.0.version str = 2501 open-webui-rocm | llama_model_loader: - kv 11: general.base_model.0.organization str = Mistralai open-webui-rocm | llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/mistralai/Mist... open-webui-rocm | llama_model_loader: - kv 13: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ... open-webui-rocm | llama_model_loader: - kv 14: llama.block_count u32 = 40 open-webui-rocm | llama_model_loader: - kv 15: llama.context_length u32 = 32768 open-webui-rocm | llama_model_loader: - kv 16: llama.embedding_length u32 = 5120 open-webui-rocm | llama_model_loader: - kv 17: llama.feed_forward_length u32 = 32768 open-webui-rocm | llama_model_loader: - kv 18: llama.attention.head_count u32 = 32 open-webui-rocm | llama_model_loader: - kv 19: llama.attention.head_count_kv u32 = 8 open-webui-rocm | llama_model_loader: - kv 20: llama.rope.freq_base f32 = 100000000.000000 open-webui-rocm | llama_model_loader: - kv 21: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 open-webui-rocm | llama_model_loader: - kv 22: llama.attention.key_length u32 = 128 open-webui-rocm | llama_model_loader: - kv 23: llama.attention.value_length u32 = 128 open-webui-rocm | llama_model_loader: - kv 24: general.file_type u32 = 15 open-webui-rocm | llama_model_loader: - kv 25: llama.vocab_size u32 = 131072 open-webui-rocm | llama_model_loader: - kv 26: llama.rope.dimension_count u32 = 128 open-webui-rocm | llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2 open-webui-rocm | llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen2 open-webui-rocm | llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[... open-webui-rocm | llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... open-webui-rocm | llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ ▒... open-webui-rocm | llama_model_loader: - kv 32: tokenizer.ggml.bos_token_id u32 = 1 open-webui-rocm | llama_model_loader: - kv 33: tokenizer.ggml.eos_token_id u32 = 2 open-webui-rocm | llama_model_loader: - kv 34: tokenizer.ggml.unknown_token_id u32 = 0 open-webui-rocm | llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = true open-webui-rocm | llama_model_loader: - kv 36: tokenizer.ggml.add_eos_token bool = false open-webui-rocm | llama_model_loader: - kv 37: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... open-webui-rocm | llama_model_loader: - kv 38: tokenizer.ggml.add_space_prefix bool = false open-webui-rocm | llama_model_loader: - kv 39: general.quantization_version u32 = 2 open-webui-rocm | llama_model_loader: - type f32: 81 tensors open-webui-rocm | llama_model_loader: - type q4_K: 241 tensors open-webui-rocm | llama_model_loader: - type q6_K: 41 tensors open-webui-rocm | print_info: file format = GGUF V3 (latest) open-webui-rocm | print_info: file type = Q4_K - Medium open-webui-rocm | print_info: file size = 13.34 GiB (4.86 BPW) open-webui-rocm | load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect open-webui-rocm | load: special tokens cache size = 1000 open-webui-rocm | load: token to piece cache size = 0.8498 MB open-webui-rocm | print_info: arch = llama open-webui-rocm | print_info: vocab_only = 1 open-webui-rocm | print_info: model type = ?B open-webui-rocm | print_info: model params = 23.57 B open-webui-rocm | print_info: general.name = Mistral Small 24B Instruct 2501 open-webui-rocm | print_info: vocab type = BPE open-webui-rocm | print_info: n_vocab = 131072 open-webui-rocm | print_info: n_merges = 269443 open-webui-rocm | print_info: BOS token = 1 '<s>' open-webui-rocm | print_info: EOS token = 2 '</s>' open-webui-rocm | print_info: UNK token = 0 '<unk>' open-webui-rocm | print_info: LF token = 1010 'Ċ' open-webui-rocm | print_info: EOG token = 2 '</s>' open-webui-rocm | print_info: max token length = 150 open-webui-rocm | llama_model_load: vocab only - skipping tensors open-webui-rocm | time=2025-04-03T09:19:36.275Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f --ctx-size 16384 --batch-size 512 --n-gpu-layers 41 --threads 4 --flash-attn --no-mmap --parallel 1 --port 42673" open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=sched.go:450 msg="loaded runners" count=1 open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding" open-webui-rocm | time=2025-04-03T09:19:36.276Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" open-webui-rocm | time=2025-04-03T09:19:36.288Z level=INFO source=runner.go:846 msg="starting go runner" open-webui-rocm | ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no open-webui-rocm | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no open-webui-rocm | ggml_cuda_init: found 1 ROCm devices: open-webui-rocm | Device 0: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32 open-webui-rocm | load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so open-webui-rocm | load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so open-webui-rocm | time=2025-04-03T09:19:38.088Z level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) open-webui-rocm | time=2025-04-03T09:19:38.088Z level=INFO source=runner.go:906 msg="Server listening on 127.0.0.1:42673" open-webui-rocm | llama_model_load_from_file_impl: using device ROCm0 (Radeon RX 7900 XTX) - 21086 MiB free open-webui-rocm | llama_model_loader: loaded meta data with 40 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-102a747c137683e81d431dab05d8f2158df4ab6f162f8f9019425a43d51e0e9f (version GGUF V3 (latest)) open-webui-rocm | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. open-webui-rocm | llama_model_loader: - kv 0: general.architecture str = llama open-webui-rocm | llama_model_loader: - kv 1: general.type str = model open-webui-rocm | llama_model_loader: - kv 2: general.name str = Mistral Small 24B Instruct 2501 open-webui-rocm | llama_model_loader: - kv 3: general.version str = 2501 open-webui-rocm | llama_model_loader: - kv 4: general.finetune str = Instruct open-webui-rocm | llama_model_loader: - kv 5: general.basename str = Mistral-Small open-webui-rocm | llama_model_loader: - kv 6: general.size_label str = 24B open-webui-rocm | llama_model_loader: - kv 7: general.license str = apache-2.0 open-webui-rocm | llama_model_loader: - kv 8: general.base_model.count u32 = 1 open-webui-rocm | llama_model_loader: - kv 9: general.base_model.0.name str = Mistral Small Base 2501 open-webui-rocm | llama_model_loader: - kv 10: general.base_model.0.version str = 2501 open-webui-rocm | llama_model_loader: - kv 11: general.base_model.0.organization str = Mistralai open-webui-rocm | llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/mistralai/Mist... open-webui-rocm | llama_model_loader: - kv 13: general.languages arr[str,10] = ["en", "fr", "de", "es", "it", "pt", ... open-webui-rocm | llama_model_loader: - kv 14: llama.block_count u32 = 40 open-webui-rocm | llama_model_loader: - kv 15: llama.context_length u32 = 32768 open-webui-rocm | llama_model_loader: - kv 16: llama.embedding_length u32 = 5120 open-webui-rocm | llama_model_loader: - kv 17: llama.feed_forward_length u32 = 32768 open-webui-rocm | llama_model_loader: - kv 18: llama.attention.head_count u32 = 32 open-webui-rocm | llama_model_loader: - kv 19: llama.attention.head_count_kv u32 = 8 open-webui-rocm | llama_model_loader: - kv 20: llama.rope.freq_base f32 = 100000000.000000 open-webui-rocm | llama_model_loader: - kv 21: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 open-webui-rocm | llama_model_loader: - kv 22: llama.attention.key_length u32 = 128 open-webui-rocm | llama_model_loader: - kv 23: llama.attention.value_length u32 = 128 open-webui-rocm | llama_model_loader: - kv 24: general.file_type u32 = 15 open-webui-rocm | llama_model_loader: - kv 25: llama.vocab_size u32 = 131072 open-webui-rocm | llama_model_loader: - kv 26: llama.rope.dimension_count u32 = 128 open-webui-rocm | llama_model_loader: - kv 27: tokenizer.ggml.model str = gpt2 open-webui-rocm | llama_model_loader: - kv 28: tokenizer.ggml.pre str = qwen2 open-webui-rocm | llama_model_loader: - kv 29: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[... open-webui-rocm | llama_model_loader: - kv 30: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... open-webui-rocm | llama_model_loader: - kv 31: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ ▒... open-webui-rocm | llama_model_loader: - kv 32: tokenizer.ggml.bos_token_id u32 = 1 open-webui-rocm | llama_model_loader: - kv 33: tokenizer.ggml.eos_token_id u32 = 2 open-webui-rocm | llama_model_loader: - kv 34: tokenizer.ggml.unknown_token_id u32 = 0 open-webui-rocm | llama_model_loader: - kv 35: tokenizer.ggml.add_bos_token bool = true open-webui-rocm | llama_model_loader: - kv 36: tokenizer.ggml.add_eos_token bool = false open-webui-rocm | llama_model_loader: - kv 37: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... open-webui-rocm | llama_model_loader: - kv 38: tokenizer.ggml.add_space_prefix bool = false open-webui-rocm | llama_model_loader: - kv 39: general.quantization_version u32 = 2 open-webui-rocm | llama_model_loader: - type f32: 81 tensors open-webui-rocm | llama_model_loader: - type q4_K: 241 tensors open-webui-rocm | llama_model_loader: - type q6_K: 41 tensors open-webui-rocm | print_info: file format = GGUF V3 (latest) open-webui-rocm | print_info: file type = Q4_K - Medium open-webui-rocm | print_info: file size = 13.34 GiB (4.86 BPW) open-webui-rocm | time=2025-04-03T09:19:38.283Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" open-webui-rocm | load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect open-webui-rocm | load: special tokens cache size = 1000 open-webui-rocm | load: token to piece cache size = 0.8498 MB open-webui-rocm | print_info: arch = llama open-webui-rocm | print_info: vocab_only = 0 open-webui-rocm | print_info: n_ctx_train = 32768 open-webui-rocm | print_info: n_embd = 5120 open-webui-rocm | print_info: n_layer = 40 open-webui-rocm | print_info: n_head = 32 open-webui-rocm | print_info: n_head_kv = 8 open-webui-rocm | print_info: n_rot = 128 open-webui-rocm | print_info: n_swa = 0 open-webui-rocm | print_info: n_embd_head_k = 128 open-webui-rocm | print_info: n_embd_head_v = 128 open-webui-rocm | print_info: n_gqa = 4 open-webui-rocm | print_info: n_embd_k_gqa = 1024 open-webui-rocm | print_info: n_embd_v_gqa = 1024 open-webui-rocm | print_info: f_norm_eps = 0.0e+00 open-webui-rocm | print_info: f_norm_rms_eps = 1.0e-05 open-webui-rocm | print_info: f_clamp_kqv = 0.0e+00 open-webui-rocm | print_info: f_max_alibi_bias = 0.0e+00 open-webui-rocm | print_info: f_logit_scale = 0.0e+00 open-webui-rocm | print_info: n_ff = 32768 open-webui-rocm | print_info: n_expert = 0 open-webui-rocm | print_info: n_expert_used = 0 open-webui-rocm | print_info: causal attn = 1 open-webui-rocm | print_info: pooling type = 0 open-webui-rocm | print_info: rope type = 0 open-webui-rocm | print_info: rope scaling = linear open-webui-rocm | print_info: freq_base_train = 100000000.0 open-webui-rocm | print_info: freq_scale_train = 1 open-webui-rocm | print_info: n_ctx_orig_yarn = 32768 open-webui-rocm | print_info: rope_finetuned = unknown open-webui-rocm | print_info: ssm_d_conv = 0 open-webui-rocm | print_info: ssm_d_inner = 0 open-webui-rocm | print_info: ssm_d_state = 0 open-webui-rocm | print_info: ssm_dt_rank = 0 open-webui-rocm | print_info: ssm_dt_b_c_rms = 0 open-webui-rocm | print_info: model type = 13B open-webui-rocm | print_info: model params = 23.57 B open-webui-rocm | print_info: general.name = Mistral Small 24B Instruct 2501 open-webui-rocm | print_info: vocab type = BPE open-webui-rocm | print_info: n_vocab = 131072 open-webui-rocm | print_info: n_merges = 269443 open-webui-rocm | print_info: BOS token = 1 '<s>' open-webui-rocm | print_info: EOS token = 2 '</s>' open-webui-rocm | print_info: UNK token = 0 '<unk>' open-webui-rocm | print_info: LF token = 1010 'Ċ' open-webui-rocm | print_info: EOG token = 2 '</s>' open-webui-rocm | print_info: max token length = 150 open-webui-rocm | load_tensors: loading model tensors, this can take a while... (mmap = false) open-webui-rocm | load_tensors: offloading 40 repeating layers to GPU open-webui-rocm | load_tensors: offloading output layer to GPU open-webui-rocm | load_tensors: offloaded 41/41 layers to GPU open-webui-rocm | load_tensors: ROCm0 model buffer size = 13302.36 MiB open-webui-rocm | load_tensors: CPU model buffer size = 360.00 MiB open-webui-rocm | llama_init_from_model: n_seq_max = 1 open-webui-rocm | llama_init_from_model: n_ctx = 16384 open-webui-rocm | llama_init_from_model: n_ctx_per_seq = 16384 open-webui-rocm | llama_init_from_model: n_batch = 512 open-webui-rocm | llama_init_from_model: n_ubatch = 512 open-webui-rocm | llama_init_from_model: flash_attn = 1 open-webui-rocm | llama_init_from_model: freq_base = 100000000.0 open-webui-rocm | llama_init_from_model: freq_scale = 1 open-webui-rocm | llama_init_from_model: n_ctx_per_seq (16384) < n_ctx_train (32768) -- the full capacity of the model will not be utilized open-webui-rocm | llama_kv_cache_init: kv_size = 16384, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1 open-webui-rocm | llama_kv_cache_init: ROCm0 KV buffer size = 2560.00 MiB open-webui-rocm | llama_init_from_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB open-webui-rocm | llama_init_from_model: ROCm_Host output buffer size = 0.52 MiB open-webui-rocm | llama_init_from_model: ROCm0 compute buffer size = 266.00 MiB open-webui-rocm | llama_init_from_model: ROCm_Host compute buffer size = 42.01 MiB open-webui-rocm | llama_init_from_model: graph nodes = 1127 open-webui-rocm | llama_init_from_model: graph splits = 2 open-webui-rocm | time=2025-04-03T09:20:12.462Z level=INFO source=server.go:619 msg="llama runner started in 36.19 seconds" open-webui-rocm | [GIN] 2025/04/03 - 09:20:13 | 200 | 38.184212593s | 127.0.0.1 | POST "/api/chat" open-webui-rocm | /opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/__init__.py:30: UserWarning: Importing debug from langchain root module is no longer supported. Please use langchain.globals.set_debug() / langchain.globals.get_debug() instead. open-webui-rocm | warnings.warn( open-webui-rocm | 2025-04-03 09:20:56.059 | ERROR | open_webui.retrieval.utils:process_query:320 - Error when querying the collection with hybrid_search: - {} open-webui-rocm | Traceback (most recent call last): open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1032, in _bootstrap open-webui-rocm | self._bootstrap_inner() open-webui-rocm | │ └ <function Thread._bootstrap_inner at 0x7e10af439580> open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1075, in _bootstrap_inner open-webui-rocm | self.run() open-webui-rocm | │ └ <function Thread.run at 0x7e10af439260> open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/threading.py", line 1012, in run open-webui-rocm | self._target(*self._args, **self._kwargs) open-webui-rocm | │ │ │ │ │ └ {} open-webui-rocm | │ │ │ │ └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)> open-webui-rocm | │ │ │ └ (<weakref at 0x7e0d745370b0; to 'ThreadPoolExecutor' at 0x7e0d3dbe34a0>, <_queue.SimpleQueue object at 0x7e0d745e3f10>, None,... open-webui-rocm | │ │ └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)> open-webui-rocm | │ └ <function _worker at 0x7e10ae2f1300> open-webui-rocm | └ <Thread(ThreadPoolExecutor-5_1, started 138590835181248)> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 93, in _worker open-webui-rocm | work_item.run() open-webui-rocm | │ └ <function _WorkItem.run at 0x7e10ae2f1440> open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30> open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/concurrent/futures/thread.py", line 59, in run open-webui-rocm | result = self.fn(*self.args, **self.kwargs) open-webui-rocm | │ │ │ │ │ └ {} open-webui-rocm | │ │ │ │ └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30> open-webui-rocm | │ │ │ └ ('ab37fb86-7213-422c-b37d-01c83b58e6e5', 'nonwoven research summaries') open-webui-rocm | │ │ └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30> open-webui-rocm | │ └ <function query_collection_with_hybrid_search.<locals>.process_query at 0x7e0d363b0400> open-webui-rocm | └ <concurrent.futures.thread._WorkItem object at 0x7e0d344afb30> open-webui-rocm | open-webui-rocm | > File "/app/backend/open_webui/retrieval/utils.py", line 308, in process_query open-webui-rocm | result = query_doc_with_hybrid_search( open-webui-rocm | └ <function query_doc_with_hybrid_search at 0x7e0d6bd54040> open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 171, in query_doc_with_hybrid_search open-webui-rocm | raise e open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 145, in query_doc_with_hybrid_search open-webui-rocm | result = compression_retriever.invoke(query) open-webui-rocm | │ │ └ 'nonwoven research summaries' open-webui-rocm | │ └ <function BaseRetriever.invoke at 0x7e0d6c297b00> open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke open-webui-rocm | result = self._get_relevant_documents( open-webui-rocm | │ └ <function ContextualCompressionRetriever._get_relevant_documents at 0x7e0d6c297740> open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents open-webui-rocm | docs = self.base_retriever.invoke( open-webui-rocm | │ │ └ <function EnsembleRetriever.invoke at 0x7e0d6bfb00e0> open-webui-rocm | │ └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7e0d3497ab70>, k=30), VectorSearchRet... open-webui-rocm | └ ContextualCompressionRetriever(base_compressor=RerankCompressor(embedding_function=<function chat_completion_files_handler.<l... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 118, in invoke open-webui-rocm | raise e open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 115, in invoke open-webui-rocm | result = self.rank_fusion(input, run_manager=run_manager, config=config) open-webui-rocm | │ │ │ │ └ {'tags': [], 'metadata': {}, 'callbacks': <langchain_core.callbacks.manager.CallbackManager object at 0x7e0ccb0bc200>, 'recur... open-webui-rocm | │ │ │ └ <langchain_core.callbacks.manager.CallbackManagerForRetrieverRun object at 0x7e0ccb0bcf80> open-webui-rocm | │ │ └ 'nonwoven research summaries' open-webui-rocm | │ └ <function EnsembleRetriever.rank_fusion at 0x7e0d6bfb0400> open-webui-rocm | └ EnsembleRetriever(retrievers=[BM25Retriever(vectorizer=<rank_bm25.BM25Okapi object at 0x7e0d3497ab70>, k=30), VectorSearchRet... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain/retrievers/ensemble.py", line 223, in rank_fusion open-webui-rocm | retriever.invoke( open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/langchain_core/retrievers.py", line 259, in invoke open-webui-rocm | result = self._get_relevant_documents( open-webui-rocm | │ └ <function VectorSearchRetriever._get_relevant_documents at 0x7e0d6bd83ec0> open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi... open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 57, in _get_relevant_documents open-webui-rocm | vectors=[self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)], open-webui-rocm | │ │ │ └ None open-webui-rocm | │ │ └ 'nonwoven research summaries' open-webui-rocm | │ └ <function chat_completion_files_handler.<locals>.<lambda>.<locals>.<lambda> at 0x7e0d3457a700> open-webui-rocm | └ VectorSearchRetriever(collection_name='ab37fb86-7213-422c-b37d-01c83b58e6e5', embedding_function=<function chat_completion_fi... open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/utils/middleware.py", line 598, in <lambda> open-webui-rocm | embedding_function=lambda query, prefix: request.app.state.EMBEDDING_FUNCTION( open-webui-rocm | │ │ │ └ <property object at 0x7e10abba62a0> open-webui-rocm | │ │ └ <starlette.requests.Request object at 0x7e0d3418b5c0> open-webui-rocm | │ └ None open-webui-rocm | └ 'nonwoven research summaries' open-webui-rocm | open-webui-rocm | File "/app/backend/open_webui/retrieval/utils.py", line 356, in <lambda> open-webui-rocm | return lambda query, prefix=None, user=None: embedding_function.encode( open-webui-rocm | │ │ └ <function SentenceTransformer.encode at 0x7e0d3f260d60> open-webui-rocm | │ └ SentenceTransformer( open-webui-rocm | │ (transformer): Transformer( open-webui-rocm | │ (auto_model): XLMRobertaLoRA( open-webui-rocm | │ (roberta): XLMRobertaModel( open-webui-rocm | │ ... open-webui-rocm | └ 'nonwoven research summaries' open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 557, in encode open-webui-rocm | out_features = self.forward(features, **kwargs) open-webui-rocm | │ │ │ └ {} open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | │ └ <function SentenceTransformer.forward at 0x7e0d3f260e00> open-webui-rocm | └ SentenceTransformer( open-webui-rocm | (transformer): Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 624, in forward open-webui-rocm | input = module(input, **module_kwargs) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {} open-webui-rocm | │ │ └ ({'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': ... open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ ({'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ device='cuda:0'), 'attention_mask': ... open-webui-rocm | └ <bound method Transformer.forward of Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embe... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/f1944de8402dcd5f2b03f822a4bc22a7f2de2eb9/custom_st.py", line 143, in forward open-webui-rocm | output_states = self.auto_model.forward(**features, **lora_arguments, return_dict=False) open-webui-rocm | │ │ └ {} open-webui-rocm | │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | └ Transformer( open-webui-rocm | (auto_model): XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | ... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_lora.py", line 372, in forward open-webui-rocm | return self.roberta(*args, **kwargs) open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | │ └ () open-webui-rocm | └ XLMRobertaLoRA( open-webui-rocm | (roberta): XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEm... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ │ │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | │ │ └ () open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEmbedding( open-webui-rocm | 250002, 1024, padd... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'input_ids': tensor([[ 0, 351, 3613, 1353, 25188, 29334, 10484, 2]], open-webui-rocm | │ │ device='cuda:0'), 'attention_mask': t... open-webui-rocm | │ └ () open-webui-rocm | └ <bound method XLMRobertaModel.forward of XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): Parame... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 734, in forward open-webui-rocm | sequence_output = self.encoder( open-webui-rocm | └ XLMRobertaModel( open-webui-rocm | (embeddings): XLMRobertaEmbeddings( open-webui-rocm | (word_embeddings): ParametrizedEmbedding( open-webui-rocm | 250002, 1024, padd... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'key_padding_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_... open-webui-rocm | │ │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ... open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ XLMRobertaEncoder( open-webui-rocm | (layers): ModuleList( open-webui-rocm | (0-23): 24 x Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding(... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'key_padding_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0'), 'subset_mask': None, 'adapter_mask': None, 'output_... open-webui-rocm | │ └ (tensor([[[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, ... open-webui-rocm | └ <bound method XLMRobertaEncoder.forward of XLMRobertaEncoder( open-webui-rocm | (layers): ModuleList( open-webui-rocm | (0-23): 24 x Block( open-webui-rocm | (mixer): ... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/modeling_xlm_roberta.py", line 258, in forward open-webui-rocm | hidden_states = layer(hidden_states, mixer_kwargs=mixer_kwargs) open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None} open-webui-rocm | │ └ tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.0... open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}} open-webui-rocm | │ │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.... open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'mixer_kwargs': {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None}} open-webui-rocm | │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.... open-webui-rocm | └ <bound method Block.forward of Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResid... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/block.py", line 201, in forward open-webui-rocm | mixer_out = self.mixer( open-webui-rocm | └ Block( open-webui-rocm | (mixer): MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, ou... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None} open-webui-rocm | │ │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.... open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, out_features=3072, bias=Tr... open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8, 'adapter_mask': None} open-webui-rocm | │ └ (tensor([[-0.0913, -0.1147, -0.3457, ..., 0.1426, -0.3750, -0.5352], open-webui-rocm | │ [ 0.1177, 0.1758, 0.6406, ..., 0.3984, 0.... open-webui-rocm | └ <bound method MHA.forward of MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=102... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/mha.py", line 732, in forward open-webui-rocm | qkv = self.rotary_emb( open-webui-rocm | └ MHA( open-webui-rocm | (rotary_emb): RotaryEmbedding() open-webui-rocm | (Wqkv): ParametrizedLinearResidual( open-webui-rocm | in_features=1024, out_features=3072, bias=Tr... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl open-webui-rocm | return self._call_impl(*args, **kwargs) open-webui-rocm | │ │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8} open-webui-rocm | │ │ └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01, open-webui-rocm | │ │ 1.3125e+00, 1.0781e+00], open-webui-rocm | │ │ [-6.796... open-webui-rocm | │ └ <function Module._call_impl at 0x7e0e7f720f40> open-webui-rocm | └ RotaryEmbedding() open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl open-webui-rocm | return forward_call(*args, **kwargs) open-webui-rocm | │ │ └ {'seqlen_offset': 0, 'cu_seqlens': tensor([0, 8], device='cuda:0', dtype=torch.int32), 'max_seqlen': 8} open-webui-rocm | │ └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01, open-webui-rocm | │ 1.3125e+00, 1.0781e+00], open-webui-rocm | │ [-6.796... open-webui-rocm | └ <bound method RotaryEmbedding.forward of RotaryEmbedding()> open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 604, in forward open-webui-rocm | return apply_rotary_emb_qkv_( open-webui-rocm | └ <function apply_rotary_emb_qkv_ at 0x7e0d3da023e0> open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 327, in apply_rotary_emb_qkv_ open-webui-rocm | return ApplyRotaryEmbQKV_.apply( open-webui-rocm | │ └ <classmethod(<function Function.apply at 0x7e0e7f499760>)> open-webui-rocm | └ <class 'transformers_modules.jinaai.xlm-roberta-flash-implementation.2b6bc3f30750b3a9648fe9b63448c09920efe9be.rotary.ApplyRot... open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply open-webui-rocm | return super().apply(*args, **kwargs) # type: ignore[misc] open-webui-rocm | │ └ {} open-webui-rocm | └ (tensor([[[[-6.4453e-01, -9.0820e-02, -1.8262e-01, ..., 2.2559e-01, open-webui-rocm | 1.3125e+00, 1.0781e+00], open-webui-rocm | [-6.796... open-webui-rocm | open-webui-rocm | File "/app/backend/data/cache/embedding/models/modules/transformers_modules/jinaai/xlm-roberta-flash-implementation/2b6bc3f30750b3a9648fe9b63448c09920efe9be/rotary.py", line 186, in forward open-webui-rocm | apply_rotary( open-webui-rocm | └ <function apply_rotary at 0x7e0d3da01ee0> open-webui-rocm | open-webui-rocm | File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary open-webui-rocm | assert sin.shape == cos.shape open-webui-rocm | │ │ │ └ <attribute 'shape' of 'torch._C.TensorBase' objects> open-webui-rocm | │ │ └ tensor([[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, open-webui-rocm | │ │ 1.0000, 1.0000, 1.0000, 1.0000,... open-webui-rocm | │ └ <attribute 'shape' of 'torch._C.TensorBase' objects> open-webui-rocm | └ tensor([[ 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, open-webui-rocm | 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.... open-webui-rocm | open-webui-rocm | AssertionError: assert sin.shape == cos.shape open-webui-rocm | [GIN] 2025/04/03 - 09:21:11 | 200 | 12.946911352s | 127.0.0.1 | POST "/api/chat" open-webui-rocm | [GIN] 2025/04/03 - 09:21:13 | 200 | 1.468138859s | 127.0.0.1 | POST "/api/chat" open-webui-rocm | [GIN] 2025/04/03 - 09:21:15 | 200 | 1.365315684s | 127.0.0.1 | POST "/api/chat"
Author
Owner

@Phlogi commented on GitHub (Apr 3, 2025):

Please also share exactly how run the image and what tag etc. you're using. Also, what are you doing in the app exactly to get this error? I assume searching through a collection, however then some default logging output is missing. So it seems you're not using the exact 0.6.0 version or doing something else that triggers hybrid search.
Coud you try to reproduce it without ROCm, using standard image and CPU only?

<!-- gh-comment-id:2775395275 --> @Phlogi commented on GitHub (Apr 3, 2025): Please also share exactly how run the image and what tag etc. you're using. Also, what are you doing in the app exactly to get this error? I assume searching through a collection, however then some default logging output is missing. So it seems you're not using the exact 0.6.0 version or doing something else that triggers hybrid search. Coud you try to reproduce it without ROCm, using standard image and CPU only?
Author
Owner

@DrZoidberg09 commented on GitHub (Apr 3, 2025):

I just had a look into the Dockerfile. And it seems that sentencetransformer is installed separately (dont know why). I dont have access right now to check, but I think it will install sentencetransformer 4, which heavily refactored the reranking. I will bake a new Docker image without that and will let you know. In best case this will solve the issue and just be a heads up that there need to be adaptions for sentencetransformer 4, if you plan to update it.

<!-- gh-comment-id:2775768643 --> @DrZoidberg09 commented on GitHub (Apr 3, 2025): I just had a look into the Dockerfile. And it seems that sentencetransformer is installed separately (dont know why). I dont have access right now to check, but I think it will install sentencetransformer 4, which heavily refactored the reranking. I will bake a new Docker image without that and will let you know. In best case this will solve the issue and just be a heads up that there need to be adaptions for sentencetransformer 4, if you plan to update it.
Author
Owner

@dromeuf commented on GitHub (Apr 3, 2025):

Hi, for me too with pip local installation, since version 0.6, the hybrid search option switch doesn't even appear anymore. I can't specify a reranker model, the text input file no longer appears.

<!-- gh-comment-id:2775959439 --> @dromeuf commented on GitHub (Apr 3, 2025): Hi, for me too with pip local installation, since version 0.6, the hybrid search option switch doesn't even appear anymore. I can't specify a reranker model, the text input file no longer appears.
Author
Owner

@rgaricano commented on GitHub (Apr 3, 2025):

I think it's because XLM Roberta model that is used,
If i'm not wrong, Roberta work with subwords embeddings and tensor output it's different than expected.
(the output is a tensor with a shape type [1, 4, 505], because LM will output the probability distribution across all the words in the vocabulary, and we expect another shape, like [1,68])

If this is it, I read that there are ways to deal doing token embeddings averages of the subwords to generate an embedding for the whole word....

<!-- gh-comment-id:2776133987 --> @rgaricano commented on GitHub (Apr 3, 2025): I think it's because XLM Roberta model that is used, If i'm not wrong, Roberta work with subwords embeddings and tensor output it's different than expected. (the output is a tensor with a shape type [1, 4, 505], because LM will output the probability distribution across all the words in the vocabulary, and we expect another shape, like [1,68]) If this is it, I read that there are ways to deal doing token embeddings averages of the subwords to generate an embedding for the whole word....
Author
Owner

@bet0x commented on GitHub (Apr 4, 2025):

You error is on flash attn using rocm, dunno if AMD is properly supported:

File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary
open-webui-rocm | assert sin.shape == cos.shape
<!-- gh-comment-id:2779169471 --> @bet0x commented on GitHub (Apr 4, 2025): You error is on flash attn using rocm, dunno if AMD is properly supported: ``` File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12.egg/flash_attn/ops/triton/rotary.py", line 166, in apply_rotary open-webui-rocm | assert sin.shape == cos.shape ```
Author
Owner

@DrZoidberg09 commented on GitHub (Apr 8, 2025):

I did use Triton ROCm implementation of flash attention without any issue before the refactoring of the hybrid search (using JinaAI based on XLM Roberta). And all embedding/reranking with models not using flash attention still work fine. Hence, the refactoring somehow has broken something.

<!-- gh-comment-id:2785622787 --> @DrZoidberg09 commented on GitHub (Apr 8, 2025): I did use Triton ROCm implementation of flash attention without any issue before the refactoring of the hybrid search (using JinaAI based on XLM Roberta). And all embedding/reranking with models not using flash attention still work fine. Hence, the refactoring somehow has broken something.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#16566