[GH-ISSUE #14355] issue: could not detect encoding for redacted.msg with Apache Tika #136947

Closed
opened 2026-05-25 05:49:31 -05:00 by GiteaMirror · 0 comments
Owner

Originally created by @DediCATeD88 on GitHub (May 26, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/14355

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.10

Ollama Version (if applicable)

No response

Operating System

Debian Linux

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have listed steps to reproduce the bug in detail.

Expected Behavior

Upload File xyz.msg to chat, and chat about xyz.msg

Actual Behavior

Error: could not detect encoding for xyz.msg

Steps to Reproduce

Use Apache Tika as Contect Extraction
Upload xyz.msg or drag and drop from outlook
Error could not detect encoding for redacted.msg

According to http://ip:9998/mime-types .msg is supported:
Description: Microsoft Outlook Message
Default Extension: .msg
Parser: org.apache.tika.parser.microsoft.OfficeParser

Logs & Screenshots

2025-05-26 18:45:16.000 | ERROR | open_webui.routers.retrieval:process_file:1260 - Could not detect encoding for /app/backend/data/uploads/redacted.msg - {}

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load

text = f.read()

       │ └ <method 'read' of '_io.TextIOWrapper' objects>

       └ <_io.TextIOWrapper name='/app/backend/data/uploads/redacted...

File "", line 322, in decode

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap

self._bootstrap_inner()

│    └ <function Thread._bootstrap_inner at 0x7fb1a4e18860>

└ <WorkerThread(AnyIO worker thread, started 140396962694848)>

File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner

self.run()

│    └ <function WorkerThread.run at 0x7fb16ab184a0>

└ <WorkerThread(AnyIO worker thread, started 140396962694848)>

File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run

result = context.run(func, *args)

         │       │   │      └ ()

         │       │   └ functools.partial(<function upload_file at 0x7fb17723dc60>, user=UserModel(id='redacted...

         │       └ <method 'run' of '_contextvars.Context' objects>

         └ <_contextvars.Context object at 0x7fb16551d380>

File "/app/backend/open_webui/routers/files.py", line 162, in upload_file

process_file(request, ProcessFileForm(file_id=id), user=user)

│            │        │                       │         └ UserModel(id='redacted', email='redacted', role='user',...

│            │        │                       └ '9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66'

│            │        └ <class 'open_webui.routers.retrieval.ProcessFileForm'>

│            └ <starlette.requests.Request object at 0x7fb165887bd0>

└ <function process_file at 0x7fb174088e00>

File "/app/backend/open_webui/routers/retrieval.py", line 1179, in process_file

docs = loader.load(

       │      └ <function Loader.load at 0x7fb17502f6a0>

       └ <open_webui.retrieval.loaders.main.Loader object at 0x7fb156d88710>

File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load

docs = loader.load()

       │      └ <function BaseLoader.load at 0x7fb175525da0>

       └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load

return list(self.lazy_load())

            │    └ <function TextLoader.lazy_load at 0x7fb17502c2c0>

            └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load

detected_encodings = detect_file_encodings(self.file_path)

                     │                     │    └ '/app/backend/data/uploads/redacted.msg'

                     │                     └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

                     └ <function detect_file_encodings at 0x7fb175526fc0>

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings

raise RuntimeError(f"Could not detect encoding for {file_path}")

RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg

2025-05-26 18:45:16.002 | ERROR | open_webui.routers.files:upload_file:171 - 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg - {}

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load

text = f.read()

       │ └ <method 'read' of '_io.TextIOWrapper' objects>

       └ <_io.TextIOWrapper name='/app/backend/data/uploads/redacted...

File "", line 322, in decode

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/app/backend/open_webui/routers/retrieval.py", line 1179, in process_file

docs = loader.load(

       │      └ <function Loader.load at 0x7fb17502f6a0>

       └ <open_webui.retrieval.loaders.main.Loader object at 0x7fb156d88710>

File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load

docs = loader.load()

       │      └ <function BaseLoader.load at 0x7fb175525da0>

       └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load

return list(self.lazy_load())

            │    └ <function TextLoader.lazy_load at 0x7fb17502c2c0>

            └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load

detected_encodings = detect_file_encodings(self.file_path)

                     │                     │    └ '/app/backend/data/uploads/redacted.msg'

                     │                     └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50>

                     └ <function detect_file_encodings at 0x7fb175526fc0>

File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings

raise RuntimeError(f"Could not detect encoding for {file_path}")

RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap

self._bootstrap_inner()

│    └ <function Thread._bootstrap_inner at 0x7fb1a4e18860>

└ <WorkerThread(AnyIO worker thread, started 140396962694848)>

File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner

self.run()

│    └ <function WorkerThread.run at 0x7fb16ab184a0>

└ <WorkerThread(AnyIO worker thread, started 140396962694848)>

File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run

result = context.run(func, *args)

         │       │   │      └ ()

         │       │   └ functools.partial(<function upload_file at 0x7fb17723dc60>, user=UserModel(id=redacted...

         │       └ <method 'run' of '_contextvars.Context' objects>

         └ <_contextvars.Context object at 0x7fb16551d380>

File "/app/backend/open_webui/routers/files.py", line 162, in upload_file

process_file(request, ProcessFileForm(file_id=id), user=user)

│            │        │                       │         └ UserModel(id='692de42a-f0f1-40d2-8fdd-6d61e11e1a86', name='redacted', email='redacted', role='user',...

│            │        │                       └ '9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66'

│            │        └ <class 'open_webui.routers.retrieval.ProcessFileForm'>

│            └ <starlette.requests.Request object at 0x7fb165887bd0>

└ <function process_file at 0x7fb174088e00>

File "/app/backend/open_webui/routers/retrieval.py", line 1267, in process_file

raise HTTPException(

      └ <class 'fastapi.exceptions.HTTPException'>

fastapi.exceptions.HTTPException: 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg

2025-05-26 18:45:16.004 | ERROR | open_webui.routers.files:upload_file:172 - Error processing file: 9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66 - {}

Additional Information

Analysis
The core of the problem lies in how Open WebUI and its underlying library, LangChain, are attempting to process the .msg file. Although you have selected Apache Tika, which explicitly supports the MIME type for .msg files, the system first tries to handle the file using a default text loader. This loader incorrectly assumes the file is text-based and expects a standard encoding like UTF-8.

The error traceback clearly shows the issue originates within the langchain_community.document_loaders.text.TextLoader. This component attempts to read the .msg file as if it were a simple text document. Since .msg files are a binary format, this process fails immediately, leading to the UnicodeDecodeError.

The key points from the traceback are:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0...: This is the initial failure. The system tries to read the binary .msg file using a UTF-8 text decoder, which is incorrect.
RuntimeError: Could not detect encoding for ...: After the initial read fails, the system tries to automatically detect the file's encoding. Because it's a binary file, not a text file with a consistent encoding, this detection also fails.
fastapi.exceptions.HTTPException: 400: Could not detect encoding...: This is the final error returned to the user, indicating that the file processing failed at a fundamental level.
In summary, the file processing pipeline in Open WebUI is mistakenly routing the .msg file to a text loader before it gets to the Apache Tika processor. The system fails at the first step and never gets a chance to use Tika's capabilities.

Originally created by @DediCATeD88 on GitHub (May 26, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/14355 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.10 ### Ollama Version (if applicable) _No response_ ### Operating System Debian Linux ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior Upload File xyz.msg to chat, and chat about xyz.msg ### Actual Behavior Error: could not detect encoding for xyz.msg ### Steps to Reproduce Use Apache Tika as Contect Extraction Upload xyz.msg or drag and drop from outlook Error could not detect encoding for redacted.msg According to http://ip:9998/mime-types .msg is supported: Description: Microsoft Outlook Message Default Extension: .msg Parser: org.apache.tika.parser.microsoft.OfficeParser ### Logs & Screenshots 2025-05-26 18:45:16.000 | ERROR | open_webui.routers.retrieval:process_file:1260 - Could not detect encoding for /app/backend/data/uploads/redacted.msg - {} Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load text = f.read() │ └ <method 'read' of '_io.TextIOWrapper' objects> └ <_io.TextIOWrapper name='/app/backend/data/uploads/redacted... File "<frozen codecs>", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() │ └ <function Thread._bootstrap_inner at 0x7fb1a4e18860> └ <WorkerThread(AnyIO worker thread, started 140396962694848)> File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner self.run() │ └ <function WorkerThread.run at 0x7fb16ab184a0> └ <WorkerThread(AnyIO worker thread, started 140396962694848)> File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run result = context.run(func, *args) │ │ │ └ () │ │ └ functools.partial(<function upload_file at 0x7fb17723dc60>, user=UserModel(id='redacted... │ └ <method 'run' of '_contextvars.Context' objects> └ <_contextvars.Context object at 0x7fb16551d380> File "/app/backend/open_webui/routers/files.py", line 162, in upload_file process_file(request, ProcessFileForm(file_id=id), user=user) │ │ │ │ └ UserModel(id='redacted', email='redacted', role='user',... │ │ │ └ '9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66' │ │ └ <class 'open_webui.routers.retrieval.ProcessFileForm'> │ └ <starlette.requests.Request object at 0x7fb165887bd0> └ <function process_file at 0x7fb174088e00> > File "/app/backend/open_webui/routers/retrieval.py", line 1179, in process_file docs = loader.load( │ └ <function Loader.load at 0x7fb17502f6a0> └ <open_webui.retrieval.loaders.main.Loader object at 0x7fb156d88710> File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load docs = loader.load() │ └ <function BaseLoader.load at 0x7fb175525da0> └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load return list(self.lazy_load()) │ └ <function TextLoader.lazy_load at 0x7fb17502c2c0> └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load detected_encodings = detect_file_encodings(self.file_path) │ │ └ '/app/backend/data/uploads/redacted.msg' │ └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> └ <function detect_file_encodings at 0x7fb175526fc0> File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings raise RuntimeError(f"Could not detect encoding for {file_path}") RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg 2025-05-26 18:45:16.002 | ERROR | open_webui.routers.files:upload_file:171 - 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg - {} Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load text = f.read() │ └ <method 'read' of '_io.TextIOWrapper' objects> └ <_io.TextIOWrapper name='/app/backend/data/uploads/redacted... File "<frozen codecs>", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/backend/open_webui/routers/retrieval.py", line 1179, in process_file docs = loader.load( │ └ <function Loader.load at 0x7fb17502f6a0> └ <open_webui.retrieval.loaders.main.Loader object at 0x7fb156d88710> File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load docs = loader.load() │ └ <function BaseLoader.load at 0x7fb175525da0> └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load return list(self.lazy_load()) │ └ <function TextLoader.lazy_load at 0x7fb17502c2c0> └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load detected_encodings = detect_file_encodings(self.file_path) │ │ └ '/app/backend/data/uploads/redacted.msg' │ └ <langchain_community.document_loaders.text.TextLoader object at 0x7fb156d88f50> └ <function detect_file_encodings at 0x7fb175526fc0> File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings raise RuntimeError(f"Could not detect encoding for {file_path}") RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() │ └ <function Thread._bootstrap_inner at 0x7fb1a4e18860> └ <WorkerThread(AnyIO worker thread, started 140396962694848)> File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner self.run() │ └ <function WorkerThread.run at 0x7fb16ab184a0> └ <WorkerThread(AnyIO worker thread, started 140396962694848)> File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run result = context.run(func, *args) │ │ │ └ () │ │ └ functools.partial(<function upload_file at 0x7fb17723dc60>, user=UserModel(id=redacted... │ └ <method 'run' of '_contextvars.Context' objects> └ <_contextvars.Context object at 0x7fb16551d380> > File "/app/backend/open_webui/routers/files.py", line 162, in upload_file process_file(request, ProcessFileForm(file_id=id), user=user) │ │ │ │ └ UserModel(id='692de42a-f0f1-40d2-8fdd-6d61e11e1a86', name='redacted', email='redacted', role='user',... │ │ │ └ '9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66' │ │ └ <class 'open_webui.routers.retrieval.ProcessFileForm'> │ └ <starlette.requests.Request object at 0x7fb165887bd0> └ <function process_file at 0x7fb174088e00> File "/app/backend/open_webui/routers/retrieval.py", line 1267, in process_file raise HTTPException( └ <class 'fastapi.exceptions.HTTPException'> fastapi.exceptions.HTTPException: 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg 2025-05-26 18:45:16.004 | ERROR | open_webui.routers.files:upload_file:172 - Error processing file: 9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66 - {} ### Additional Information **Analysis** The core of the problem lies in how Open WebUI and its underlying library, LangChain, are attempting to process the .msg file. Although you have selected Apache Tika, which explicitly supports the MIME type for .msg files, the system first tries to handle the file using a default text loader. This loader incorrectly assumes the file is text-based and expects a standard encoding like UTF-8. The error traceback clearly shows the issue originates within the langchain_community.document_loaders.text.TextLoader. This component attempts to read the .msg file as if it were a simple text document. Since .msg files are a binary format, this process fails immediately, leading to the UnicodeDecodeError. The key points from the traceback are: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0...: This is the initial failure. The system tries to read the binary .msg file using a UTF-8 text decoder, which is incorrect. RuntimeError: Could not detect encoding for ...: After the initial read fails, the system tries to automatically detect the file's encoding. Because it's a binary file, not a text file with a consistent encoding, this detection also fails. fastapi.exceptions.HTTPException: 400: Could not detect encoding...: This is the final error returned to the user, indicating that the file processing failed at a fundamental level. In summary, the file processing pipeline in Open WebUI is mistakenly routing the .msg file to a text loader before it gets to the Apache Tika processor. The system fails at the first step and never gets a chance to use Tika's capabilities.
GiteaMirror added the bug label 2026-05-25 05:49:31 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#136947