mirror of
https://github.com/open-webui/open-webui.git
synced 2026-11-15 10:36:20 -06:00
[GH-ISSUE #14355] issue: could not detect encoding for redacted.msg with Apache Tika #55886
Reference in New Issue
Block a user
Originally created by @DediCATeD88 on GitHub (May 26, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/14355
Check Existing Issues
Installation Method
Docker
Open WebUI Version
v0.6.10
Ollama Version (if applicable)
No response
Operating System
Debian Linux
Browser (if applicable)
No response
Confirmation
README.md.Expected Behavior
Upload File xyz.msg to chat, and chat about xyz.msg
Actual Behavior
Error: could not detect encoding for xyz.msg
Steps to Reproduce
Use Apache Tika as Contect Extraction
Upload xyz.msg or drag and drop from outlook
Error could not detect encoding for redacted.msg
According to http://ip:9998/mime-types .msg is supported:
Description: Microsoft Outlook Message
Default Extension: .msg
Parser: org.apache.tika.parser.microsoft.OfficeParser
Logs & Screenshots
2025-05-26 18:45:16.000 | ERROR | open_webui.routers.retrieval:process_file:1260 - Could not detect encoding for /app/backend/data/uploads/redacted.msg - {}
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
File "/app/backend/open_webui/routers/files.py", line 162, in upload_file
File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load
File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings
RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg
2025-05-26 18:45:16.002 | ERROR | open_webui.routers.files:upload_file:171 - 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg - {}
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 43, in lazy_load
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/backend/open_webui/routers/retrieval.py", line 1179, in process_file
File "/app/backend/open_webui/retrieval/loaders/main.py", line 201, in load
File "/usr/local/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 32, in load
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/text.py", line 46, in lazy_load
File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/helpers.py", line 50, in detect_file_encodings
RuntimeError: Could not detect encoding for /app/backend/data/uploads/redacted.msg
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
File "/app/backend/open_webui/routers/retrieval.py", line 1267, in process_file
fastapi.exceptions.HTTPException: 400: Could not detect encoding for /app/backend/data/uploads/redacted.msg
2025-05-26 18:45:16.004 | ERROR | open_webui.routers.files:upload_file:172 - Error processing file: 9f8bdc63-ccc0-4931-b9a8-f0d0f6691e66 - {}
Additional Information
Analysis
The core of the problem lies in how Open WebUI and its underlying library, LangChain, are attempting to process the .msg file. Although you have selected Apache Tika, which explicitly supports the MIME type for .msg files, the system first tries to handle the file using a default text loader. This loader incorrectly assumes the file is text-based and expects a standard encoding like UTF-8.
The error traceback clearly shows the issue originates within the langchain_community.document_loaders.text.TextLoader. This component attempts to read the .msg file as if it were a simple text document. Since .msg files are a binary format, this process fails immediately, leading to the UnicodeDecodeError.
The key points from the traceback are:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0...: This is the initial failure. The system tries to read the binary .msg file using a UTF-8 text decoder, which is incorrect.
RuntimeError: Could not detect encoding for ...: After the initial read fails, the system tries to automatically detect the file's encoding. Because it's a binary file, not a text file with a consistent encoding, this detection also fails.
fastapi.exceptions.HTTPException: 400: Could not detect encoding...: This is the final error returned to the user, indicating that the file processing failed at a fundamental level.
In summary, the file processing pipeline in Open WebUI is mistakenly routing the .msg file to a text loader before it gets to the Apache Tika processor. The system fails at the first step and never gets a chance to use Tika's capabilities.