[GH-ISSUE #24393] bug: PPTX/DOC file processing hangs at status=pending in fresh container — averaged_perceptron_tagger_eng NLTK resource not bundled #107280

New Issue

GiteaMirror · 2026-05-18T06:01:06-05:00

GiteaMirror commented

2026-05-18 06:01:06 -05:00

Originally created by @vzd3v on GitHub (May 5, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24393

Summary

Uploading a .pptx (or any other format that goes through UnstructuredPowerPointLoader-like loaders) to a freshly built/recreated open-webui container leaves the file stuck at status=pending forever. Knowledge-base ingestion silently never advances. Chat works, embeddings work, but RAG document processing is blocked end-to-end.

Root cause

unstructured==0.18.31 + nltk==3.9.3 (both pinned in backend/requirements.txt on main, v0.9.1, and v0.9.2) require the language-tagged NLTK resource averaged_perceptron_tagger_eng for sentence/POS tagging in PPTX/Word partitioning. NLTK 3.9.x renamed several taggers to add an _eng suffix.

The Dockerfile (and backend/start.sh) bundles only punkt_tab:

python -c "import nltk; nltk.download('punkt_tab')"

averaged_perceptron_tagger_eng is never pre-downloaded. On first PPTX upload after a fresh docker compose up --force-recreate (or a clean image pull), unstructured raises:

LookupError: Resource 'averaged_perceptron_tagger_eng' not found.
  Please use the NLTK Downloader to obtain the resource:
  >>> import nltk
  >>> nltk.download('averaged_perceptron_tagger_eng')

Confirmed by direct probe inside the container:

from langchain_community.document_loaders import UnstructuredPowerPointLoader
UnstructuredPowerPointLoader('/path/to/file.pptx').load()
# -> LookupError: Resource 'averaged_perceptron_tagger_eng' not found.

After python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng', download_dir='/root/nltk_data')", the same probe succeeds in ~1.5s and any subsequent file upload completes within seconds.

Why it appears as 'pending' rather than 'failed'

The exception path in process_uploaded_file is supposed to set status='failed' with the error string, but in practice the file row stays at status='pending' (no failed, no completed, no log line at WARN/ERROR), making this look like a silent hang rather than a deterministic failure. The frontend SSE on /api/v1/files/{id}/process/status?stream=true keeps spinning. Worth a separate look on its own — the exception is being swallowed somewhere in the BackgroundTask/asyncio chain. Reproduces with both process_in_background=true and inline.

Reproduction

Pull ghcr.io/open-webui/open-webui:v0.9.2 (or v0.9.1, identical packages).
Start fresh, configure any embedding backend.
Upload any .pptx (e.g. via Knowledge → Upload).
File row stays at data->>'status' = 'pending' indefinitely. No log lines mentioning the file ID after the initial POST /api/v1/files/?process=true HTTP/1.1 200.

Probe to confirm root cause:

docker exec -it open-webui-app python3 -c "
from langchain_community.document_loaders import UnstructuredPowerPointLoader
import glob
p = glob.glob('/app/backend/data/uploads/*.pptx')[0]
UnstructuredPowerPointLoader(p).load()
"

Proposed fix

Two parts:

Bundle the missing NLTK resource alongside punkt_tab. Mirror the existing pattern in both Dockerfile branches (USE_CUDA=true and else) and in backend/start.sh / backend/start_windows.bat:
```
python -c "import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng')"
```
Keeps the airgapped-friendliness improvement from #21165 consistent across the resources unstructured 0.18.x actually needs.
Surface the failure. Make sure the process_uploaded_file exception path actually transitions data->>'status' to failed with the original error string in data->>'error' so the UI shows a clear error instead of an infinite spinner. Today the SSE polls data->>'status' and never sees a terminal state.

Happy to put up a PR for (1) — it's a 3-character diff.

Environment

Open WebUI: v0.9.2 (and v0.9.1, same deps)
unstructured: 0.18.31
nltk: 3.9.3
Vector store: pgvector 0.8.2
Embedding engine: openai-compatible (LiteLLM → GigaChat)

Originally created by @vzd3v on GitHub (May 5, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24393 ### Summary Uploading a `.pptx` (or any other format that goes through `UnstructuredPowerPointLoader`-like loaders) to a freshly built/recreated `open-webui` container leaves the file stuck at `status=pending` forever. Knowledge-base ingestion silently never advances. Chat works, embeddings work, but RAG document processing is blocked end-to-end. ### Root cause `unstructured==0.18.31` + `nltk==3.9.3` (both pinned in `backend/requirements.txt` on `main`, `v0.9.1`, and `v0.9.2`) require the language-tagged NLTK resource `averaged_perceptron_tagger_eng` for sentence/POS tagging in PPTX/Word partitioning. NLTK 3.9.x renamed several taggers to add an `_eng` suffix. The Dockerfile (and `backend/start.sh`) bundles only `punkt_tab`: ```dockerfile python -c "import nltk; nltk.download('punkt_tab')" ``` `averaged_perceptron_tagger_eng` is never pre-downloaded. On first PPTX upload after a fresh `docker compose up --force-recreate` (or a clean image pull), `unstructured` raises: ``` LookupError: Resource 'averaged_perceptron_tagger_eng' not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('averaged_perceptron_tagger_eng') ``` Confirmed by direct probe inside the container: ```python from langchain_community.document_loaders import UnstructuredPowerPointLoader UnstructuredPowerPointLoader('/path/to/file.pptx').load() # -> LookupError: Resource 'averaged_perceptron_tagger_eng' not found. ``` After `python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng', download_dir='/root/nltk_data')"`, the same probe succeeds in ~1.5s and any subsequent file upload completes within seconds. ### Why it appears as 'pending' rather than 'failed' The exception path in `process_uploaded_file` is supposed to set `status='failed'` with the error string, but in practice the file row stays at `status='pending'` (no `failed`, no `completed`, no log line at WARN/ERROR), making this look like a silent hang rather than a deterministic failure. The frontend SSE on `/api/v1/files/{id}/process/status?stream=true` keeps spinning. Worth a separate look on its own — the exception is being swallowed somewhere in the BackgroundTask/asyncio chain. Reproduces with both `process_in_background=true` and inline. ### Reproduction 1. Pull `ghcr.io/open-webui/open-webui:v0.9.2` (or `v0.9.1`, identical packages). 2. Start fresh, configure any embedding backend. 3. Upload any `.pptx` (e.g. via Knowledge → Upload). 4. File row stays at `data->>'status' = 'pending'` indefinitely. No log lines mentioning the file ID after the initial `POST /api/v1/files/?process=true HTTP/1.1 200`. Probe to confirm root cause: ```bash docker exec -it open-webui-app python3 -c " from langchain_community.document_loaders import UnstructuredPowerPointLoader import glob p = glob.glob('/app/backend/data/uploads/*.pptx')[0] UnstructuredPowerPointLoader(p).load() " ``` ### Proposed fix Two parts: 1. **Bundle the missing NLTK resource alongside `punkt_tab`.** Mirror the existing pattern in both Dockerfile branches (`USE_CUDA=true` and `else`) and in `backend/start.sh` / `backend/start_windows.bat`: ```dockerfile python -c "import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng')" ``` Keeps the airgapped-friendliness improvement from #21165 consistent across the resources `unstructured` 0.18.x actually needs. 2. **Surface the failure.** Make sure the `process_uploaded_file` exception path actually transitions `data->>'status'` to `failed` with the original error string in `data->>'error'` so the UI shows a clear error instead of an infinite spinner. Today the SSE polls `data->>'status'` and never sees a terminal state. Happy to put up a PR for (1) — it's a 3-character diff. ### Environment - Open WebUI: `v0.9.2` (and `v0.9.1`, same deps) - `unstructured`: `0.18.31` - `nltk`: `3.9.3` - Vector store: `pgvector` 0.8.2 - Embedding engine: `openai`-compatible (LiteLLM → GigaChat)

GiteaMirror commented

2026-05-18 06:01:07 -05:00

@owui-terminator[bot] commented on GitHub (May 5, 2026):

🔍 Similar Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

#20107 issue:PPTX not parsed correctly in temporary chat
by mengdeer589 · bug
#16260 feat: nltk_data taggers and tokenizers to docker image
by artokarj
#17594 issue: Dragging .xlsx file into open-webui triggers punkt_tab not found error
by peuportier · bug
#16158 issue: Processing does not continue after open_webui.retrieval.utils:generate_openai_batch_embeddings call
by BAngelis · bug
#4642 bug: upload md file, but got error message 'Resource punkt_tab not found.'
by qaz-t

💡 If this is a duplicate, consider closing it and adding details to the existing issue.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

@owui-terminator[bot] commented on GitHub (May 5, 2026): 🔍 **Similar Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. [#20107](https://github.com/open-webui/open-webui/issues/20107) **issue:PPTX not parsed correctly in temporary chat** *by mengdeer589 · `bug`* 2. [#16260](https://github.com/open-webui/open-webui/issues/16260) **feat: nltk_data taggers and tokenizers to docker image** *by artokarj* 3. [#17594](https://github.com/open-webui/open-webui/issues/17594) **issue: Dragging .xlsx file into open-webui triggers punkt_tab not found error** *by peuportier · `bug`* 4. [#16158](https://github.com/open-webui/open-webui/issues/16158) **issue: Processing does not continue after open_webui.retrieval.utils:generate_openai_batch_embeddings call** *by BAngelis · `bug`* 5. [#4642](https://github.com/open-webui/open-webui/issues/4642) **bug: upload md file, but got error message 'Resource punkt_tab not found.'** *by qaz-t* --- 💡 If this is a duplicate, consider closing it and adding details to the existing issue. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.

GiteaMirror commented

2026-05-18 06:01:08 -05:00

@vzd3v commented on GitHub (May 5, 2026):

Putting up a tightly-scoped fix as #24396 (mirrors #21165 in scope). The status='pending' / silent-swallow part of the symptom is left for a separate fix.

@vzd3v commented on GitHub (May 5, 2026): Putting up a tightly-scoped fix as #24396 (mirrors #21165 in scope). The status='pending' / silent-swallow part of the symptom is left for a separate fix.

GiteaMirror commented

2026-05-18 06:01:09 -05:00

@vzd3v commented on GitHub (May 5, 2026):

PR moved to #24396 — added the required PR-template sections and CLA confirmation that auto-closed #24395. (Earlier #24394 was auto-closed for targeting main.)

@vzd3v commented on GitHub (May 5, 2026): PR moved to #24396 — added the required PR-template sections and CLA confirmation that auto-closed #24395. (Earlier #24394 was auto-closed for targeting main.)

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#107280