Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources? Documentation will be added to the relevant tutorial via a separate PR once this PR is accepted.
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Have you written and run sufficient tests for validating the changes?
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Prefix: To cleary categorize this pull request, prefix the pull request title, using one of the following:
BREAKING CHANGE: Significant changes that may affect compatibility
build: Changes that affect the build system or external dependencies
ci: Changes to our continuous integration processes or workflows
chore: Refactor, cleanup, or other non-functional code changes
docs: Documentation update or addition
feat: Introduces a new feature or enhancement to the codebase
fix: Bug fix or error correction
i18n: Internationalization or localization changes
perf: Performance improvement
refactor: Code restructuring for better maintainability, readability, or scalability
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
test: Adding missing tests or correcting existing tests
WIP: Work in progress, a temporary label for incomplete or ongoing work
Changelog Entry
Description
Before any non-image files attached to chats can be used by LLMs, their content text must be extracted. This PR adds support to use Apache Tika for this rather than using the Langchain community document loaders.
Added
Code in backend/apps/rag/main.py and supporting files to conditionally make calls out to an Apache Tika server to extract document text rather than using the community-supported loaders from Langchain.
Additions to the stored configuration schema to support text extraction engine selection and configuration.
UI components for the Admin Panel->Settings->Documents page to allow engine selection and configuration.
Changed
Minor (incidental) tweak to .dockerignore to prevent local virtual environments in venv directory from being copied into Docker images.
Deprecated
None
Removed
None
Fixed
None
Security
Note that the use of a new service to which user content will be sent opens up a new attack surface. Apache Tika is a mature product and will typically be deployed in a sidecar container, so the risk is very small.
Breaking Changes
None
Additional Information
Apache Tika is a mature tool for extracting text and metadata from files. It supports hundreds of different document types, is under active development and is well-supported. Features include fully integrated OCR of embedded images in hierarchical documents and output of both plain and structured text.
The Tika server is available as a pre-packaged Docker image for multiple platforms. While this code supports communicating with a server in any network-accessible location, the suggested configuration is to deploy it as a sidecar container along side the Open WebUI server. This can be done by adding the following to the docker-compose.yaml file:
tika:image:apache/tika:latest-full
and updating the configuration for the Open WebUI container with an additional environment variable:
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/3582
**Author:** [@nickovs](https://github.com/nickovs)
**Created:** 7/1/2024
**Status:** ✅ Merged
**Merged:** 7/2/2024
**Merged by:** [@tjbck](https://github.com/tjbck)
**Base:** `dev` ← **Head:** `tika-document-text`
---
### 📝 Commits (2)
- [`9cf622d`](https://github.com/open-webui/open-webui/commit/9cf622d98118e40a92194a4ef76c1cacd92d2640) Added support for using Apache Tika as a document loader.
- [`7aa35a3`](https://github.com/open-webui/open-webui/commit/7aa35a37573c1d0af136176756a16ba73b74f74b) Added HTML and Typescript UI components to support configration of text extraction engine.
### 📊 Changes
**5 files changed** (+181 additions, -41 deletions)
<details>
<summary>View changed files</summary>
📝 `.dockerignore` (+2 -1)
📝 `backend/apps/rag/main.py` (+107 -40)
📝 `backend/config.py` (+16 -0)
📝 `src/lib/apis/rag/index.ts` (+6 -0)
📝 `src/lib/components/admin/Settings/Documents.svelte` (+50 -0)
</details>
### 📄 Description
See discussion #3475 for background.
- [X] **Target branch:** Please verify that the pull request targets the `dev` branch.
- [X] **Description:** Provide a concise description of the changes made in this pull request.
- [X] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description.
- [X] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources? *Documentation will be added to the [relevant tutorial](https://docs.openwebui.com/tutorial/rag) via a separate PR once this PR is accepted.*
- [X] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation?
- [X] **Testing:** Have you written and run sufficient tests for validating the changes?
- [X] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
- [X] **Prefix:** To cleary categorize this pull request, prefix the pull request title, using one of the following:
- **BREAKING CHANGE**: Significant changes that may affect compatibility
- **build**: Changes that affect the build system or external dependencies
- **ci**: Changes to our continuous integration processes or workflows
- **chore**: Refactor, cleanup, or other non-functional code changes
- **docs**: Documentation update or addition
- **feat**: Introduces a new feature or enhancement to the codebase
- **fix**: Bug fix or error correction
- **i18n**: Internationalization or localization changes
- **perf**: Performance improvement
- **refactor**: Code restructuring for better maintainability, readability, or scalability
- **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
- **test**: Adding missing tests or correcting existing tests
- **WIP**: Work in progress, a temporary label for incomplete or ongoing work
# Changelog Entry
### Description
Before any non-image files attached to chats can be used by LLMs, their content text must be extracted. This PR adds support to use [Apache Tika](https://tika.apache.org) for this rather than using the Langchain community document loaders.
### Added
- Code in `backend/apps/rag/main.py` and supporting files to conditionally make calls out to an Apache Tika server to extract document text rather than using the community-supported loaders from Langchain.
- Additions to the stored configuration schema to support text extraction engine selection and configuration.
- UI components for the Admin Panel->Settings->Documents page to allow engine selection and configuration.
### Changed
- Minor (incidental) tweak to `.dockerignore` to prevent local virtual environments in `venv` directory from being copied into Docker images.
### Deprecated
*None*
### Removed
*None*
### Fixed
*None*
### Security
- Note that the use of a new service to which user content will be sent opens up a new attack surface. Apache Tika is a mature product and will typically be deployed in a sidecar container, so the risk is very small.
### Breaking Changes
*None*
---
### Additional Information
[Apache Tika](https://tika.apache.org) is a mature tool for extracting text and metadata from files. It supports hundreds of different document types, is under active development and is well-supported. Features include fully integrated OCR of embedded images in hierarchical documents and output of both plain and structured text.
The Tika server is available as a pre-packaged Docker image for multiple platforms. While this code supports communicating with a server in any network-accessible location, the suggested configuration is to deploy it as a sidecar container along side the Open WebUI server. This can be done by adding the following to the `docker-compose.yaml` file:
```yaml
tika:
image: apache/tika:latest-full
```
and updating the configuration for the Open WebUI container with an additional environment variable:
```yaml
open-webui:
...
environment:
...
- 'TEXT_EXTRACTION_ENGINE=tika'
```
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/3582
Author: @nickovs
Created: 7/1/2024
Status: ✅ Merged
Merged: 7/2/2024
Merged by: @tjbck
Base:
dev← Head:tika-document-text📝 Commits (2)
9cf622dAdded support for using Apache Tika as a document loader.7aa35a3Added HTML and Typescript UI components to support configration of text extraction engine.📊 Changes
5 files changed (+181 additions, -41 deletions)
View changed files
📝
.dockerignore(+2 -1)📝
backend/apps/rag/main.py(+107 -40)📝
backend/config.py(+16 -0)📝
src/lib/apis/rag/index.ts(+6 -0)📝
src/lib/components/admin/Settings/Documents.svelte(+50 -0)📄 Description
See discussion #3475 for background.
devbranch.Changelog Entry
Description
Before any non-image files attached to chats can be used by LLMs, their content text must be extracted. This PR adds support to use Apache Tika for this rather than using the Langchain community document loaders.
Added
backend/apps/rag/main.pyand supporting files to conditionally make calls out to an Apache Tika server to extract document text rather than using the community-supported loaders from Langchain.Changed
.dockerignoreto prevent local virtual environments invenvdirectory from being copied into Docker images.Deprecated
None
Removed
None
Fixed
None
Security
Breaking Changes
None
Additional Information
Apache Tika is a mature tool for extracting text and metadata from files. It supports hundreds of different document types, is under active development and is well-supported. Features include fully integrated OCR of embedded images in hierarchical documents and output of both plain and structured text.
The Tika server is available as a pre-packaged Docker image for multiple platforms. While this code supports communicating with a server in any network-accessible location, the suggested configuration is to deploy it as a sidecar container along side the Open WebUI server. This can be done by adding the following to the
docker-compose.yamlfile:and updating the configuration for the Open WebUI container with an additional environment variable:
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.