[PR #15903] [MERGED] feat: Add configurable API URL (for self-hosting) and additional_config parameter for Datalab Marker API #95017

New Issue

GiteaMirror · 2026-05-15T21:08:53-05:00

GiteaMirror commented

2026-05-15 21:08:53 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/15903
Author: @Hisma
Created: 7/21/2025
Status: ✅ Merged
Merged: 8/4/2025
Merged by: @tjbck

Base: dev ← Head: marker-api-update

📝 Commits (6)

5fbfe2b Merge pull request #15879 from open-webui/dev
f31cc07 feat: update marker api
a99e20c add format_lines
21337a2 ci fix
216c991 move api key below url
9590897 Merge branch 'open-webui:main' into marker-api-update

📊 Changes

6 files changed (+214 additions, -97 deletions)

View changed files

📝 backend/open_webui/config.py (+16 -4)
📝 backend/open_webui/main.py (+6 -2)
📝 backend/open_webui/retrieval/loaders/datalab_marker.py (+92 -67)
📝 backend/open_webui/retrieval/loaders/main.py (+7 -1)
📝 backend/open_webui/routers/retrieval.py (+25 -8)
📝 src/lib/components/admin/Settings/Documents.svelte (+68 -15)

📄 Description

Before submitting, make sure you've checked the following:

Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Have you written and run sufficient tests to validate the changes?
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Prefix: To clearly categorize this pull request, prefix the pull request title using one of the following:

Changelog Entry

Description

Reference issue - https://github.com/open-webui/open-webui/issues/13137#issuecomment-3014486758

This PR enhances the Datalab Marker API integration by adding configurable self-hosted Marker API URL support and replacing the deprecated language selection feature with the new additional_config parameter. Users can now specify custom Datalab Marker API endpoints and have control over the latest marker API processing options.

Added

Configurable API Base URL field allowing users to specify custom Datalab Marker API endpoints
- Defaults to https://www.datalab.to/api/v1/marker when left empty
- Includes tooltip showing the default endpoint
Additional Config field for Datalab Marker API supporting all documented configuration options:
- disable_links, keep_pageheader_in_output, keep_pagefooter_in_output
- filter_blank_pages, drop_repeated_text, layout_coverage_threshold
- merge_threshold, height_tolerance, gap_threshold, image_threshold
- min_line_length, level_count, default_level
JSON validation for additional_config input field
Fallback logic for empty API base URL handling

Changed

Use LLM tooltip now correctly shows "Defaults to False" instead of "Defaults to True"
API URL handling with proper validation and fallback mechanisms

Deprecated

Language selection feature for Datalab Marker API (replaced by additional_config)

Removed

Deprecated language selection UI components and backend handling

Security

Input validation for additional_config JSON parameter
Proper URL validation and sanitization for custom API endpoints

Key Enhancements:

Custom API URL Configuration - Supports enterprise/self-hosted Datalab Marker instances
Comprehensive Parameter Control - Full access to all Datalab API configuration options
Robust Error Handling - Proper fallbacks and validation throughout the stack

Testing Environment:

Built with: docker build --build-arg USE_CUDA=true --build-arg USE_CUDA_VER=cu121 --build-arg USE_OLLAMA=false -t openwebui-custom:dev .
Container available at: docker.io/hisma/openwebui:dev
Successfully tested document upload and processing with custom additional_config parameters & API URL as configurable parameter

Backend Logs Confirm Success:

INFO | open_webui.retrieval.loaders.datalab_marker:load:104 - Datalab Marker POST request parameters: {'filename': '36d5ee52-a5b4-47b0-bd04-cec9c821276a_81250143.pdf', 'mime_type': 'application/pdf', **{'use_llm': 'true', 'skip_cache': 'false', 'force_ocr': 'false', 'paginate': 'false', 'strip_existing_ocr': 'false', 'disable_image_extraction': 'false', 'output_format': 'markdown', 'additional_config': '{"keep_pageheader_in_output": true, "keep_pagefooter_in_output": true}'}}

INFO | open_webui.retrieval.loaders.datalab_marker:load:173 - Marker processing completed successfully

Screenshots

468440433-71d30b37-e0bb-4e91-9bfe-8a9372b1d8aa

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/15903 **Author:** [@Hisma](https://github.com/Hisma) **Created:** 7/21/2025 **Status:** ✅ Merged **Merged:** 8/4/2025 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `marker-api-update` --- ### 📝 Commits (6) - [`5fbfe2b`](https://github.com/open-webui/open-webui/commit/5fbfe2bdcadf5f157926f6551891e4dc0802b9f3) Merge pull request #15879 from open-webui/dev - [`f31cc07`](https://github.com/open-webui/open-webui/commit/f31cc07a9d2c49267cb161cfdf73dab031fd8ff1) feat: update marker api - [`a99e20c`](https://github.com/open-webui/open-webui/commit/a99e20cc3d9b7d17f79495a134589c0f182e1b7c) add format_lines - [`21337a2`](https://github.com/open-webui/open-webui/commit/21337a2fd824e943d2e50ef35c64f8e69942ae3f) ci fix - [`216c991`](https://github.com/open-webui/open-webui/commit/216c991c01f83aaaa63df59e8bf12737f051ec45) move api key below url - [`9590897`](https://github.com/open-webui/open-webui/commit/9590897f3a1d2388254c4a3deea8ba9832ef771e) Merge branch 'open-webui:main' into marker-api-update ### 📊 Changes **6 files changed** (+214 additions, -97 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/config.py` (+16 -4) 📝 `backend/open_webui/main.py` (+6 -2) 📝 `backend/open_webui/retrieval/loaders/datalab_marker.py` (+92 -67) 📝 `backend/open_webui/retrieval/loaders/main.py` (+7 -1) 📝 `backend/open_webui/routers/retrieval.py` (+25 -8) 📝 `src/lib/components/admin/Settings/Documents.svelte` (+68 -15) </details> ### 📄 Description **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Please verify that the pull request targets the `dev` branch. - [x] **Description:** Provide a concise description of the changes made in this pull request. - [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [ ] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources? - [ ] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation? - [x] **Testing:** Have you written and run sufficient tests to validate the changes? - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Prefix:** To clearly categorize this pull request, prefix the pull request title using one of the following: # Changelog Entry ### Description Reference issue - https://github.com/open-webui/open-webui/issues/13137#issuecomment-3014486758 This PR enhances the Datalab Marker API integration by adding configurable self-hosted Marker API URL support and replacing the deprecated language selection feature with the new `additional_config` parameter. Users can now specify custom Datalab Marker API endpoints and have control over the latest marker API processing options. ### Added - **Configurable API Base URL** field allowing users to specify custom Datalab Marker API endpoints - Defaults to `https://www.datalab.to/api/v1/marker` when left empty - Includes tooltip showing the default endpoint - **Additional Config field** for Datalab Marker API supporting all documented configuration options: - `disable_links`, `keep_pageheader_in_output`, `keep_pagefooter_in_output` - `filter_blank_pages`, `drop_repeated_text`, `layout_coverage_threshold` - `merge_threshold`, `height_tolerance`, `gap_threshold`, `image_threshold` - `min_line_length`, `level_count`, `default_level` - **JSON validation** for additional_config input field - **Fallback logic** for empty API base URL handling ### Changed - **Use LLM tooltip** now correctly shows "Defaults to False" instead of "Defaults to True" - **API URL handling** with proper validation and fallback mechanisms ### Deprecated - **Language selection feature** for Datalab Marker API (replaced by additional_config) ### Removed - Deprecated language selection UI components and backend handling ### Security - Input validation for additional_config JSON parameter - Proper URL validation and sanitization for custom API endpoints --- **Key Enhancements:** 1. **Custom API URL Configuration** - Supports enterprise/self-hosted Datalab Marker instances 2. **Comprehensive Parameter Control** - Full access to all Datalab API configuration options 3. **Robust Error Handling** - Proper fallbacks and validation throughout the stack **Testing Environment:** - Built with: `docker build --build-arg USE_CUDA=true --build-arg USE_CUDA_VER=cu121 --build-arg USE_OLLAMA=false -t openwebui-custom:dev .` - Container available at: `docker.io/hisma/openwebui:dev` - Successfully tested document upload and processing with custom additional_config parameters & API URL as configurable parameter **Backend Logs Confirm Success:** ``` INFO | open_webui.retrieval.loaders.datalab_marker:load:104 - Datalab Marker POST request parameters: {'filename': '36d5ee52-a5b4-47b0-bd04-cec9c821276a_81250143.pdf', 'mime_type': 'application/pdf', **{'use_llm': 'true', 'skip_cache': 'false', 'force_ocr': 'false', 'paginate': 'false', 'strip_existing_ocr': 'false', 'disable_image_extraction': 'false', 'output_format': 'markdown', 'additional_config': '{"keep_pageheader_in_output": true, "keep_pagefooter_in_output": true}'}} INFO | open_webui.retrieval.loaders.datalab_marker:load:173 - Marker processing completed successfully ``` ### Screenshots <img width="3373" height="860" alt="468440433-71d30b37-e0bb-4e91-9bfe-8a9372b1d8aa" src="https://github.com/user-attachments/assets/4feba187-758d-471e-92b3-9ffc3f9acf6e" /> ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-15 21:08:53 -05:00

GiteaMirror closed this issue

2026-05-15 21:08:56 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#95017