[GH-ISSUE #15002] issue: JSON Parsing Error in Docling Picture Description Configuration Field #137165

Closed
opened 2026-05-25 06:24:55 -05:00 by GiteaMirror · 10 comments
Owner

Originally created by @healdgar on GitHub (Jun 15, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/15002

Check Existing Issues

  • I have searched the existing issues and discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.14

Ollama Version (if applicable)

0.9.0

Operating System

ghcr.io/open-webui/open-webui:main

Browser (if applicable)

Chrome (latest)

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When entering valid JSON configuration in either the picture_description_local or picture_description_api fields in the Docling settings, the configuration should:

  1. Parse the JSON as a complete object
  2. Validate the JSON structure properly
  3. Save the configuration successfully
  4. Enable picture description functionality in Docling
  5. Process documents with image description capabilities

The UI tooltip states "Enter config in JSON format" which implies JSON should be accepted and parsed correctly.

Actual Behavior

Any JSON configuration entered in either picture_description_local or picture_description_api fields results in validation errors where individual JSON field names are treated as strings instead of parsing the complete JSON object.

Error Pattern:

Error calling Docling: Error calling Docling API: Unprocessable Entity - [{'type': 'model_attributes_type', 'loc': ['body', 'picture_description_[local|api]'], 'msg': 'Input should be a valid dictionary or object to extract fields from', 'input': '[field_name]'}]

The 'input' value changes to individual field names like:

  • 'url'
  • 'params'
  • 'model'
  • 'temperature'
  • 'options'

This indicates the JSON is being split and individual keys are being validated as complete objects, which fails.

Steps to Reproduce

Environment Setup:

  1. Start with macOS system with Docker Desktop installed
  2. Use latest Open WebUI Docker image (docker pull ghcr.io/open-webui/open-webui:main)
  3. Run Ollama locally with qwen2.5vl:3b model installed
  4. Start Docling service on port 5001

Docker Compose Configuration:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://192.168.68.87:11434
      - DOCLING_SERVER_URL=http://docling:5001
    depends_on:
      - docling
    
  docling:
    image: ds4sd/docling:latest
    ports:
      - "5001:5001"

Reproduction Steps:

  1. Start services with docker compose up -d
  2. Open browser (tested in Chrome 131.0, Safari 18.1, Firefox 133.0)
  3. Navigate to http://localhost:3000
  4. Log in as admin user
  5. Go to Admin Panel → Settings → Documents → Docling
  6. Scroll to "Picture Description" section
  7. Enable "Picture Description" toggle
  8. In the "Picture Description API Config" field, enter this JSON:
    {
      "url": "http://192.168.68.87:11434/v1/chat/completions",
      "params": {
        "model": "qwen2.5vl:3b"
      }
    }
    
  9. Click "Save"
  10. Upload a PDF document with images
  11. Observe the error in the UI

Alternative test with minimal JSON:

  1. Clear the field and enter minimal JSON:
    {"url": "http://192.168.68.87:11434/v1/chat/completions", "model": "qwen2.5vl:3b"}
    
  2. Save configuration
  3. Same error occurs

Test with picture_description_local field:

  1. Clear API config field
  2. Enter same JSON in "Picture Description Local Config" field
  3. Save configuration
  4. Same error pattern occurs

Logs & Screenshots

Browser Console Error:

POST http://localhost:3000/api/v1/documents/docling 422 (Unprocessable Entity)

Detailed Error Response:

{
  "detail": [
    {
      "type": "model_attributes_type",
      "loc": ["body", "picture_description_api"],
      "msg": "Input should be a valid dictionary or object to extract fields from",
      "input": "params"
    }
  ]
}

Docker Container Logs (open-webui):

INFO:     192.168.65.1:63832 - "POST /api/v1/documents/docling HTTP/1.1" 422 Unprocessable Entity

Docker Container Logs (docling):

INFO: Docling service running on port 5001
INFO: Health check endpoint accessible

Additional Information

Configuration Details:

  • Open WebUI Version: Latest main branch (docker image)
  • Ollama Version: 0.5.2
  • Docling Version: Latest (ds4sd/docling:latest)
  • Browser: Chrome 131.0.6778.86 (also tested Firefox, Safari)
  • OS: macOS Sequoia 15.1
  • Docker Desktop: 4.36.0

Environment Variables:

OLLAMA_BASE_URL=http://192.168.68.87:11434
DOCLING_SERVER_URL=http://docling:5001

Test JSON Configurations Attempted:

// Standard format
{
  "url": "http://192.168.68.87:11434/v1/chat/completions",
  "params": {
    "model": "qwen2.5vl:3b"
  }
}


// Minimal format
{"url": "http://192.168.68.87:11434/v1/chat/completions", "model": "qwen2.5vl:3b"}


// Flat structure
{
  "url": "http://192.168.68.87:11434/v1/chat/completions",
  "model": "qwen2.5vl:3b",
  "temperature": 0.1
}


// All produce the same error pattern

Additional Information

Root Cause Analysis:
The error pattern suggests that the frontend or backend code is incorrectly processing the JSON input by:

  1. Splitting the JSON string (possibly by keys or delimiters)
  2. Iterating through individual field names
  3. Attempting to validate each field name as a complete dictionary object
  4. Failing validation when a string is passed instead of an object

Impact:

  • Severity: High - Complete feature dysfunction
  • Affected Users: Anyone attempting to use Docling picture description
  • Current Workaround: Use Docling API directly, bypassing Open WebUI

Verification that Docling works independently:
Direct API call to Docling with the same JSON configuration works correctly:

curl -X POST http://localhost:5001/v1/documents/convert \
  -H "Content-Type: application/json" \
  -d '{
    "do_picture_description": true,
    "picture_description_api": {
      "url": "http://192.168.68.87:11434/v1/chat/completions",
      "params": {"model": "qwen2.5vl:3b"}
    }
  }'

This confirms the issue is in Open WebUI's JSON parsing/validation, not in Docling itself.

Additional Testing:

  • Tested multiple JSON escaping strategies (double-escaped, base64, URL-encoded)
  • Tested alternative formats (YAML-style, key-value pairs)
  • Tested both picture_description_local and picture_description_api fields
  • All exhibit the same systematic parsing bug

This bug prevents the Docling picture description feature from being usable through the Open WebUI interface and requires investigation of the JSON parsing logic in the configuration validation system.

Originally created by @healdgar on GitHub (Jun 15, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/15002 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.14 ### Ollama Version (if applicable) 0.9.0 ### Operating System ghcr.io/open-webui/open-webui:main ### Browser (if applicable) Chrome (latest) ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When entering valid JSON configuration in either the `picture_description_local` or `picture_description_api` fields in the Docling settings, the configuration should: 1. Parse the JSON as a complete object 2. Validate the JSON structure properly 3. Save the configuration successfully 4. Enable picture description functionality in Docling 5. Process documents with image description capabilities The UI tooltip states "Enter config in JSON format" which implies JSON should be accepted and parsed correctly. ### Actual Behavior Any JSON configuration entered in either `picture_description_local` or `picture_description_api` fields results in validation errors where individual JSON field names are treated as strings instead of parsing the complete JSON object. **Error Pattern:** ``` Error calling Docling: Error calling Docling API: Unprocessable Entity - [{'type': 'model_attributes_type', 'loc': ['body', 'picture_description_[local|api]'], 'msg': 'Input should be a valid dictionary or object to extract fields from', 'input': '[field_name]'}] ``` The `'input'` value changes to individual field names like: - `'url'` - `'params'` - `'model'` - `'temperature'` - `'options'` This indicates the JSON is being split and individual keys are being validated as complete objects, which fails. ### Steps to Reproduce **Environment Setup:** 1. Start with macOS system with Docker Desktop installed 2. Use latest Open WebUI Docker image (docker pull ghcr.io/open-webui/open-webui:main) 3. Run Ollama locally with qwen2.5vl:3b model installed 4. Start Docling service on port 5001 **Docker Compose Configuration:** ```yaml services: open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://192.168.68.87:11434 - DOCLING_SERVER_URL=http://docling:5001 depends_on: - docling docling: image: ds4sd/docling:latest ports: - "5001:5001" ``` **Reproduction Steps:** 1. Start services with `docker compose up -d` 2. Open browser (tested in Chrome 131.0, Safari 18.1, Firefox 133.0) 3. Navigate to `http://localhost:3000` 4. Log in as admin user 5. Go to Admin Panel → Settings → Documents → Docling 6. Scroll to "Picture Description" section 7. Enable "Picture Description" toggle 8. In the "Picture Description API Config" field, enter this JSON: ```json { "url": "http://192.168.68.87:11434/v1/chat/completions", "params": { "model": "qwen2.5vl:3b" } } ``` 9. Click "Save" 10. Upload a PDF document with images 11. Observe the error in the UI **Alternative test with minimal JSON:** 1. Clear the field and enter minimal JSON: ```json {"url": "http://192.168.68.87:11434/v1/chat/completions", "model": "qwen2.5vl:3b"} ``` 2. Save configuration 3. Same error occurs **Test with picture_description_local field:** 1. Clear API config field 2. Enter same JSON in "Picture Description Local Config" field 3. Save configuration 4. Same error pattern occurs ### Logs & Screenshots **Browser Console Error:** ``` POST http://localhost:3000/api/v1/documents/docling 422 (Unprocessable Entity) ``` **Detailed Error Response:** ```json { "detail": [ { "type": "model_attributes_type", "loc": ["body", "picture_description_api"], "msg": "Input should be a valid dictionary or object to extract fields from", "input": "params" } ] } ``` **Docker Container Logs (open-webui):** ``` INFO: 192.168.65.1:63832 - "POST /api/v1/documents/docling HTTP/1.1" 422 Unprocessable Entity ``` **Docker Container Logs (docling):** ``` INFO: Docling service running on port 5001 INFO: Health check endpoint accessible ``` ### Additional Information **Configuration Details:** - Open WebUI Version: Latest main branch (docker image) - Ollama Version: 0.5.2 - Docling Version: Latest (ds4sd/docling:latest) - Browser: Chrome 131.0.6778.86 (also tested Firefox, Safari) - OS: macOS Sequoia 15.1 - Docker Desktop: 4.36.0 **Environment Variables:** ``` OLLAMA_BASE_URL=http://192.168.68.87:11434 DOCLING_SERVER_URL=http://docling:5001 ``` **Test JSON Configurations Attempted:** ```json // Standard format { "url": "http://192.168.68.87:11434/v1/chat/completions", "params": { "model": "qwen2.5vl:3b" } } // Minimal format {"url": "http://192.168.68.87:11434/v1/chat/completions", "model": "qwen2.5vl:3b"} // Flat structure { "url": "http://192.168.68.87:11434/v1/chat/completions", "model": "qwen2.5vl:3b", "temperature": 0.1 } // All produce the same error pattern ``` ## Additional Information **Root Cause Analysis:** The error pattern suggests that the frontend or backend code is incorrectly processing the JSON input by: 1. Splitting the JSON string (possibly by keys or delimiters) 2. Iterating through individual field names 3. Attempting to validate each field name as a complete dictionary object 4. Failing validation when a string is passed instead of an object **Impact:** - Severity: High - Complete feature dysfunction - Affected Users: Anyone attempting to use Docling picture description - Current Workaround: Use Docling API directly, bypassing Open WebUI **Verification that Docling works independently:** Direct API call to Docling with the same JSON configuration works correctly: ```bash curl -X POST http://localhost:5001/v1/documents/convert \ -H "Content-Type: application/json" \ -d '{ "do_picture_description": true, "picture_description_api": { "url": "http://192.168.68.87:11434/v1/chat/completions", "params": {"model": "qwen2.5vl:3b"} } }' ``` This confirms the issue is in Open WebUI's JSON parsing/validation, not in Docling itself. **Additional Testing:** - Tested multiple JSON escaping strategies (double-escaped, base64, URL-encoded) - Tested alternative formats (YAML-style, key-value pairs) - Tested both `picture_description_local` and `picture_description_api` fields - All exhibit the same systematic parsing bug This bug prevents the Docling picture description feature from being usable through the Open WebUI interface and requires investigation of the JSON parsing logic in the configuration validation system.
GiteaMirror added the bug label 2026-05-25 06:24:55 -05:00
Author
Owner

@tjbck commented on GitHub (Jun 16, 2025):

#14903 Most likely related, confirmation wanted here!

<!-- gh-comment-id:2975180848 --> @tjbck commented on GitHub (Jun 16, 2025): #14903 Most likely related, confirmation wanted here!
Author
Owner

@healdgar commented on GitHub (Jun 16, 2025):

Yes, I think https://github.com/open-webui/open-webui/pull/14903 may have solved the issue. I will pull the image and test. If you don't hear anything from me in an hour, it means it was resolve and you can close this.

The Fix: The PR serializes the content before sending it to Docling
Exact Problem: Open WebUI was trying to parse the JSON and send individual fields instead of the complete JSON string

<!-- gh-comment-id:2977503677 --> @healdgar commented on GitHub (Jun 16, 2025): Yes, I think https://github.com/open-webui/open-webui/pull/14903 may have solved the issue. I will pull the image and test. If you don't hear anything from me in an hour, it means it was resolve and you can close this. The Fix: The PR serializes the content before sending it to Docling Exact Problem: Open WebUI was trying to parse the JSON and send individual fields instead of the complete JSON string
Author
Owner

@healdgar commented on GitHub (Jun 16, 2025):

It's not working still. It seems as though Docling expects an object and not a JSON string? This might be a Docling-side parsing bug.

<!-- gh-comment-id:2977700259 --> @healdgar commented on GitHub (Jun 16, 2025): It's not working still. It seems as though Docling expects an object and not a JSON string? This might be a Docling-side parsing bug.
Author
Owner

@vaclcer commented on GitHub (Jun 17, 2025):

Hmm, I can confirm it was functioning correctly in my feature branch, as shown at pictures in https://github.com/open-webui/open-webui/pull/14903

I will investigate what is going on..


Edit: And it still is (with v0.6.15)! Just make sure you are using a valid JSON, for example:

Edit 2: docs updated https://github.com/open-webui/docs/pull/584

Image Image
<!-- gh-comment-id:2979650359 --> @vaclcer commented on GitHub (Jun 17, 2025): Hmm, I can confirm it was functioning correctly in my feature branch, as shown at pictures in https://github.com/open-webui/open-webui/pull/14903 I will investigate what is going on.. ----- Edit: And it still is (with v0.6.15)! Just make sure you are using a valid JSON, for example: Edit 2: docs updated https://github.com/open-webui/docs/pull/584 <img width="535" alt="Image" src="https://github.com/user-attachments/assets/b6c3d980-dd26-4748-ae48-ecdfb67071f6" /> <img width="595" alt="Image" src="https://github.com/user-attachments/assets/84363ec9-d9b7-4052-8453-bc44c6f1d52c" />
Author
Owner

@arunbugkiller commented on GitHub (Jun 20, 2025):

@vaclcer @healdgar I have been experimenting with self-hosted docling, as it happens it is taking too long to process a simple PDF file. Any recommendation on the underlying hardware based on your experience with the same. I am currently hosting it on a 32GB RAM, 16 core vCPU (shared) server.

<!-- gh-comment-id:2989814217 --> @arunbugkiller commented on GitHub (Jun 20, 2025): @vaclcer @healdgar I have been experimenting with self-hosted docling, as it happens it is taking too long to process a simple PDF file. Any recommendation on the underlying hardware based on your experience with the same. I am currently hosting it on a 32GB RAM, 16 core vCPU (shared) server.
Author
Owner

@vaclcer commented on GitHub (Jun 20, 2025):

picture description feature is by default using a VLM, so I would guess you need a GPU to make it reasonable fast.

Since this is not an issue anymore, I would suggest continuing in a Discussion ..

<!-- gh-comment-id:2989993419 --> @vaclcer commented on GitHub (Jun 20, 2025): picture description feature is by default using a VLM, so I would guess you need a GPU to make it reasonable fast. Since this is not an issue anymore, I would suggest continuing in a Discussion ..
Author
Owner

@badgerde commented on GitHub (Jul 4, 2025):

I am using openwebui 0.6.15 and running docling in a separate docker which works fine without image description.
Getting picture description to work seems still impossible for me,
I have been trying using the configuration examples provided, but to no success.

fastapi.exceptions.HTTPException: 400: Error calling Docling: Error calling Docling API: Unprocessable Entity - [{'type': 'model_attributes_type', 'loc': ['body', 'picture_description_local'], 'msg': 'Input should be a valid dictionary or object to extract fields from', 'input': '{"repo_id": "HuggingFaceTB/SmolVLM-256-Instruct", "generation_config": {"max_new_tokens": 200, "do_sample": false}, "prompt": "Describe this image in a few sentences."}'}]

This is the config used (taken from docling.md)

{
"repo_id": "HuggingFaceTB/SmolVLM-256-Instruct",
"generation_config": {
"max_new_tokens": 200,
"do_sample": false
},
"prompt": "Describe this image in a few sentences."
}

<!-- gh-comment-id:3035085624 --> @badgerde commented on GitHub (Jul 4, 2025): I am using openwebui 0.6.15 and running docling in a separate docker which works fine without image description. Getting picture description to work seems still impossible for me, I have been trying using the configuration examples provided, but to no success. fastapi.exceptions.HTTPException: 400: Error calling Docling: Error calling Docling API: Unprocessable Entity - [{'type': 'model_attributes_type', 'loc': ['body', 'picture_description_local'], 'msg': 'Input should be a valid dictionary or object to extract fields from', 'input': '{"repo_id": "HuggingFaceTB/SmolVLM-256-Instruct", "generation_config": {"max_new_tokens": 200, "do_sample": false}, "prompt": "Describe this image in a few sentences."}'}] This is the config used (taken from docling.md) { "repo_id": "HuggingFaceTB/SmolVLM-256-Instruct", "generation_config": { "max_new_tokens": 200, "do_sample": false }, "prompt": "Describe this image in a few sentences." }
Author
Owner

@Soliver84 commented on GitHub (Jul 18, 2025):

Yes, they should release a patch as soon as possible.

So that Docling works with /V1/ and the picture problem is fixed.

<!-- gh-comment-id:3086947504 --> @Soliver84 commented on GitHub (Jul 18, 2025): Yes, they should release a patch as soon as possible. So that Docling works with /V1/ and the picture problem is fixed.
Author
Owner

@rgaricano commented on GitHub (Jul 18, 2025):

there are already a PR that address it: https://github.com/open-webui/open-webui/pull/15785

<!-- gh-comment-id:3088617462 --> @rgaricano commented on GitHub (Jul 18, 2025): there are already a PR that address it: https://github.com/open-webui/open-webui/pull/15785
Author
Owner

@ju6ge commented on GitHub (Mar 5, 2026):

Hey guys,

I am hitting on the same issue as described here, just with a different field from the docling api, i am using the vlm_pipeline_model_api parameter. But I think this is a more general problem with the implementation in OpenWebUi and not a problem on the docling side of things. #14903 only partly addresses this for the fields that where relevant to the submitter.

I would have opened a separate Issue for this, but the Issue creation process requests a lot of information that is entirely irrelevant to the problem I am describing. So I have opted to but the information here instead. I hope this does not go under as this Issue is marked as closed.

Generally there is a difference between using docling/v1/convert/url and docling/v1/convert/file api endpoints. The difference being that v1/convert/url expects a json body while the docling/v1/convert/file endpoint expects multipart/form-data this is where the issue comes in, when OpenWebUi sends the json in a convert file request all complex json fields need to have the same work around applied as in #14903 that is quite a few fields. If any field is no converted to a json string using json.dumps it will cause issues with deserialization on the docling serve endpoint.

Kind regards,
ju6ge

<!-- gh-comment-id:4004904492 --> @ju6ge commented on GitHub (Mar 5, 2026): Hey guys, I am hitting on the same issue as described here, just with a different field from the docling api, i am using the `vlm_pipeline_model_api` parameter. But I think this is a more general problem with the implementation in OpenWebUi and not a problem on the `docling` side of things. #14903 only partly addresses this for the fields that where relevant to the submitter. I would have opened a separate Issue for this, but the Issue creation process requests a lot of information that is entirely irrelevant to the problem I am describing. So I have opted to but the information here instead. I hope this does not go under as this Issue is marked as closed. Generally there is a difference between using `docling/v1/convert/url` and `docling/v1/convert/file` api endpoints. The difference being that `v1/convert/url` expects a json body while the `docling/v1/convert/file` endpoint expects `multipart/form-data` this is where the issue comes in, when OpenWebUi sends the json in a convert file request all complex json fields need to have the same work around applied as in #14903 that is quite a few fields. If any field is no converted to a json string using `json.dumps` it will cause issues with deserialization on the docling serve endpoint. Kind regards, ju6ge
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#137165