github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-11-09 10:50:35 -06:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #24870] feat: Add per-file control to disable chunking and embedding for individually uploaded documents #123728

New Issue
Open
opened 2026-05-21 03:12:24 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-05-21 03:12:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @raocg on GitHub (May 18, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24870

Check Existing Issues

  • I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

  • I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

Problem Statement

Currently, there is no granular control over how individually uploaded documents (files added directly to a chat context via "Upload Files") are processed compared to documents in the Knowledge Base. When users upload files for a specific conversation, these files may be subject to the same chunking, embedding, and retrieval pipeline configured globally.

Use Cases Requiring Per-File Control

  1. Large Documents: Users may want to keep large PDFs or documents intact without chunking them into smaller pieces for context.

  2. Structured Data: Some users prefer to pass structured data (JSON, CSV) or code files directly to the model without embedding/retrieval processing.

  3. Context Documents: Users may have reference documents that should be included as-is in the context window without being split and embedded.

  4. API Efficiency: Per-file control could reduce unnecessary processing costs when files don't need the full RAG pipeline.

Proposed Solution

Add a per-file configuration option when uploading documents that allows users to choose:

  • Process with Embedding & Retrieval: Use the global document processing settings (chunking, embedding, etc.)
  • Include as Raw Context: Add the document to the context without chunking or embedding

This could be implemented as:

  1. A checkbox/toggle in the upload dialog
  2. A dropdown menu in the file attachment UI
  3. A setting in the chat context panel

Benefits

  • Flexibility: Different documents can be processed differently based on their purpose
  • Cost Efficiency: Reduce unnecessary embedding operations
  • Improved Context Management: Better control over what gets retrieved vs. what's used as direct context
  • User Experience: More intuitive handling of various document types

Desired Solution you'd like

Proposed Implementation

Option 1: Simple Toggle in Upload Dialog (Recommended for MVP)

When clicking "Upload Files", show a checkbox: "[ ] Chunk and Embed for Retrieval"

  • Checked (default): Process normally with chunking and embedding
  • Unchecked: Include as raw context without processing

Option 2: Advanced Control Panel

Add a sidebar panel showing:

  • File list with icons
  • Per-file processing mode selector (dropdown/toggle)
  • Preview of how each file will be handled
  • Ability to change mode before sending the message

Option 3: Integration with Attachment Options

Extend the existing "Attach Knowledge" feature to show:

  • Attach from Knowledge Base (current behavior)
  • Attach Raw File (new option - no processing)
  • Attach with Custom Processing (advanced settings)

Implementation Details

  1. UI Changes: Minimal - just add a toggle in the file upload interface
  2. Backend Changes: Add a process_method parameter to document handling:
    • 'full_rag' (default): Existing chunking + embedding pipeline
    • 'raw_context': Add file directly without processing
  3. Database: Update document metadata to track processing method
  4. API: Expose the option through the REST API for programmatic control

Testing Suggestions

  • Upload same file with both methods and compare context usage
  • Verify large files remain intact when using 'raw_context'
  • Ensure knowledge base files are unaffected
  • Test mixed scenarios (some files raw, some processed)

Alternatives Considered

No response

Additional Context

No response

Originally created by @raocg on GitHub (May 18, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24870 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description ## Problem Statement Currently, there is no granular control over how individually uploaded documents (files added directly to a chat context via "Upload Files") are processed compared to documents in the Knowledge Base. When users upload files for a specific conversation, these files may be subject to the same chunking, embedding, and retrieval pipeline configured globally. ### Use Cases Requiring Per-File Control 1. **Large Documents**: Users may want to keep large PDFs or documents intact without chunking them into smaller pieces for context. 2. **Structured Data**: Some users prefer to pass structured data (JSON, CSV) or code files directly to the model without embedding/retrieval processing. 3. **Context Documents**: Users may have reference documents that should be included as-is in the context window without being split and embedded. 4. **API Efficiency**: Per-file control could reduce unnecessary processing costs when files don't need the full RAG pipeline. ## Proposed Solution Add a per-file configuration option when uploading documents that allows users to choose: - **Process with Embedding & Retrieval**: Use the global document processing settings (chunking, embedding, etc.) - **Include as Raw Context**: Add the document to the context without chunking or embedding This could be implemented as: 1. A checkbox/toggle in the upload dialog 2. A dropdown menu in the file attachment UI 3. A setting in the chat context panel ## Benefits - **Flexibility**: Different documents can be processed differently based on their purpose - **Cost Efficiency**: Reduce unnecessary embedding operations - **Improved Context Management**: Better control over what gets retrieved vs. what's used as direct context - **User Experience**: More intuitive handling of various document types ### Desired Solution you'd like ## Proposed Implementation ### Option 1: Simple Toggle in Upload Dialog (Recommended for MVP) When clicking "Upload Files", show a checkbox: "[ ] Chunk and Embed for Retrieval" - Checked (default): Process normally with chunking and embedding - Unchecked: Include as raw context without processing ### Option 2: Advanced Control Panel Add a sidebar panel showing: - File list with icons - Per-file processing mode selector (dropdown/toggle) - Preview of how each file will be handled - Ability to change mode before sending the message ### Option 3: Integration with Attachment Options Extend the existing "Attach Knowledge" feature to show: - Attach from Knowledge Base (current behavior) - Attach Raw File (new option - no processing) - Attach with Custom Processing (advanced settings) ## Implementation Details 1. **UI Changes**: Minimal - just add a toggle in the file upload interface 2. **Backend Changes**: Add a `process_method` parameter to document handling: - `'full_rag'` (default): Existing chunking + embedding pipeline - `'raw_context'`: Add file directly without processing 3. **Database**: Update document metadata to track processing method 4. **API**: Expose the option through the REST API for programmatic control ## Testing Suggestions - Upload same file with both methods and compare context usage - Verify large files remain intact when using 'raw_context' - Ensure knowledge base files are unaffected - Test mixed scenarios (some files raw, some processed) ### Alternatives Considered _No response_ ### Additional Context _No response_
GiteaMirror commented 2026-05-21 03:12:26 -05:00
Author
Owner
Copy Link
Copy Source

@owui-terminator[bot] commented on GitHub (May 18, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

  1. 🟢 #12228 feat: uploading files without backend processing
    This request is about bypassing document processing for uploaded files so a backend can use the raw file directly. That overlaps strongly with adding a per-file raw-context mode instead of the current chunking/embedding pipeline.
    by wangjiyang

  2. 🟢 #18431 feat: Option to define default mode for file uploads
    This open feature request asks for a default mode for chat file uploads between using the entire document and focused retrieval. It is very close to the new issue’s goal of finer-grained control over processing uploaded files.
    by Master-Pr0grammer

  3. 🟣 #18583 feat: Option to set "Full context mode" for file upload and knowledge retrieval individually.
    This issue specifically discusses full-context mode for uploaded file attachments and contrasts it with knowledge base retrieval. It matches the uploaded-file side of the new request, which wants raw-context handling instead of chunking/embedding.
    by deliciousbob

  4. 🟣 #18144 feat: document upload available raw for other usages
    This closed request asks for a raw upload option that bypasses RAG and keeps the full document available for other uses. That is essentially the same capability as the new per-file raw-context control.
    by MarceloMassarente

  5. 🟣 #3556 Add switch to disable RAG on attachments.
    This issue requests a switch to disable RAG on attachments so the full extracted text can be passed to the LLM. While older and broader, it is directly related to the same attachment-processing control being requested here.
    by nickovs

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:4478626630 --> @owui-terminator[bot] commented on GitHub (May 18, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟢 [#12228](https://github.com/open-webui/open-webui/issues/12228) **feat: uploading files without backend processing** *This request is about bypassing document processing for uploaded files so a backend can use the raw file directly. That overlaps strongly with adding a per-file raw-context mode instead of the current chunking/embedding pipeline.* *by wangjiyang* 2. 🟢 [#18431](https://github.com/open-webui/open-webui/issues/18431) **feat: Option to define default mode for file uploads** *This open feature request asks for a default mode for chat file uploads between using the entire document and focused retrieval. It is very close to the new issue’s goal of finer-grained control over processing uploaded files.* *by Master-Pr0grammer* 3. 🟣 [#18583](https://github.com/open-webui/open-webui/issues/18583) **feat: Option to set "Full context mode" for file upload and knowledge retrieval individually.** *This issue specifically discusses full-context mode for uploaded file attachments and contrasts it with knowledge base retrieval. It matches the uploaded-file side of the new request, which wants raw-context handling instead of chunking/embedding.* *by deliciousbob* 4. 🟣 [#18144](https://github.com/open-webui/open-webui/issues/18144) **feat: document upload available raw for other usages** *This closed request asks for a raw upload option that bypasses RAG and keeps the full document available for other uses. That is essentially the same capability as the new per-file raw-context control.* *by MarceloMassarente* 5. 🟣 [#3556](https://github.com/open-webui/open-webui/issues/3556) **Add switch to disable RAG on attachments.** *This issue requests a switch to disable RAG on attachments so the full extracted text can be passed to the LLM. While older and broader, it is directly related to the same attachment-processing control being requested here.* *by nickovs* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#123728