[GH-ISSUE #12059] feat: Add an embedding option that specifies document separator characters #31984

New Issue

GiteaMirror · 2026-04-25T05:52:07-05:00

GiteaMirror commented

2026-04-25 05:52:07 -05:00

Originally created by @fgfg54321 on GitHub (Mar 26, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12059

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

Add an embedding option that specifies document separator characters

Desired Solution you'd like

In retrieval. py files the RecursiveCharacterTextSplitter separators options can be set environment variables or the background It can be more flexible processing document segmentation

now i manually modify the code here
separators_str = "\n\n\n\n,.,。"
separators = [s.strip() for s in separators_str.split(",") if s.strip()]

text_splitter = RecursiveCharacterTextSplitter(
separators=separators, # New separator parameter
chunk_size=request.app.state.config.CHUNK_SIZE,
chunk_overlap=request.app.state.config.CHUNK_OVERLAP,
add_start_index=True,
)

Alternatives Considered

No response

Additional Context

No response

Originally created by @fgfg54321 on GitHub (Mar 26, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/12059 ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description Add an embedding option that specifies document separator characters ### Desired Solution you'd like In retrieval. py files the RecursiveCharacterTextSplitter separators options can be set environment variables or the background It can be more flexible processing document segmentation now i manually modify the code here separators_str = "\n\n\n\n,.,。" separators = [s.strip() for s in separators_str.split(",") if s.strip()] text_splitter = RecursiveCharacterTextSplitter( separators=separators, # New separator parameter chunk_size=request.app.state.config.CHUNK_SIZE, chunk_overlap=request.app.state.config.CHUNK_OVERLAP, add_start_index=True, ) ### Alternatives Considered _No response_ ### Additional Context _No response_

GiteaMirror closed this issue

2026-04-25 05:52:08 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#31984