The chunking is done only after punctuation, ignoring newlines. I noticed that by looking at the logs from my openedai speech instance. The issue is that markdown bullet points frequently end abruptly with no punctuation.
A newline, albeit not strictly a "punctuation" should be a splitting delimiter when we set "punctuation" as the splitter. Also, each splitted sentence should have "str.strip()" applied of course.
Here's a particularly broken text:
Additional Information
Pinging @kiosion as they introduced the punctuation split in #4886
Originally created by @thiswillbeyourgithub on GitHub (Oct 5, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/5924
# Bug Report
**Confirmation:**
- [X] I have read and followed all the instructions provided in the README.md.
- [X] I am on the latest version of both Open WebUI and Ollama.
## Expected Behavior:
The speech model should consider two lines with no punctuation as two sentences, instead of a single sentence containing a newline between them.
## Actual Behavior:
2 lines with no punctuation are considered a single sentence. So the speech model receives unsplited text when I use markdown bullet point lists.
## Description
If I have the choice between splitting on paragraphs or on punctuation, it is implied that the latter will always result in smaller chunks. But as we can see at this line:
https://github.com/open-webui/open-webui/blob/1d225dd804575af9ae5981528dfdce695f7f7040/src/lib/utils/index.ts#L543
The chunking is done only after punctuation, ignoring newlines. I noticed that by looking at the logs from my [openedai speech](https://github.com/matatonic/openedai-speech/) instance. The issue is that markdown bullet points frequently end abruptly with no punctuation.
A newline, albeit not strictly a "punctuation" should be a splitting delimiter when we set "punctuation" as the splitter. Also, each splitted sentence should have "str.strip()" applied of course.
Here's a particularly broken text:
## Additional Information
Pinging @kiosion as they introduced the punctuation split in #4886
GiteaMirror
added the bug label 2026-05-20 19:00:15 -05:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @thiswillbeyourgithub on GitHub (Oct 5, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/5924
Bug Report
Confirmation:
Expected Behavior:
The speech model should consider two lines with no punctuation as two sentences, instead of a single sentence containing a newline between them.
Actual Behavior:
2 lines with no punctuation are considered a single sentence. So the speech model receives unsplited text when I use markdown bullet point lists.
Description
If I have the choice between splitting on paragraphs or on punctuation, it is implied that the latter will always result in smaller chunks. But as we can see at this line:
https://github.com/open-webui/open-webui/blob/1d225dd804575af9ae5981528dfdce695f7f7040/src/lib/utils/index.ts#L543
The chunking is done only after punctuation, ignoring newlines. I noticed that by looking at the logs from my openedai speech instance. The issue is that markdown bullet points frequently end abruptly with no punctuation.
A newline, albeit not strictly a "punctuation" should be a splitting delimiter when we set "punctuation" as the splitter. Also, each splitted sentence should have "str.strip()" applied of course.
Here's a particularly broken text:
Additional Information
Pinging @kiosion as they introduced the punctuation split in #4886