Description: Briefly describe the changes in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation?
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Description
Slightly controversial change, for OpenAI streamed responses, check if an individual chunk is >5 characters. If so, randomly split it into chunks of 1-3 characters each, and feed them into the frontend 5ms apart.
This is to improve the perceived responsiveness of the webapp when a provider streams large chunks of text instead of streaming each token as generated, such as the Gemini or Anthropic models.
Potential side-effect, for models that return large chunks and are able to generate a lot of tokens very fast will be limited in the speed at which the frontend will render responses, since it'll only emit 1-3 characters every 5 ms.
Examples:
Cohere's Command-R+'s behaviour pre-change as an example of a provider that streams in per-token:
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/1630
**Author:** [@cheahjs](https://github.com/cheahjs)
**Created:** 4/20/2024
**Status:** ✅ Merged
**Merged:** 4/22/2024
**Merged by:** [@tjbck](https://github.com/tjbck)
**Base:** `dev` ← **Head:** `feat/split-large-chunks`
---
### 📝 Commits (2)
- [`efa258c`](https://github.com/open-webui/open-webui/commit/efa258c69504aff9fab6c47f62e16dce42665318) feat: split large openai responses into smaller chunkers
- [`67df928`](https://github.com/open-webui/open-webui/commit/67df928c7ae953e4b725c548de08c0b61ce7d1e6) feat: make chunk splitting a configurable option
### 📊 Changes
**5 files changed** (+121 additions, -52 deletions)
<details>
<summary>View changed files</summary>
➕ `src/lib/apis/streaming/index.ts` (+70 -0)
📝 `src/lib/components/chat/Settings/Interface.svelte` (+28 -0)
📝 `src/lib/i18n/locales/en-US/translation.json` (+1 -0)
📝 `src/routes/(app)/+page.svelte` (+11 -26)
📝 `src/routes/(app)/c/[id]/+page.svelte` (+11 -26)
</details>
### 📄 Description
## Pull Request Checklist
- [x] **Description:** Briefly describe the changes in this pull request.
- [ ] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description.
- [ ] **Documentation:** Have you updated relevant documentation?
- [ ] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation?
---
## Description
Slightly controversial change, for OpenAI streamed responses, check if an individual chunk is >5 characters. If so, randomly split it into chunks of 1-3 characters each, and feed them into the frontend 5ms apart.
This is to improve the perceived responsiveness of the webapp when a provider streams large chunks of text instead of streaming each token as generated, such as the Gemini or Anthropic models.
Potential side-effect, for models that return large chunks and are able to generate a lot of tokens very fast will be limited in the speed at which the frontend will render responses, since it'll only emit 1-3 characters every 5 ms.
**Examples:**
* Cohere's Command-R+'s behaviour pre-change as an example of a provider that streams in per-token:
https://github.com/open-webui/open-webui/assets/818368/255ae2a1-fe98-4e9a-8653-e75f77f127bc
* Cohere's Command-R+'s behaviour post-change:
https://github.com/open-webui/open-webui/assets/818368/00373b6a-555f-4e16-a546-8452a9800848
* Google's Gemini 1.5 Pro's behaviour pre-change as an example of a provider that streams big chunks:
https://github.com/open-webui/open-webui/assets/818368/2e8a4b35-208e-43c7-9c64-13522ac1e673
* Google's Gemini 1.5 Pro's behaviour post-change:
https://github.com/open-webui/open-webui/assets/818368/92d932fe-c6e2-4784-9038-ecbeeeda696b
---
### Changelog Entry
### Added
- [List any new features or additions]
### Fixed
- [List any fixes or corrections]
### Changed
- [List any changes or updates]
### Removed
- [List any removed features or files]
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/1630
Author: @cheahjs
Created: 4/20/2024
Status: ✅ Merged
Merged: 4/22/2024
Merged by: @tjbck
Base:
dev← Head:feat/split-large-chunks📝 Commits (2)
efa258cfeat: split large openai responses into smaller chunkers67df928feat: make chunk splitting a configurable option📊 Changes
5 files changed (+121 additions, -52 deletions)
View changed files
➕
src/lib/apis/streaming/index.ts(+70 -0)📝
src/lib/components/chat/Settings/Interface.svelte(+28 -0)📝
src/lib/i18n/locales/en-US/translation.json(+1 -0)📝
src/routes/(app)/+page.svelte(+11 -26)📝
src/routes/(app)/c/[id]/+page.svelte(+11 -26)📄 Description
Pull Request Checklist
Description
Slightly controversial change, for OpenAI streamed responses, check if an individual chunk is >5 characters. If so, randomly split it into chunks of 1-3 characters each, and feed them into the frontend 5ms apart.
This is to improve the perceived responsiveness of the webapp when a provider streams large chunks of text instead of streaming each token as generated, such as the Gemini or Anthropic models.
Potential side-effect, for models that return large chunks and are able to generate a lot of tokens very fast will be limited in the speed at which the frontend will render responses, since it'll only emit 1-3 characters every 5 ms.
Examples:
https://github.com/open-webui/open-webui/assets/818368/255ae2a1-fe98-4e9a-8653-e75f77f127bc
https://github.com/open-webui/open-webui/assets/818368/00373b6a-555f-4e16-a546-8452a9800848
https://github.com/open-webui/open-webui/assets/818368/2e8a4b35-208e-43c7-9c64-13522ac1e673
https://github.com/open-webui/open-webui/assets/818368/92d932fe-c6e2-4784-9038-ecbeeeda696b
Changelog Entry
Added
Fixed
Changed
Removed
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.