[PR #2731] [MERGED] fix: ollama and openai stream cancellation #7869

New Issue

GiteaMirror · 2025-11-11T17:38:49-06:00

GiteaMirror commented

2025-11-11 17:38:49 -06:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/2731
Author: @cheahjs
Created: 6/2/2024
Status: ✅ Merged
Merged: 6/2/2024
Merged by: @tjbck

Base: dev ← Head: fix/ollama-cancellation

📝 Commits (5)

24c35c3 fix: stream defaults to true, return request ID
4dd51ba fix: ollama streaming cancellation using aiohttp
7f74426 fix: openai streaming cancellation using aiohttp
b5b2b70 fix: bad payload refactor
c5ff4c2 Merge branch 'dev' into fix/ollama-cancellation

📊 Changes

7 files changed (+186 additions, -503 deletions)

View changed files

📝 backend/apps/ollama/main.py (+54 -347)
📝 backend/apps/openai/main.py (+29 -10)
📝 src/lib/apis/ollama/index.ts (+3 -22)
📝 src/lib/components/chat/Chat.svelte (+50 -63)
📝 src/lib/components/chat/ModelSelector/Selector.svelte (+21 -21)
📝 src/lib/components/chat/Settings/Models.svelte (+28 -27)
📝 src/lib/components/workspace/Playground.svelte (+1 -13)

📄 Description

Pull Request Checklist

Before submitting, make sure you've checked the following:

Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Have you written and run sufficient tests for validating the changes?
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Label: To cleary categorize this pull request, assign a relevant label to the pull request title, using one of the following:
- BREAKING CHANGE: Significant changes that may affect compatibility
- build: Changes that affect the build system or external dependencies
- ci: Changes to our continuous integration processes or workflows
- chore: Refactor, cleanup, or other non-functional code changes
- docs: Documentation update or addition
- feat: Introduces a new feature or enhancement to the codebase
- fix: Bug fix or error correction
- i18n: Internationalization or localization changes
- perf: Performance improvement
- refactor: Code restructuring for better maintainability, readability, or scalability
- style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
- test: Adding missing tests or correcting existing tests
- WIP: Work in progress, a temporary label for incomplete or ongoing work

Changelog Entry

Description

There was some wonky behaviour with the previous approach to cancelling Ollama requests, where Response.close() was not being called at the right time (https://github.com/psf/requests/issues/5372 might be related, but didn't dig too deep). Instead, fix cancellation of streaming responses on both Ollama and OpenAI by:

switching from requests to aiohttp for making streaming requests
using Starlette's Background Task to run close() when the response is done
switch from the explicit request_id cancellation model to using an AbortController on the frontend, and closing the connection on the frontend.

The request ID was also broken because it was only sent if stream: true was set in the request, but Ollama's API defaults it to true if absent, and the frontend doesn't send stream: true for streaming chat completions.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/2731 **Author:** [@cheahjs](https://github.com/cheahjs) **Created:** 6/2/2024 **Status:** ✅ Merged **Merged:** 6/2/2024 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `fix/ollama-cancellation` --- ### 📝 Commits (5) - [`24c35c3`](https://github.com/open-webui/open-webui/commit/24c35c308d75140717cab82a15a06d3fac55ab89) fix: stream defaults to true, return request ID - [`4dd51ba`](https://github.com/open-webui/open-webui/commit/4dd51badfe2be1a0a85b817a1325227e92938949) fix: ollama streaming cancellation using aiohttp - [`7f74426`](https://github.com/open-webui/open-webui/commit/7f74426a228d592cf4822eff94220547e0142e04) fix: openai streaming cancellation using aiohttp - [`b5b2b70`](https://github.com/open-webui/open-webui/commit/b5b2b70f4a4bc8e608f16e6c9febddaa2e110d84) fix: bad payload refactor - [`c5ff4c2`](https://github.com/open-webui/open-webui/commit/c5ff4c24e16737dd59e7fd63538a543d86b77da1) Merge branch 'dev' into fix/ollama-cancellation ### 📊 Changes **7 files changed** (+186 additions, -503 deletions) <details> <summary>View changed files</summary> 📝 `backend/apps/ollama/main.py` (+54 -347) 📝 `backend/apps/openai/main.py` (+29 -10) 📝 `src/lib/apis/ollama/index.ts` (+3 -22) 📝 `src/lib/components/chat/Chat.svelte` (+50 -63) 📝 `src/lib/components/chat/ModelSelector/Selector.svelte` (+21 -21) 📝 `src/lib/components/chat/Settings/Models.svelte` (+28 -27) 📝 `src/lib/components/workspace/Playground.svelte` (+1 -13) </details> ### 📄 Description # Pull Request Checklist **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Please verify that the pull request targets the `dev` branch. - [x] **Description:** Provide a concise description of the changes made in this pull request. - [ ] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [ ] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources? - [ ] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation? - [x] **Testing:** Have you written and run sufficient tests for validating the changes? - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Label:** To cleary categorize this pull request, assign a relevant label to the pull request title, using one of the following: - **BREAKING CHANGE**: Significant changes that may affect compatibility - **build**: Changes that affect the build system or external dependencies - **ci**: Changes to our continuous integration processes or workflows - **chore**: Refactor, cleanup, or other non-functional code changes - **docs**: Documentation update or addition - **feat**: Introduces a new feature or enhancement to the codebase - **fix**: Bug fix or error correction - **i18n**: Internationalization or localization changes - **perf**: Performance improvement - **refactor**: Code restructuring for better maintainability, readability, or scalability - **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.) - **test**: Adding missing tests or correcting existing tests - **WIP**: Work in progress, a temporary label for incomplete or ongoing work # Changelog Entry ### Description There was some wonky behaviour with the previous approach to cancelling Ollama requests, where `Response.close()` was not being called at the right time (https://github.com/psf/requests/issues/5372 might be related, but didn't dig too deep). Instead, fix cancellation of streaming responses on both Ollama and OpenAI by: 1. switching from `requests` to `aiohttp` for making streaming requests 2. using Starlette's [Background Task](https://www.starlette.io/background/) to run `close()` when the response is done 3. switch from the explicit request_id cancellation model to using an `AbortController` on the frontend, and closing the connection on the frontend. The request ID was also broken because it was only sent if `stream: true` was set in the request, but Ollama's API defaults it to true if absent, and the frontend doesn't send `stream: true` for streaming chat completions. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2025-11-11 17:38:49 -06:00

GiteaMirror closed this issue

2025-11-11 17:38:49 -06:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#7869