{"model":"llama3.2-3b-it-q6","messages":[{"role":"user","content":"You're a helpful assistant"}],"stream":true}
Relevant bit of response:
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< cache-control: no-cache
< content-type: text/event-stream; charset=utf-8
< date: Mon, 09 Dec 2024 19:33:20 GMT
< date: Mon, 09 Dec 2024 19:33:20 GMT
< openai-processing-ms: 54
< server: uvicorn
< server: uvicorn
< vary: Accept-Encoding
< vary: Accept-Encoding
< x-accel-buffering: no
< x-process-time: 1
< x-request-id: 4337d48fe6654f5d847867da4fbf9229
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <REDACTED> left intact
(i.e. empty response)
vs
Same, but with stream parameter set to false, i.e.
{"model":"llama3.2-3b-it-q6","messages":[{"role":"user","content":"You're a helpful assistant"}],"stream":false}
response:
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-type: application/json
< date: Mon, 09 Dec 2024 19:35:16 GMT
< server: uvicorn
< vary: Accept-Encoding
< x-process-time: 2
< content-length: 410
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <redacted> left intact
{"id":"chatcmpl-a9b62564-75da-402d-9e0d-5925b7f457d4","object":"chat.completion","created":1733772917,"model":"llama3.2-3b-it-q6","choices":[{"index":0,"message":{"content":"I'm here to assist you with any questions, tasks, or topics you'd like to discuss. What's on your mind?","role":"assistant"},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":26,"total_tokens":41}}
vs
Calling llama-cpp-python directly, with "stream": true
TypeError: Error in input stream Chat.svelte:1798:10
Ue Chat.svelte:1798
fe Chat.svelte:1731
Docker Container Logs:
Server logs appear to indicate that call to API was successful:
INFO: 10.42.7.0:0 - "POST /api/task/tags/completions HTTP/1.1" 200 OK
INFO: 10.42.7.0:0 - "GET /api/v1/chats/1cd9a8ae-afad-40ae-a7a2-24b753d2a6b5 HTTP/1.1" 200 OK
Screenshots/Screen Recordings (if applicable):
Additional Information
Model is being served up by llama-cpp-python, v0.3.2 - can provide more details on this config.
Note
If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!
Originally created by @Gordonei on GitHub (Dec 9, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7726
# Bug Report
Firstly, thanks for this awesome project! I suspect this is maybe related to #7183, but I've tried to put a bit more effort into the bug report.
## Installation Method
Helm chart deploying open-webui to k8s cluster. Using traefik ingress.
## Environment
- **Open WebUI Version:** `v0.4.8`
- **Operating System:** `Ubuntu 20.04`
- **Browser (if applicable):** Firefox 133
**Confirmation:**
- [x] I have read and followed all the instructions provided in the README.md.
- [x] I am on the latest version of both Open WebUI ~~and Ollama~~ (not using Ollama).
- [x] I have included the browser console logs.
- [x] I have included the Docker container logs.
- [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.
## Expected Behavior:
When chatting via the WebUI, content to be streamed out.
## Actual Behavior:
Chat isn't displayed, and there is an error message:
```
Uh-oh! There was an issue connecting to llama3.2-3b-it-q6.
Error in input stream
```
## Description
**Bug Summary:**
It appears there is something wrong in handling streamed content from OpenAI-like endpoints
## Reproduction Details
**Steps to Reproduce:**
*WebUI*
1. Configure an OpenAI API Connections to an endpoint serving up models (e.g. llama-cpp-python)
2. Start a chat with one of the models served up by that API
3. If disabling streaming (*Stream Chat Response: Off*, under Advanced Params on the right), then it works as expected
*API*
```bash
curl -v https://<endpoint>/api/chat/completions -X POST -H "Content-Type: application/json" -H "Authorization: Bearer <censored>" -T ~/scratch/llama3.2-simple-test.json
```
Contents of `llama3.2-simple-test.json`:
```json
{
"model": "llama3.2-3b-it-q6",
"messages": [
{
"role": "user",
"content": "You're a helpful assistant"
}
],
"stream": true
}
```
Relevant bit of response:
```
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< cache-control: no-cache
< content-type: text/event-stream; charset=utf-8
< date: Mon, 09 Dec 2024 19:33:20 GMT
< date: Mon, 09 Dec 2024 19:33:20 GMT
< openai-processing-ms: 54
< server: uvicorn
< server: uvicorn
< vary: Accept-Encoding
< vary: Accept-Encoding
< x-accel-buffering: no
< x-process-time: 1
< x-request-id: 4337d48fe6654f5d847867da4fbf9229
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <REDACTED> left intact
```
(i.e. empty response)
vs
Same, but with `stream` parameter set to false, i.e.
```json
{
"model": "llama3.2-3b-it-q6",
"messages": [
{
"role": "user",
"content": "You're a helpful assistant"
}
],
"stream": false
}
```
response:
```
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-type: application/json
< date: Mon, 09 Dec 2024 19:35:16 GMT
< server: uvicorn
< vary: Accept-Encoding
< x-process-time: 2
< content-length: 410
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <redacted> left intact
{"id":"chatcmpl-a9b62564-75da-402d-9e0d-5925b7f457d4","object":"chat.completion","created":1733772917,"model":"llama3.2-3b-it-q6","choices":[{"index":0,"message":{"content":"I'm here to assist you with any questions, tasks, or topics you'd like to discuss. What's on your mind?","role":"assistant"},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":26,"total_tokens":41}}
```
vs
Calling `llama-cpp-python` directly, with `"stream": true`
```
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< cache-control: no-cache
< content-type: text/event-stream; charset=utf-8
< date: Mon, 09 Dec 2024 19:39:58 GMT
< openai-processing-ms: 55
< server: uvicorn
< vary: Accept-Encoding
< x-accel-buffering: no
< x-request-id: a293313d23a14b36aa9511a782c45dd8
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"role": "assistant"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "I"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "'d"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " be"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " delighted"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " to"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " help"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " you"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " with"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " anything"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "."}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " What"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "'s"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " on"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " your"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " mind"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "?"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " Do"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " you"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " have"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " a"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " question"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": ","}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " need"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " assistance"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " with"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " a"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " task"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": ","}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " or"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " just"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " want"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " to"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " chat"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "?"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " I"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "'m"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " here"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " to"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " listen"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " and"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " provide"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " support"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " in"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " any"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " way"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " I"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": " can"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": "!"}, "logprobs": null, "finish_reason": null}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: {"id": "chatcmpl-81ac59df-6615-4967-9462-a0d4bcb002dd", "model": "llama3.2-3b-it-q6", "created": 1733773199, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {}, "logprobs": null, "finish_reason": "stop"}]}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
data: [DONE]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <REDACTED> left intact
```
## Logs and Screenshots
**Browser Console Logs:**
Error that appears in console:
```
TypeError: Error in input stream Chat.svelte:1798:10
Ue Chat.svelte:1798
fe Chat.svelte:1731
```
**Docker Container Logs:**
Server logs appear to indicate that call to API was successful:
```
INFO: 10.42.7.0:0 - "POST /api/task/tags/completions HTTP/1.1" 200 OK
INFO: 10.42.7.0:0 - "GET /api/v1/chats/1cd9a8ae-afad-40ae-a7a2-24b753d2a6b5 HTTP/1.1" 200 OK
```
**Screenshots/Screen Recordings (if applicable):**
## Additional Information
Model is being served up by `llama-cpp-python`, [v0.3.2](https://github.com/abetlen/llama-cpp-python/releases/tag/v0.3.2) - can provide more details on this config.
## Note
If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @Gordonei on GitHub (Dec 9, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7726
Bug Report
Firstly, thanks for this awesome project! I suspect this is maybe related to #7183, but I've tried to put a bit more effort into the bug report.
Installation Method
Helm chart deploying open-webui to k8s cluster. Using traefik ingress.
Environment
v0.4.8Ubuntu 20.04Confirmation:
and Ollama(not using Ollama).Expected Behavior:
When chatting via the WebUI, content to be streamed out.
Actual Behavior:
Chat isn't displayed, and there is an error message:
Description
Bug Summary:
It appears there is something wrong in handling streamed content from OpenAI-like endpoints
Reproduction Details
Steps to Reproduce:
WebUI
API
Contents of
llama3.2-simple-test.json:Relevant bit of response:
(i.e. empty response)
vs
Same, but with
streamparameter set to false, i.e.response:
vs
Calling
llama-cpp-pythondirectly, with"stream": trueLogs and Screenshots
Browser Console Logs:
Error that appears in console:
Docker Container Logs:
Server logs appear to indicate that call to API was successful:
Screenshots/Screen Recordings (if applicable):
Additional Information
Model is being served up by
llama-cpp-python, v0.3.2 - can provide more details on this config.Note
If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!