I have read and followed all the instructions provided in the README.md.
I am on the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.
Expected Behavior:
Context shouldn't be longer than the number set in any of the options.
Actual Behavior:
It doesn't seem to trim the context regardless of where you set it (general settings, per model or per chat).
Description
Bug Summary:
Setting up a number doesn't trim the context, still sends the whole chat to the API.
Reproduction Details
Steps to Reproduce:
Set up a context length in Admin Console>Models, also in the Personal settings and in the chat settings.
Logs and Screenshots
Screenshots/Screen Recordings (if applicable):
Additional Information
This makes APIs like groq's unusable for longer chats.
Originally created by @nengoxx on GitHub (Dec 21, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7991
# Bug Report
## Installation Method
pip
## Environment
- **Open WebUI Version:** 0.4.7
- **Operating System:** Windows 11
- **Browser (if applicable):** Firefox
**Confirmation:**
- [x] I have read and followed all the instructions provided in the README.md.
- [ ] I am on the latest version of both Open WebUI and Ollama.
- [ ] I have included the browser console logs.
- [ ] I have included the Docker container logs.
- [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.
## Expected Behavior:
Context shouldn't be longer than the number set in any of the options.
## Actual Behavior:
It doesn't seem to trim the context regardless of where you set it (general settings, per model or per chat).
## Description
**Bug Summary:**
Setting up a number doesn't trim the context, still sends the whole chat to the API.
## Reproduction Details
**Steps to Reproduce:**
Set up a context length in Admin Console>Models, also in the Personal settings and in the chat settings.
## Logs and Screenshots
**Screenshots/Screen Recordings (if applicable):**
## Additional Information
This makes APIs like groq's unusable for longer chats.
Is it just for local inference requests with ollama perhaps?
Edit: I didn't have time to check the code this afternoon, but I guessed that was it, did a quick and dirty fix to hardcode it (in the main.py), not sure how usable this is but just in case you wanna check it, and also to have some future reference for myself.
import tiktoken
def count_tokens(messages, model):
encoding = tiktoken.get_encoding("gpt2")
#encoding = tiktoken.encoding_for_model(model) #extract external model name??
total_tokens = 0
for message in messages:
total_tokens += len(encoding.encode(message['role']))
total_tokens += len(encoding.encode(message['content']))
return total_tokens
def trim_context(messages, model, max_tokens):
#Leave the system role.
preserved_messages = [msg for msg in messages if msg['role'] not in ['user', 'assistant']]
trimmable_messages = [msg for msg in messages if msg['role'] in ['user', 'assistant']]
while count_tokens(preserved_messages + trimmable_messages, model) > max_tokens:
trimmable_messages.pop(0)
return preserved_messages + trimmable_messages
Then inside the /chat/completions route:
max_tokens = api_config.get("max_context_tokens", 8192) #Hardcoded max tokens.
if "messages" in payload:
payload["messages"] = trim_context(payload["messages"], payload["model"], max_tokens)
<!-- gh-comment-id:2558181757 -->
@nengoxx commented on GitHub (Dec 21, 2024):
Is it just for local inference requests with ollama perhaps?
Edit: I didn't have time to check the code this afternoon, but I guessed that was it, did a quick and dirty fix to hardcode it (in the main.py), not sure how usable this is but just in case you wanna check it, and also to have some future reference for myself.
```
import tiktoken
def count_tokens(messages, model):
encoding = tiktoken.get_encoding("gpt2")
#encoding = tiktoken.encoding_for_model(model) #extract external model name??
total_tokens = 0
for message in messages:
total_tokens += len(encoding.encode(message['role']))
total_tokens += len(encoding.encode(message['content']))
return total_tokens
def trim_context(messages, model, max_tokens):
#Leave the system role.
preserved_messages = [msg for msg in messages if msg['role'] not in ['user', 'assistant']]
trimmable_messages = [msg for msg in messages if msg['role'] in ['user', 'assistant']]
while count_tokens(preserved_messages + trimmable_messages, model) > max_tokens:
trimmable_messages.pop(0)
return preserved_messages + trimmable_messages
```
Then inside the /chat/completions route:
```
max_tokens = api_config.get("max_context_tokens", 8192) #Hardcoded max tokens.
if "messages" in payload:
payload["messages"] = trim_context(payload["messages"], payload["model"], max_tokens)
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @nengoxx on GitHub (Dec 21, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7991
Bug Report
Installation Method
pip
Environment
Open WebUI Version: 0.4.7
Operating System: Windows 11
Browser (if applicable): Firefox
Confirmation:
Expected Behavior:
Context shouldn't be longer than the number set in any of the options.
Actual Behavior:
It doesn't seem to trim the context regardless of where you set it (general settings, per model or per chat).
Description
Bug Summary:
Setting up a number doesn't trim the context, still sends the whole chat to the API.
Reproduction Details
Steps to Reproduce:
Set up a context length in Admin Console>Models, also in the Personal settings and in the chat settings.
Logs and Screenshots
Screenshots/Screen Recordings (if applicable):
Additional Information
This makes APIs like groq's unusable for longer chats.
@tjbck commented on GitHub (Dec 21, 2024):
Not what context length param is for.
@nengoxx commented on GitHub (Dec 21, 2024):
Is it just for local inference requests with ollama perhaps?
Edit: I didn't have time to check the code this afternoon, but I guessed that was it, did a quick and dirty fix to hardcode it (in the main.py), not sure how usable this is but just in case you wanna check it, and also to have some future reference for myself.
Then inside the /chat/completions route: