I am trying to build a "token counter" filter for Open WebUI; it works fine, adding the "Tokens" at the end of each System output but removing them from the messages again before sending the message to the LLM.
The issue is: The token count is incorrect, because the "body" object does not seem to contain the system message:
Is your feature request related to a problem? Please describe.
This is particularly important when using RAG, as the context is added into the system message and is therefore not counted by my "token counter" filter.
Describe the solution you'd like
I believe it would be beneficial to also have acces to the system message with the filter, i.e. add the system message to the chain of messages.
Describe alternatives you've considered
Perhaps we should just add a "Token counter" for openAI models as is already present for Ollama models?
Originally created by @clang88 on GitHub (Nov 13, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6925
# Feature Request
I am trying to build a "token counter" filter for Open WebUI; it works fine, adding the "Tokens" at the end of each System output but removing them from the messages again before sending the message to the LLM.
The issue is: The token count is incorrect, because the "body" object does not seem to contain the system message:
---
**Is your feature request related to a problem? Please describe.**
This is particularly important when using RAG, as the context is added into the system message and is therefore not counted by my "token counter" filter.
**Describe the solution you'd like**
I believe it would be beneficial to also have acces to the system message with the filter, i.e. add the system message to the chain of messages.
**Describe alternatives you've considered**
Perhaps we should just add a "Token counter" for openAI models as is already present for Ollama models?
<!-- gh-comment-id:2474994327 -->
@clang88 commented on GitHub (Nov 13, 2024):
This is the code for the filter, btw. I commented out the trick I used to see the body contents.
```python
"""
title: Token Counter
author: clang88
version: 1.0
"""
from pydantic import BaseModel, Field
from typing import Optional
import tiktoken
import json
import os
class Filter:
class Valves(BaseModel):
priority: int = Field(
default=0, description="Priority level for the filter operations."
)
pass
def __init__(self):
self.valves = self.Valves()
self.input_cost = 0
self.output_cost = 0
pass
def inlet(self, body: dict, __user__: Optional[dict] = None) -> dict:
if __user__.get("role", "admin") in ["user", "admin", "system"]:
messages = body.get("messages", [])
model = body.get("model")
try:
enc = tiktoken.encoding_for_model(model)
except Exception as e:
if "gpt-3.5-turbo" in model:
enc = tiktoken.encoding_for_model("gpt-3.5-turbo-")
else:
print(f"Error: {e}")
enc = tiktoken.encoding_for_model("gpt-4o")
for message in messages:
if "\n\n**Token consumption:**" in message["content"]:
message["content"] = "".join(
message["content"].split("\n\n**Token consumption:**")
)[:-1]
return body
def outlet(self, body: dict, __user__: Optional[dict] = None) -> dict:
messages = body.get("messages", [])
model = body.get("model")
try:
enc = tiktoken.encoding_for_model(model)
except Exception as e:
if "gpt-3.5-turbo" in model:
enc = tiktoken.encoding_for_model("gpt-3.5-turbo-")
else:
print(f"Error: {e}")
enc = tiktoken.encoding_for_model("gpt-4o")
total_tokens = sum(len(enc.encode(message["content"])) for message in messages)
# messages[-1]["content"] = str(body)
messages[-1]["content"] = str(
messages[-1]["content"] + "\n\n**Token consumption:** " + str(total_tokens)
)
return body
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @clang88 on GitHub (Nov 13, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6925
Feature Request
I am trying to build a "token counter" filter for Open WebUI; it works fine, adding the "Tokens" at the end of each System output but removing them from the messages again before sending the message to the LLM.
The issue is: The token count is incorrect, because the "body" object does not seem to contain the system message:
Is your feature request related to a problem? Please describe.
This is particularly important when using RAG, as the context is added into the system message and is therefore not counted by my "token counter" filter.
Describe the solution you'd like
I believe it would be beneficial to also have acces to the system message with the filter, i.e. add the system message to the chain of messages.
Describe alternatives you've considered
Perhaps we should just add a "Token counter" for openAI models as is already present for Ollama models?
@clang88 commented on GitHub (Nov 13, 2024):
This is the code for the filter, btw. I commented out the trick I used to see the body contents.