[GH-ISSUE #23139] issue: analytics uses 'input' and 'output' tokens but api return 'prompt' and 'completion' #123206

New Issue

GiteaMirror · 2026-05-21T02:28:41-05:00

GiteaMirror commented

2026-05-21 02:28:41 -05:00

Originally created by @Art-1313 on GitHub (Mar 27, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23139

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
I am using the latest version of Open WebUI.

Installation Method

Pip Install

Open WebUI Version

v0.8.12

Ollama Version (if applicable)

No response

Operating System

macos

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When token count occurs in analytics prompt and completion token should be counted

Actual Behavior

Code is trying to count input and output tokens

Steps to Reproduce

try to get analytics and compare it with database

Logs & Screenshots

in open_webui.models.chat_messages

`

 def get_token_usage_by_user(
    self,
    start_date: Optional[int] = None,
    end_date: Optional[int] = None,
    group_id: Optional[str] = None,
    db: Optional[Session] = None,
) -> dict[str, dict]:
    """Aggregate token usage by user using database-level aggregation."""
    with get_db_context(db) as db:
        from sqlalchemy import func, cast, Integer
        from open_webui.models.groups import GroupMember
        dialect = db.bind.dialect.name

        if dialect == "sqlite":
            input_tokens = cast(
                func.json_extract(ChatMessage.usage, "$.input_tokens"), Integer
            )
            output_tokens = cast(
                func.json_extract(ChatMessage.usage, "$.output_tokens"), Integer
            )
        elif dialect == "postgresql":
            # Use json_extract_path_text for PostgreSQL JSON columns
            input_tokens = cast(
                func.json_extract_path_text(ChatMessage.usage, "input_tokens"),
                Integer,
            )
            output_tokens = cast(
                func.json_extract_path_text(ChatMessage.usage, "output_tokens"),
                Integer,
            )
        else:
            raise NotImplementedError(f"Unsupported dialect: {dialect}")

        query = db.query(
            ChatMessage.user_id,
            func.coalesce(func.sum(input_tokens), 0).label("input_tokens"),
            func.coalesce(func.sum(output_tokens), 0).label("output_tokens"),
            func.count(ChatMessage.id).label("message_count"),
        ).filter(
            ChatMessage.role == "assistant",
            ChatMessage.user_id.isnot(None),
            ChatMessage.usage.isnot(None),
            ~ChatMessage.user_id.like("shared-%"),
        )

        if start_date:
            query = query.filter(ChatMessage.created_at >= start_date)
        if end_date:
            query = query.filter(ChatMessage.created_at <= end_date)
        if group_id:
            group_users = (
                db.query(GroupMember.user_id)
                .filter(GroupMember.group_id == group_id)
                .subquery()
            )
            query = query.filter(ChatMessage.user_id.in_(group_users))

        results = query.group_by(ChatMessage.user_id).all()

        return {
            row.user_id: {
                "input_tokens": row.input_tokens,
                "output_tokens": row.output_tokens,
                "total_tokens": row.input_tokens + row.output_tokens,
                "message_count": row.message_count,
            }
            for row in results
        }

`

Additional Information

No response

Originally created by @Art-1313 on GitHub (Mar 27, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23139 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Pip Install ### Open WebUI Version v0.8.12 ### Ollama Version (if applicable) _No response_ ### Operating System macos ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When token count occurs in analytics prompt and completion token should be counted ### Actual Behavior Code is trying to count input and output tokens ### Steps to Reproduce try to get analytics and compare it with database ### Logs & Screenshots in open_webui.models.chat_messages ` def get_token_usage_by_user( self, start_date: Optional[int] = None, end_date: Optional[int] = None, group_id: Optional[str] = None, db: Optional[Session] = None, ) -> dict[str, dict]: """Aggregate token usage by user using database-level aggregation.""" with get_db_context(db) as db: from sqlalchemy import func, cast, Integer from open_webui.models.groups import GroupMember dialect = db.bind.dialect.name if dialect == "sqlite": input_tokens = cast( func.json_extract(ChatMessage.usage, "$.input_tokens"), Integer ) output_tokens = cast( func.json_extract(ChatMessage.usage, "$.output_tokens"), Integer ) elif dialect == "postgresql": # Use json_extract_path_text for PostgreSQL JSON columns input_tokens = cast( func.json_extract_path_text(ChatMessage.usage, "input_tokens"), Integer, ) output_tokens = cast( func.json_extract_path_text(ChatMessage.usage, "output_tokens"), Integer, ) else: raise NotImplementedError(f"Unsupported dialect: {dialect}") query = db.query( ChatMessage.user_id, func.coalesce(func.sum(input_tokens), 0).label("input_tokens"), func.coalesce(func.sum(output_tokens), 0).label("output_tokens"), func.count(ChatMessage.id).label("message_count"), ).filter( ChatMessage.role == "assistant", ChatMessage.user_id.isnot(None), ChatMessage.usage.isnot(None), ~ChatMessage.user_id.like("shared-%"), ) if start_date: query = query.filter(ChatMessage.created_at >= start_date) if end_date: query = query.filter(ChatMessage.created_at <= end_date) if group_id: group_users = ( db.query(GroupMember.user_id) .filter(GroupMember.group_id == group_id) .subquery() ) query = query.filter(ChatMessage.user_id.in_(group_users)) results = query.group_by(ChatMessage.user_id).all() return { row.user_id: { "input_tokens": row.input_tokens, "output_tokens": row.output_tokens, "total_tokens": row.input_tokens + row.output_tokens, "message_count": row.message_count, } for row in results } ` ### Additional Information _No response_

GiteaMirror added the bug label 2026-05-21 02:28:41 -05:00

GiteaMirror closed this issue

2026-05-21 02:28:43 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#123206