issue: OpenTelemetry not exporting traces on any image > 0.6.22 #6470

New Issue

2025-11-11T16:56:14-06:00

GiteaMirror commented

2025-11-11 16:56:14 -06:00

Originally created by @damienpontifex on GitHub (Sep 23, 2025).

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.23+

Ollama Version (if applicable)

No response

Operating System

macOS, Linux, Docker

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

Use some combination of Otel environment variables. In this case, I've got:

ENABLE_OTEL: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: http://otlp:4317
OTEL_EXPORTER_OTLP_PROTOCOL: grpc
OTEL_EXPORTER_OTLP_INSECURE: true
ENABLE_OTEL_METRICS: false
ENABLE_OTEL_LOGS: true

And should receive OpenTelemetry traces through to the OTLP receiver/backend

Actual Behavior

For any image > 0.6.22 no traces are appearing in OpenTelemetry backend/receiver

I've iterated through all of the image tags from 0.6.23 through to 0.6.30 with the same result

Steps to Reproduce

Use the following docker compose (some local env variable substitution)

services:
  otlp:
    image: mcr.microsoft.com/dotnet/aspire-dashboard:latest
    container_name: telemetry
    ports:
      - 18888:18888 # Aspire Dashboard UI
    expose:
      - 4317 # OTLP/gRPC
      - 4318 # OTLP/HTTP
    restart: unless-stopped
    environment:
      - DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true
      - DOTNET_DASHBOARD_OTLP_ENDPOINT_URL=http://[::]:4317
      - DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URL=http://[::]:4318
  open_webui:
    image: ghcr.io/open-webui/open-webui:0.6.22
    container_name: open-webui
    ports:
      - 8080:8080
    environment:
      GLOBAL_LOG_LEVEL: DEBUG
      ENV: dev
      OFFLINE_MODE: True
      ENABLE_EVALUATION_ARENA_MODELS: False
      ENABLE_OAUTH_SIGNUP: True
      ENABLE_LOGIN_FORM: True
      ENABLE_FORWARD_USER_INFO_HEADERS: "true"
      MICROSOFT_CLIENT_ID: ${MICROSOFT_CLIENT_ID}
      MICROSOFT_CLIENT_SECRET: ${MICROSOFT_CLIENT_SECRET}
      MICROSOFT_CLIENT_TENANT_ID: ${MICROSOFT_CLIENT_TENANT_ID}
      OPENAI_API_BASE_URL: ${OPENAI_API_BASE_URL:-http://litellm:4000}
      OPENAI_API_KEY: ${LITELLM_MASTER_KEY}
      ENABLE_OLLAMA_API: False
      DATABASE_URL: postgres://open_webui:dbpassword9090@open_webui_db:5432/open_webui?sslmode=disable
      ENABLE_OTEL: "true"
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otlp:4317
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_EXPORTER_OTLP_INSECURE: true
      ENABLE_OTEL_METRICS: false
      ENABLE_OTEL_LOGS: true
      VECTOR_DB: pgvector
      ENABLE_PERSISTENT_CONFIG: False
      REDIS_URL: redis://redis:6379/0
      WEBSOCKET_REDIS_URL: redis://redis:6379/0
    depends_on:
      open_webui_db:
        condition: service_healthy
      redis:
        condition: service_started
    volumes:
      - open-webui:/app/backend/data
    restart: no
    healthcheck:
      test:
        - CMD-SHELL
        - python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')" || exit 1
      interval: 10s
      timeout: 10s
      retries: 3
      start_period: 20s
  open_webui_db:
    image: pgvector/pgvector:pg17
    container_name: open_webui_db
    restart: always
    environment:
      POSTGRES_USER: open_webui
      POSTGRES_PASSWORD: dbpassword9090
    volumes:
      - open_webui_postgres_data:/var/lib/postgresql/data
      - ./enable-vector.sql:/docker-entrypoint-initdb.d/enable-vector.sql
    healthcheck:
      test:
        - CMD-SHELL
        - pg_isready -d open_webui -U open_webui
      interval: 1s
      timeout: 5s
      retries: 10
  redis:
    image: redis:8
    restart: always
volumes:
  open-webui:
  open_webui_postgres_data:

And notice traces show up in Aspire dashboard.

Update the image to 0.6.23 or anything newer and notice no traces show up

Logs & Screenshots

There were logs, but nothing related to OTLPL etc, so didn't put them in here to avoid noise...the logs for the requests were visible i.e. GET /static/... etc

Let me know if you'd like those logs, but thought it wouldn't add value as no notice/errors related to OTLP/OpenTelemetry that I could see

Additional Information

No response

Originally created by @damienpontifex on GitHub (Sep 23, 2025). ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.23+ ### Ollama Version (if applicable) _No response_ ### Operating System macOS, Linux, Docker ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior Use some combination of Otel environment variables. In this case, I've got: ```yaml ENABLE_OTEL: "true" OTEL_EXPORTER_OTLP_ENDPOINT: http://otlp:4317 OTEL_EXPORTER_OTLP_PROTOCOL: grpc OTEL_EXPORTER_OTLP_INSECURE: true ENABLE_OTEL_METRICS: false ENABLE_OTEL_LOGS: true ``` And should receive OpenTelemetry traces through to the OTLP receiver/backend ### Actual Behavior For any image > 0.6.22 no traces are appearing in OpenTelemetry backend/receiver I've iterated through all of the image tags from 0.6.23 through to 0.6.30 with the same result ### Steps to Reproduce Use the following docker compose (some local env variable substitution) ```yaml services: otlp: image: mcr.microsoft.com/dotnet/aspire-dashboard:latest container_name: telemetry ports: - 18888:18888 # Aspire Dashboard UI expose: - 4317 # OTLP/gRPC - 4318 # OTLP/HTTP restart: unless-stopped environment: - DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true - DOTNET_DASHBOARD_OTLP_ENDPOINT_URL=http://[::]:4317 - DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URL=http://[::]:4318 open_webui: image: ghcr.io/open-webui/open-webui:0.6.22 container_name: open-webui ports: - 8080:8080 environment: GLOBAL_LOG_LEVEL: DEBUG ENV: dev OFFLINE_MODE: True ENABLE_EVALUATION_ARENA_MODELS: False ENABLE_OAUTH_SIGNUP: True ENABLE_LOGIN_FORM: True ENABLE_FORWARD_USER_INFO_HEADERS: "true" MICROSOFT_CLIENT_ID: ${MICROSOFT_CLIENT_ID} MICROSOFT_CLIENT_SECRET: ${MICROSOFT_CLIENT_SECRET} MICROSOFT_CLIENT_TENANT_ID: ${MICROSOFT_CLIENT_TENANT_ID} OPENAI_API_BASE_URL: ${OPENAI_API_BASE_URL:-http://litellm:4000} OPENAI_API_KEY: ${LITELLM_MASTER_KEY} ENABLE_OLLAMA_API: False DATABASE_URL: postgres://open_webui:dbpassword9090@open_webui_db:5432/open_webui?sslmode=disable ENABLE_OTEL: "true" OTEL_EXPORTER_OTLP_ENDPOINT: http://otlp:4317 OTEL_EXPORTER_OTLP_PROTOCOL: grpc OTEL_EXPORTER_OTLP_INSECURE: true ENABLE_OTEL_METRICS: false ENABLE_OTEL_LOGS: true VECTOR_DB: pgvector ENABLE_PERSISTENT_CONFIG: False REDIS_URL: redis://redis:6379/0 WEBSOCKET_REDIS_URL: redis://redis:6379/0 depends_on: open_webui_db: condition: service_healthy redis: condition: service_started volumes: - open-webui:/app/backend/data restart: no healthcheck: test: - CMD-SHELL - python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')" || exit 1 interval: 10s timeout: 10s retries: 3 start_period: 20s open_webui_db: image: pgvector/pgvector:pg17 container_name: open_webui_db restart: always environment: POSTGRES_USER: open_webui POSTGRES_PASSWORD: dbpassword9090 volumes: - open_webui_postgres_data:/var/lib/postgresql/data - ./enable-vector.sql:/docker-entrypoint-initdb.d/enable-vector.sql healthcheck: test: - CMD-SHELL - pg_isready -d open_webui -U open_webui interval: 1s timeout: 5s retries: 10 redis: image: redis:8 restart: always volumes: open-webui: open_webui_postgres_data: ``` And notice traces show up in Aspire dashboard. Update the image to 0.6.23 or anything newer and notice no traces show up ### Logs & Screenshots There were logs, but nothing related to OTLPL etc, so didn't put them in here to avoid noise...the logs for the requests were visible i.e. `GET /static/...` etc Let me know if you'd like those logs, but thought it wouldn't add value as no notice/errors related to OTLP/OpenTelemetry that I could see ### Additional Information _No response_

GiteaMirror added the bug label 2025-11-11 16:56:14 -06:00

GiteaMirror closed this issue

2025-11-11 16:56:14 -06:00

GiteaMirror commented

2025-11-11 16:56:15 -06:00

@damienpontifex commented on GitHub (Sep 23, 2025):

Oh, I can see https://github.com/open-webui/open-webui/compare/v0.6.22...v0.6.23#diff-0c34b59d563890dde1f4990656c9021d9356781c6604043bd452c18cc657155d there was a configuration/environment variable introduced of ENABLE_OTEL_TRACES but isn't documented https://docs.openwebui.com/getting-started/env-configuration/

Tested on 0.6.30 and setting ENABLE_OTEL_TRACES: "true" for environment variable did resolve the issue.

So I think this bug report becomes a request for documentation update

@damienpontifex commented on GitHub (Sep 23, 2025): Oh, I can see https://github.com/open-webui/open-webui/compare/v0.6.22...v0.6.23#diff-0c34b59d563890dde1f4990656c9021d9356781c6604043bd452c18cc657155d there was a configuration/environment variable introduced of `ENABLE_OTEL_TRACES` but isn't documented https://docs.openwebui.com/getting-started/env-configuration/ Tested on 0.6.30 and setting `ENABLE_OTEL_TRACES: "true"` for environment variable did resolve the issue. So I think this bug report becomes a request for documentation update

GiteaMirror commented

2025-11-11 16:56:15 -06:00