github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-05 18:38:17 -05:00
Code Issues 1.1k Packages Projects Releases 126 Wiki Activity

[GH-ISSUE #22192] issue: OTel collector's prometheus exporter logs "Instrument description conflict" #58321

New Issue
Closed
opened 2026-05-05 22:53:42 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-05 22:53:42 -05:00
Owner
Copy Link

Originally created by @erhhung on GitHub (Mar 3, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/22192

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Other

Open WebUI Version

v0.8.8

Ollama Version (if applicable)

v0.16.1

Operating System

Ubuntu 24.04

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When sending metrics to an OpenTelemetry collector that relays metrics to Prometheus, I expect that no errors are logged either by Open WebUI or the OpenTelemetry collector.

Actual Behavior

The errors shown below are logged by the OpenTelemetry collector's prometheus exporter every 30 seconds, interleaved with other debug output showing that metric datapoints were received.

Steps to Reproduce

  1. Enable Open WebUI OTel metrics by setting env vars:
  • ENABLE_OTEL: true
  • ENABLE_OTEL_METRICS: true
  • OTEL_EXPORTER_OTLP_ENDPOINT: https://otel-gateway-collector.observability.svc.cluster.local:4317
  1. Configure OTel collector (installed via OpenTelemetry Operator and OpenTelemetryCollector CR) with debug and prometheus exporters.
  2. Ensure there is no other client sending telemetry to the OTel collector.
  3. Check otel-gateway-collector pod logs and confirm output is quiet.
  4. Start Open WebUI and confirm in collector logs that metrics are being received.
  5. Wait a couple of minutes and confirm that "Instrument description conflict" errors are being logged every 30 seconds.

Logs & Screenshots

2026-03-03T19:00:05.295Z    info    prometheusexporter@v0.147.0/collector.go:656
    Instrument description conflict, using existing
    {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6",
     "service.name": "otelcol-contrib", "service.version": "0.147.0"},
     "otelcol.component.id": "prometheus",
     "otelcol.component.kind": "exporter",
     "otelcol.signal": "metrics",
     "instrument": "http_server_duration_milliseconds",
     "existing": "HTTP request duration",
     "dropped": "Measures the duration of inbound HTTP requests."}
2026-03-03T19:00:35.295Z    info    prometheusexporter@v0.147.0/collector.go:656    Instrument description conflict, using existing    {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6", "service.name": "otelcol-contrib", "service.version": "0.147.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "instrument": "http_server_duration_milliseconds", "existing": "HTTP request duration", "dropped": "Measures the duration of inbound HTTP requests."}
2026-03-03T19:01:05.295Z    info    prometheusexporter@v0.147.0/collector.go:656    Instrument description conflict, using existing    {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6", "service.name": "otelcol-contrib", "service.version": "0.147.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "instrument": "http_server_duration_milliseconds", "existing": "HTTP request duration", "dropped": "Measures the duration of inbound HTTP requests."}

Additional Information

From the Open WebUI container:

open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \
  "Measures the duration of inbound HTTP requests." \
  /usr/local/lib/python3.11/site-packages/

/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py:
                description="Measures the duration of inbound HTTP requests.",

open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \
  "Measures the duration of inbound HTTP requests." .

# nothing found

open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \
  "HTTP request duration" .

./utils/telemetry/metrics.py:
        description="HTTP request duration",

open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \
  "HTTP request duration" /usr/local/lib/python3.11/site-packages/

# nothing found

The above grep output shows that instrumentation of http_server_duration_milliseconds via opentelemetry.instrumentation.asgi emits "Measures the duration of inbound HTTP requests." as the metric description, while Open WebUI uses "HTTP request duration" as the description.

It appears that multiple series of the http_server_duration_milliseconds metric are being emitted by Open WebUI (see otel_scope_name values "opentelemetry.instrumentation.fastapi" and "open_webui.utils.telemetry.metrics" below), but with inconsistent descriptions:

http_server_duration_milliseconds_sum{container="otc-container", endpoint="prometheus",
exported_job="open-webui", http_method="GET", http_status_code="200", instance="10.42.2.179:8889",
job="otel-gateway-collector", namespace="observability",
otel_scope_name="opentelemetry.instrumentation.fastapi",
otel_scope_schema_url="https://opentelemetry.io/schemas/1.11.0", otel_scope_version="0.60b1",
pod="otel-gateway-collector-5d7c668f85-dg7l5", service="otel-gateway-collector"}

http_server_duration_milliseconds_sum{container="otc-container", endpoint="prometheus",
exported_job="open-webui", http_method="GET", http_route="/health/db", http_status_code="200",
instance="10.42.2.179:8889", job="otel-gateway-collector", namespace="observability",
otel_scope_name="open_webui.utils.telemetry.metrics",
pod="otel-gateway-collector-5d7c668f85-dg7l5", service="otel-gateway-collector"}
Originally created by @erhhung on GitHub (Mar 3, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/22192 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Other ### Open WebUI Version v0.8.8 ### Ollama Version (if applicable) v0.16.1 ### Operating System Ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When sending metrics to an OpenTelemetry collector that relays metrics to Prometheus, I expect that no errors are logged either by Open WebUI or the OpenTelemetry collector. ### Actual Behavior The errors shown below are logged by the OpenTelemetry collector's `prometheus` exporter **every 30 seconds**, interleaved with other `debug` output showing that metric datapoints were received. ### Steps to Reproduce 1. Enable Open WebUI OTel metrics by setting env vars: - `ENABLE_OTEL: true` - `ENABLE_OTEL_METRICS: true` - `OTEL_EXPORTER_OTLP_ENDPOINT: https://otel-gateway-collector.observability.svc.cluster.local:4317` 2. Configure OTel collector (installed via OpenTelemetry Operator and `OpenTelemetryCollector` CR) with `debug` and **`prometheus`** exporters. 3. Ensure there is no other client sending telemetry to the OTel collector. 4. Check `otel-gateway-collector` pod logs and confirm output is _quiet_. 5. Start Open WebUI and confirm in collector logs that metrics are being received. 6. Wait a couple of minutes and confirm that "**Instrument description conflict**" errors are being logged every 30 seconds. ### Logs & Screenshots ``` 2026-03-03T19:00:05.295Z info prometheusexporter@v0.147.0/collector.go:656 Instrument description conflict, using existing {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6", "service.name": "otelcol-contrib", "service.version": "0.147.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "instrument": "http_server_duration_milliseconds", "existing": "HTTP request duration", "dropped": "Measures the duration of inbound HTTP requests."} 2026-03-03T19:00:35.295Z info prometheusexporter@v0.147.0/collector.go:656 Instrument description conflict, using existing {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6", "service.name": "otelcol-contrib", "service.version": "0.147.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "instrument": "http_server_duration_milliseconds", "existing": "HTTP request duration", "dropped": "Measures the duration of inbound HTTP requests."} 2026-03-03T19:01:05.295Z info prometheusexporter@v0.147.0/collector.go:656 Instrument description conflict, using existing {"resource": {"service.instance.id": "b81ab987-b5c3-40e6-a925-393b4da0f9b6", "service.name": "otelcol-contrib", "service.version": "0.147.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "instrument": "http_server_duration_milliseconds", "existing": "HTTP request duration", "dropped": "Measures the duration of inbound HTTP requests."} ``` ### Additional Information From the Open WebUI container: ```bash open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \ "Measures the duration of inbound HTTP requests." \ /usr/local/lib/python3.11/site-packages/ /usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py: description="Measures the duration of inbound HTTP requests.", open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \ "Measures the duration of inbound HTTP requests." . # nothing found open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \ "HTTP request duration" . ./utils/telemetry/metrics.py: description="HTTP request duration", open-webui-7b8c4b4cc4-rp665:/app/backend/open_webui$ grep -r \ "HTTP request duration" /usr/local/lib/python3.11/site-packages/ # nothing found ``` The above `grep` output shows that instrumentation of `http_server_duration_milliseconds` via `opentelemetry.instrumentation.asgi` emits "Measures the duration of inbound HTTP requests." as the metric description, while Open WebUI uses "HTTP request duration" as the description. It appears that multiple series of the `http_server_duration_milliseconds` metric are being emitted by Open WebUI (see `otel_scope_name` values "opentelemetry.instrumentation.fastapi" and "open_webui.utils.telemetry.metrics" below), but with inconsistent descriptions: ```promql http_server_duration_milliseconds_sum{container="otc-container", endpoint="prometheus", exported_job="open-webui", http_method="GET", http_status_code="200", instance="10.42.2.179:8889", job="otel-gateway-collector", namespace="observability", otel_scope_name="opentelemetry.instrumentation.fastapi", otel_scope_schema_url="https://opentelemetry.io/schemas/1.11.0", otel_scope_version="0.60b1", pod="otel-gateway-collector-5d7c668f85-dg7l5", service="otel-gateway-collector"} http_server_duration_milliseconds_sum{container="otc-container", endpoint="prometheus", exported_job="open-webui", http_method="GET", http_route="/health/db", http_status_code="200", instance="10.42.2.179:8889", job="otel-gateway-collector", namespace="observability", otel_scope_name="open_webui.utils.telemetry.metrics", pod="otel-gateway-collector-5d7c668f85-dg7l5", service="otel-gateway-collector"} ```
GiteaMirror added the bug label 2026-05-05 22:53:42 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#58321
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?