[PR #18244] [CLOSED] fix: Deduplicate model entries when multiple endpoints return the same model ID #24721

New Issue

GiteaMirror · 2026-04-20T05:33:47-05:00

GiteaMirror commented

2026-04-20 05:33:47 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/18244
Author: @kindredzhang
Created: 10/11/2025
Status: ❌ Closed

Base: main ← Head: fix-duplicated-model-ids

📝 Commits (1)

6b5e14c fix: add deduplication for model IDs to prevent duplicate models in list

📊 Changes

1 file changed (+27 additions, -24 deletions)

View changed files

📝 backend/open_webui/routers/openai.py (+27 -24)

📄 Description

Summary

Deduplicate model entries when multiple endpoints return the same model ID to prevent frontend crashes and improve user experience for proxy gateway users.

Motivation / Use-Case

Most users of openai-python do not call api.openai.com directly; they route through unified gateways (One-API, New-API, Cloudflare-AI, etc.). These gateways aggregate dozens of providers into a single /v1/models response, so identical IDs (e.g., gpt-3.5-turbo, babbage-002) appear many times under different indexes.

The frontend settings page uses a Svelte each block keyed by ID; duplicate keys trigger:

Uncaught (in promise) Error: Cannot have duplicate keys in a keyed each: ...

Once thrown, the route crashes and the UI becomes unresponsive—users must force-refresh the browser and have no way to configure models manually.

This server-side deduplication lets "proxy users" consume the list out-of-the-box without any breaking changes or manual allow-lists.

Changes

Add seen_ids: set[str] inside merge_models_lists()
Skip models whose ID (or name) was already emitted
Maintain existing filtering logic unchanged
Use O(1) set lookup for efficient deduplication

Testing

Two mock endpoints returning overlapping models → no duplicates in payload
Frontend settings page renders without crashing
pytest tests/ passes locally
No breaking changes to existing functionality

Screenshots

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Checklist

My code follows the project's style guidelines
I have commented my code, particularly in hard-to-understand areas
I have performed a self-review of my own code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/18244 **Author:** [@kindredzhang](https://github.com/kindredzhang) **Created:** 10/11/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix-duplicated-model-ids` --- ### 📝 Commits (1) - [`6b5e14c`](https://github.com/open-webui/open-webui/commit/6b5e14cc2efc1c22dc958652bfed2a078efed398) fix: add deduplication for model IDs to prevent duplicate models in list ### 📊 Changes **1 file changed** (+27 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/routers/openai.py` (+27 -24) </details> ### 📄 Description ## Summary Deduplicate model entries when multiple endpoints return the same model ID to prevent frontend crashes and improve user experience for proxy gateway users. ## Motivation / Use-Case Most users of openai-python do not call api.openai.com directly; they route through unified gateways (One-API, New-API, Cloudflare-AI, etc.). These gateways aggregate dozens of providers into a single `/v1/models` response, so identical IDs (e.g., `gpt-3.5-turbo`, `babbage-002`) appear many times under different indexes. The frontend settings page uses a Svelte each block keyed by ID; duplicate keys trigger: ``` Uncaught (in promise) Error: Cannot have duplicate keys in a keyed each: ... ``` Once thrown, the route crashes and the UI becomes unresponsive—users must force-refresh the browser and have no way to configure models manually. This server-side deduplication lets "proxy users" consume the list out-of-the-box without any breaking changes or manual allow-lists. ## Changes - Add `seen_ids: set[str]` inside `merge_models_lists()` - Skip models whose ID (or name) was already emitted - Maintain existing filtering logic unchanged - Use O(1) set lookup for efficient deduplication ## Testing - [x] Two mock endpoints returning overlapping models → no duplicates in payload - [x] Frontend settings page renders without crashing - [x] `pytest tests/` passes locally - [x] No breaking changes to existing functionality ## Screenshots ![Frontend crash before fix](https://github.com/user-attachments/assets/08c72001-4cb7-4218-8623-2590d7648622) ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update ## Checklist - [x] My code follows the project's style guidelines - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-20 05:33:47 -05:00

GiteaMirror closed this issue

2026-04-20 05:33:47 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#24721