[PR #18245] [CLOSED] fix: Deduplicate model entries when multiple endpoints return the same model ID #63578

New Issue

GiteaMirror · 2026-05-06T08:27:07-05:00

GiteaMirror commented

2026-05-06 08:27:07 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/18245
Author: @kindredzhang
Created: 10/11/2025
Status: ❌ Closed

Base: main ← Head: fix-duplicated-model-ids

📝 Commits (1)

6b5e14c fix: add deduplication for model IDs to prevent duplicate models in list

📊 Changes

1 file changed (+27 additions, -24 deletions)

View changed files

📝 backend/open_webui/routers/openai.py (+27 -24)

📄 Description

Fix: Deduplicate model entries when multiple endpoints return the same model ID

Summary

Deduplicate model entries when multiple endpoints return the same model ID to prevent frontend crashes and improve user experience for proxy gateway users.

Motivation / Use-Case

Most users of openai-python do not call api.openai.com directly; they route through unified gateways (One-API, New-API, Cloudflare-AI, etc.). These gateways aggregate dozens of providers into a single /v1/models response, so identical IDs (e.g., gpt-3.5-turbo, babbage-002) appear many times under different indexes.

The frontend settings page uses a Svelte each block keyed by ID; duplicate keys trigger:

Uncaught (in promise) Error: Cannot have duplicate keys in a keyed each: ...

Once thrown, the route crashes and the UI becomes unresponsive—users must force-refresh the browser and have no way to configure models manually.

This server-side deduplication lets "proxy users" consume the list out-of-the-box without any breaking changes or manual allow-lists.

Changes

Add seen_ids: set[str] inside merge_models_lists()
Skip models whose ID (or name) was already emitted
Maintain existing filtering logic unchanged
Use O(1) set lookup for efficient deduplication

Testing

Two mock endpoints returning overlapping models → no duplicates in payload
Frontend settings page renders without crashing
pytest tests/ passes locally
No breaking changes to existing functionality

Screenshots

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Checklist

My code follows the project's style guidelines
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Contributor License Agreement

By submitting this pull request, I agree that my contributions are licensed under the same terms as the project itself. I understand that:

I am granting permission to use my contributions under the project's license
I have the right to submit these contributions
My contributions do not violate any third-party rights
I understand that this project may use my contributions in any way consistent with the project's license

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/18245 **Author:** [@kindredzhang](https://github.com/kindredzhang) **Created:** 10/11/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix-duplicated-model-ids` --- ### 📝 Commits (1) - [`6b5e14c`](https://github.com/open-webui/open-webui/commit/6b5e14cc2efc1c22dc958652bfed2a078efed398) fix: add deduplication for model IDs to prevent duplicate models in list ### 📊 Changes **1 file changed** (+27 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/routers/openai.py` (+27 -24) </details> ### 📄 Description # Fix: Deduplicate model entries when multiple endpoints return the same model ID ## Summary Deduplicate model entries when multiple endpoints return the same model ID to prevent frontend crashes and improve user experience for proxy gateway users. ## Motivation / Use-Case Most users of openai-python do not call api.openai.com directly; they route through unified gateways (One-API, New-API, Cloudflare-AI, etc.). These gateways aggregate dozens of providers into a single `/v1/models` response, so identical IDs (e.g., `gpt-3.5-turbo`, `babbage-002`) appear many times under different indexes. The frontend settings page uses a Svelte each block keyed by ID; duplicate keys trigger: ``` Uncaught (in promise) Error: Cannot have duplicate keys in a keyed each: ... ``` Once thrown, the route crashes and the UI becomes unresponsive—users must force-refresh the browser and have no way to configure models manually. This server-side deduplication lets "proxy users" consume the list out-of-the-box without any breaking changes or manual allow-lists. ## Changes - Add `seen_ids: set[str]` inside `merge_models_lists()` - Skip models whose ID (or name) was already emitted - Maintain existing filtering logic unchanged - Use O(1) set lookup for efficient deduplication ## Testing - [x] Two mock endpoints returning overlapping models → no duplicates in payload - [x] Frontend settings page renders without crashing - [x] `pytest tests/` passes locally - [x] No breaking changes to existing functionality ## Screenshots ![Frontend crash before fix](https://github.com/user-attachments/assets/08c72001-4cb7-4218-8623-2590d7648622) ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Contributor License Agreement By submitting this pull request, I agree that my contributions are licensed under the same terms as the project itself. I understand that: 1. I am granting permission to use my contributions under the project's license 2. I have the right to submit these contributions 3. My contributions do not violate any third-party rights 4. I understand that this project may use my contributions in any way consistent with the project's license --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-06 08:27:07 -05:00

GiteaMirror closed this issue

2026-05-06 08:27:11 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#63578