github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-13 11:33:32 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #12463] feat: support ktransformers minimal API #71217

New Issue
Closed
opened 2026-05-13 03:10:58 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-05-13 03:10:58 -05:00
Owner
Copy Link

Originally created by @createthis on GitHub (Apr 4, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12463

Check Existing Issues

  • I have searched the existing issues and discussions.

Problem Description

When I hit http://larry:11434/api/tags locally, while running ktransformers, I see:

{"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"123","size":123}]}

When I hit this same endpoint while running ollama, I see:

{"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","model":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"2025-03-30T01:43:13.778613781Z","size":404430201793,"digest":"1892be6c91be32f21bc6b75844b685efe25d4912895ac5f814e13334b2b104b8","details":{"parent_model":"","format":"gguf","family":"deepseek2","families":["deepseek2"],"parameter_size":"671.0B","quantization_level":"Q4_K_M"}}]}

It appears that Open WebUI is keying off the model attribute, so when I try to use Open WebUI with ktransformers, it fails with this error:

File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
2025-04-03 13:56:19     response = await f(request)
2025-04-03 13:56:19                ^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
2025-04-03 13:56:19     raw_response = await run_endpoint_function(
2025-04-03 13:56:19                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
2025-04-03 13:56:19     return await dependant.call(**values)
2025-04-03 13:56:19            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/app/backend/open_webui/main.py", line 950, in get_models
2025-04-03 13:56:19     models = await get_all_models(request, user=user)
2025-04-03 13:56:19              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/app/backend/open_webui/utils/models.py", line 63, in get_all_models
2025-04-03 13:56:19     models = await get_all_base_models(request, user=user)
2025-04-03 13:56:19              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/app/backend/open_webui/utils/models.py", line 43, in get_all_base_models
2025-04-03 13:56:19     ollama_models = await ollama.get_all_models(request, user=user)
2025-04-03 13:56:19                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 109, in wrapper
2025-04-03 13:56:19     return await self.decorator(f, *args, **kwargs)
2025-04-03 13:56:19            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 124, in decorator
2025-04-03 13:56:19     result = await f(*args, **kwargs)
2025-04-03 13:56:19              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/app/backend/open_webui/routers/ollama.py", line 369, in get_all_models
2025-04-03 13:56:19     "models": merge_models_lists(
2025-04-03 13:56:19               ^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19   File "/app/backend/open_webui/routers/ollama.py", line 359, in merge_models_lists
2025-04-03 13:56:19     id = model["model"]
2025-04-03 13:56:19          ~~~~~^^^^^^^^^
2025-04-03 13:56:19 KeyError: 'model'

NOTE: relates to https://github.com/open-webui/open-webui/discussions/10145

Desired Solution you'd like

I've hacked this locally with a new helper method (going from memory - I hacked it right in the docker terminal):

def get_model(model):
  return model.get('model') or model.get('name')

I had to touch two different files and I didn't take notes, sorry.

I'll try to get my local dev env up and running and issue a PR.

Alternatives Considered

I also opened this on the ktransformers side: https://github.com/kvcache-ai/ktransformers/issues/1039

Additional Context

Larry and I plan to take over the world, but he has to run on ktransformers to do it.

Originally created by @createthis on GitHub (Apr 4, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/12463 ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description When I hit `http://larry:11434/api/tags` locally, while running `ktransformers`, I see: ```json {"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"123","size":123}]} ``` When I hit this same endpoint while running `ollama`, I see: ```json {"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","model":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"2025-03-30T01:43:13.778613781Z","size":404430201793,"digest":"1892be6c91be32f21bc6b75844b685efe25d4912895ac5f814e13334b2b104b8","details":{"parent_model":"","format":"gguf","family":"deepseek2","families":["deepseek2"],"parameter_size":"671.0B","quantization_level":"Q4_K_M"}}]} ``` It appears that `Open WebUI` is keying off the `model` attribute, so when I try to use `Open WebUI` with `ktransformers`, it fails with this error: ``` File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app 2025-04-03 13:56:19 response = await f(request) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app 2025-04-03 13:56:19 raw_response = await run_endpoint_function( 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function 2025-04-03 13:56:19 return await dependant.call(**values) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/app/backend/open_webui/main.py", line 950, in get_models 2025-04-03 13:56:19 models = await get_all_models(request, user=user) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/app/backend/open_webui/utils/models.py", line 63, in get_all_models 2025-04-03 13:56:19 models = await get_all_base_models(request, user=user) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/app/backend/open_webui/utils/models.py", line 43, in get_all_base_models 2025-04-03 13:56:19 ollama_models = await ollama.get_all_models(request, user=user) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 109, in wrapper 2025-04-03 13:56:19 return await self.decorator(f, *args, **kwargs) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 124, in decorator 2025-04-03 13:56:19 result = await f(*args, **kwargs) 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/app/backend/open_webui/routers/ollama.py", line 369, in get_all_models 2025-04-03 13:56:19 "models": merge_models_lists( 2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^ 2025-04-03 13:56:19 File "/app/backend/open_webui/routers/ollama.py", line 359, in merge_models_lists 2025-04-03 13:56:19 id = model["model"] 2025-04-03 13:56:19 ~~~~~^^^^^^^^^ 2025-04-03 13:56:19 KeyError: 'model' ``` NOTE: relates to https://github.com/open-webui/open-webui/discussions/10145 ### Desired Solution you'd like I've hacked this locally with a new helper method (going from memory - I hacked it right in the docker terminal): ```python def get_model(model): return model.get('model') or model.get('name') ``` I had to touch two different files and I didn't take notes, sorry. I'll try to get my local dev env up and running and issue a PR. ### Alternatives Considered I also opened this on the `ktransformers` side: https://github.com/kvcache-ai/ktransformers/issues/1039 ### Additional Context Larry and I plan to take over the world, but he has to run on `ktransformers` to do it.
GiteaMirror commented 2026-05-13 03:10:59 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Apr 5, 2025):

You should use OpenAI API for this.

<!-- gh-comment-id:2780554325 --> @tjbck commented on GitHub (Apr 5, 2025): You should use OpenAI API for this.
GiteaMirror commented 2026-05-13 03:11:00 -05:00
Author
Owner
Copy Link

@createthis commented on GitHub (Apr 5, 2025):

You should use OpenAI API for this.

I went to Settings -> Admin Settings -> Connections -> +

Url: http://larry:11434/v1
Key: larry

Clicked Save.

I also deleted the ollama connection by clicking on the gear icon -> delete.

That worked! Thanks!

<!-- gh-comment-id:2780737029 --> @createthis commented on GitHub (Apr 5, 2025): > You should use OpenAI API for this. I went to `Settings` -> `Admin Settings` -> `Connections` -> `+` Url: `http://larry:11434/v1` Key: `larry` Clicked `Save`. I also deleted the ollama connection by clicking on the gear icon -> `delete`. That worked! Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#71217
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?