Larry and I plan to take over the world, but he has to run on ktransformers to do it.
Originally created by @createthis on GitHub (Apr 4, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12463
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
### Problem Description
When I hit `http://larry:11434/api/tags` locally, while running `ktransformers`, I see:
```json
{"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"123","size":123}]}
```
When I hit this same endpoint while running `ollama`, I see:
```json
{"models":[{"name":"DeepSeek-V3-0324:671b-q4_k_m","model":"DeepSeek-V3-0324:671b-q4_k_m","modified_at":"2025-03-30T01:43:13.778613781Z","size":404430201793,"digest":"1892be6c91be32f21bc6b75844b685efe25d4912895ac5f814e13334b2b104b8","details":{"parent_model":"","format":"gguf","family":"deepseek2","families":["deepseek2"],"parameter_size":"671.0B","quantization_level":"Q4_K_M"}}]}
```
It appears that `Open WebUI` is keying off the `model` attribute, so when I try to use `Open WebUI` with `ktransformers`, it fails with this error:
```
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
2025-04-03 13:56:19 response = await f(request)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
2025-04-03 13:56:19 raw_response = await run_endpoint_function(
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
2025-04-03 13:56:19 return await dependant.call(**values)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/app/backend/open_webui/main.py", line 950, in get_models
2025-04-03 13:56:19 models = await get_all_models(request, user=user)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/app/backend/open_webui/utils/models.py", line 63, in get_all_models
2025-04-03 13:56:19 models = await get_all_base_models(request, user=user)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/app/backend/open_webui/utils/models.py", line 43, in get_all_base_models
2025-04-03 13:56:19 ollama_models = await ollama.get_all_models(request, user=user)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 109, in wrapper
2025-04-03 13:56:19 return await self.decorator(f, *args, **kwargs)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/usr/local/lib/python3.11/site-packages/aiocache/decorators.py", line 124, in decorator
2025-04-03 13:56:19 result = await f(*args, **kwargs)
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/app/backend/open_webui/routers/ollama.py", line 369, in get_all_models
2025-04-03 13:56:19 "models": merge_models_lists(
2025-04-03 13:56:19 ^^^^^^^^^^^^^^^^^^^
2025-04-03 13:56:19 File "/app/backend/open_webui/routers/ollama.py", line 359, in merge_models_lists
2025-04-03 13:56:19 id = model["model"]
2025-04-03 13:56:19 ~~~~~^^^^^^^^^
2025-04-03 13:56:19 KeyError: 'model'
```
NOTE: relates to https://github.com/open-webui/open-webui/discussions/10145
### Desired Solution you'd like
I've hacked this locally with a new helper method (going from memory - I hacked it right in the docker terminal):
```python
def get_model(model):
return model.get('model') or model.get('name')
```
I had to touch two different files and I didn't take notes, sorry.
I'll try to get my local dev env up and running and issue a PR.
### Alternatives Considered
I also opened this on the `ktransformers` side: https://github.com/kvcache-ai/ktransformers/issues/1039
### Additional Context
Larry and I plan to take over the world, but he has to run on `ktransformers` to do it.
I went to Settings -> Admin Settings -> Connections -> +
Url: http://larry:11434/v1
Key: larry
Clicked Save.
I also deleted the ollama connection by clicking on the gear icon -> delete.
That worked! Thanks!
<!-- gh-comment-id:2780737029 -->
@createthis commented on GitHub (Apr 5, 2025):
> You should use OpenAI API for this.
I went to `Settings` -> `Admin Settings` -> `Connections` -> `+`
Url: `http://larry:11434/v1`
Key: `larry`
Clicked `Save`.
I also deleted the ollama connection by clicking on the gear icon -> `delete`.
That worked! Thanks!
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @createthis on GitHub (Apr 4, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12463
Check Existing Issues
Problem Description
When I hit
http://larry:11434/api/tagslocally, while runningktransformers, I see:When I hit this same endpoint while running
ollama, I see:It appears that
Open WebUIis keying off themodelattribute, so when I try to useOpen WebUIwithktransformers, it fails with this error:NOTE: relates to https://github.com/open-webui/open-webui/discussions/10145
Desired Solution you'd like
I've hacked this locally with a new helper method (going from memory - I hacked it right in the docker terminal):
I had to touch two different files and I didn't take notes, sorry.
I'll try to get my local dev env up and running and issue a PR.
Alternatives Considered
I also opened this on the
ktransformersside: https://github.com/kvcache-ai/ktransformers/issues/1039Additional Context
Larry and I plan to take over the world, but he has to run on
ktransformersto do it.@tjbck commented on GitHub (Apr 5, 2025):
You should use OpenAI API for this.
@createthis commented on GitHub (Apr 5, 2025):
I went to
Settings->Admin Settings->Connections->+Url:
http://larry:11434/v1Key:
larryClicked
Save.I also deleted the ollama connection by clicking on the gear icon ->
delete.That worked! Thanks!