github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-03-11 00:04:08 -05:00
Code Issues 308 Packages Projects Releases 100 Wiki Activity

CLIP on Open-WebUI #3332

New Issue
Closed
opened 2025-11-11 15:29:24 -06:00 by GiteaMirror · 1 comment
GiteaMirror commented 2025-11-11 15:29:24 -06:00
Owner
Copy Link

Originally created by @Tanker365 on GitHub (Jan 21, 2025).

hi everyone! I am trying to incorporate CLIP on Open-WebUI. I am trying to process both text and images, and I found CLIP as one of the good embedders to run my interest. Is it possible to do this?

Originally created by @Tanker365 on GitHub (Jan 21, 2025). hi everyone! I am trying to incorporate CLIP on Open-WebUI. I am trying to process both text and images, and I found CLIP as one of the good embedders to run my interest. Is it possible to do this?
GiteaMirror commented 2025-11-11 15:29:25 -06:00
Author
Owner
Copy Link

@JPC612 commented on GitHub (Jan 21, 2025):

I’m also trying to incorporate CLIP in Open-WebUI for processing both text and images. Chroma and Qdrant both support multimodal embedding models like CLIP. It would also be great to leverage something like sentence-transformers/clip-ViT-B-32-multilingual-v1 or llamaindex/vdr-2b-multi-v1. However, from what I can tell right now, Open-WebUI only directly supports Ollama and OpenAI embeddings.

Keep in mind that if you’re not using a multimodal LLM, then only the retrieval aspect (i.e., the normal returning of image and text results) will work. Open-WebUI does support multimodal LLMs, but to get the full image+text interaction, you need to send the images along with the prompts or messages to the multimodal model during RAG-chat completion.

@JPC612 commented on GitHub (Jan 21, 2025): I’m also trying to incorporate CLIP in Open-WebUI for processing both text and images. Chroma and Qdrant both support multimodal embedding models like CLIP. It would also be great to leverage something like [sentence-transformers/clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) or [llamaindex/vdr-2b-multi-v1](https://huggingface.co/llamaindex/vdr-2b-multi-v1). However, from what I can tell right now, Open-WebUI only directly supports Ollama and OpenAI embeddings. Keep in mind that if you’re not using a multimodal LLM, then only the retrieval aspect (i.e., the normal returning of image and text results) will work. Open-WebUI does support multimodal LLMs, but to get the full image+text interaction, you need to send the images along with the prompts or messages to the multimodal model during RAG-chat completion.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#3332
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?