mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-24 08:56:21 -05:00
CLIP on Open-WebUI #3332
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#3332
Reference in New Issue
Block a user
Originally created by @Tanker365 on GitHub (Jan 21, 2025).
hi everyone! I am trying to incorporate CLIP on Open-WebUI. I am trying to process both text and images, and I found CLIP as one of the good embedders to run my interest. Is it possible to do this?
@JPC612 commented on GitHub (Jan 21, 2025):
I’m also trying to incorporate CLIP in Open-WebUI for processing both text and images. Chroma and Qdrant both support multimodal embedding models like CLIP. It would also be great to leverage something like sentence-transformers/clip-ViT-B-32-multilingual-v1 or llamaindex/vdr-2b-multi-v1. However, from what I can tell right now, Open-WebUI only directly supports Ollama and OpenAI embeddings.
Keep in mind that if you’re not using a multimodal LLM, then only the retrieval aspect (i.e., the normal returning of image and text results) will work. Open-WebUI does support multimodal LLMs, but to get the full image+text interaction, you need to send the images along with the prompts or messages to the multimodal model during RAG-chat completion.