[GH-ISSUE #4510] Would it be possible for Ollama to support re-rank models? #28587

Closed
opened 2026-04-22 06:58:07 -05:00 by GiteaMirror · 76 comments
Owner

Originally created by @lyfuci on GitHub (May 18, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4510

I am using Ollama for my projects and it's been great. However, when using some AI app platform, like dify, build RAG app, rerank is nessesary. It's possible for Ollama to support rerank models.

Originally created by @lyfuci on GitHub (May 18, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4510 I am using Ollama for my projects and it's been great. However, when using some AI app platform, like dify, build RAG app, rerank is nessesary. It's possible for Ollama to support rerank models.
GiteaMirror added the feature request label 2026-04-22 06:58:07 -05:00
Author
Owner

@igorschlum commented on GitHub (May 18, 2024):

@lyfuci Ollama only support OpenSource models. Before adding a model to Ollama, it is often available on Hunging Face.

<!-- gh-comment-id:2118667391 --> @igorschlum commented on GitHub (May 18, 2024): @lyfuci Ollama only support OpenSource models. Before adding a model to Ollama, it is often available on Hunging Face.
Author
Owner

@lyfuci commented on GitHub (May 18, 2024):

@igorschlum sorry, just set a typo, reranker->rerank. reranker model means a model input some sentences and a query, output a list of similarity.
There is some reranker model on Hugging Face, like bge-reranker-v2-m3,jina-reranker-v1-base-en etc.

<!-- gh-comment-id:2118897344 --> @lyfuci commented on GitHub (May 18, 2024): @igorschlum sorry, just set a typo, reranker->rerank. reranker model means a model input some sentences and a query, output a list of similarity. There is some reranker model on Hugging Face, like [bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3),[jina-reranker-v1-base-en](https://jina.ai/reranker/) etc.
Author
Owner

@ah3243 commented on GitHub (Jun 1, 2024):

just to bump this, i agree, I had to switch from ollama to transformers library when doing rag to use a reranker. Also maybe a wider range of embedding models in general or some whay to search for or filter them. I found that bge embeddings like m3 or large outperformed the largest embedding model currently on ollama: mxbai-embed-large.

<!-- gh-comment-id:2143415879 --> @ah3243 commented on GitHub (Jun 1, 2024): just to bump this, i agree, I had to switch from ollama to transformers library when doing rag to use a reranker. Also maybe a wider range of embedding models in general or some whay to search for or filter them. I found that bge embeddings like m3 or large outperformed the largest embedding model currently on ollama: mxbai-embed-large.
Author
Owner

@edisonzf2020 commented on GitHub (Jun 5, 2024):

+1

<!-- gh-comment-id:2148978050 --> @edisonzf2020 commented on GitHub (Jun 5, 2024): +1
Author
Owner

@jerryandjune commented on GitHub (Jun 12, 2024):

+1

<!-- gh-comment-id:2161857093 --> @jerryandjune commented on GitHub (Jun 12, 2024): +1
Author
Owner

@yushengliao commented on GitHub (Jun 12, 2024):

+1

<!-- gh-comment-id:2162001180 --> @yushengliao commented on GitHub (Jun 12, 2024): +1
Author
Owner

@valckmir commented on GitHub (Jun 14, 2024):

Please, add reranking models to Ollama. 🥺

<!-- gh-comment-id:2168132623 --> @valckmir commented on GitHub (Jun 14, 2024): Please, add reranking models to Ollama. 🥺
Author
Owner

@fff196 commented on GitHub (Jun 28, 2024):

+1

<!-- gh-comment-id:2195975675 --> @fff196 commented on GitHub (Jun 28, 2024): +1
Author
Owner

@rjmalagon commented on GitHub (Jun 29, 2024):

rerank and colbert would be wonderful, compliment embedding nicely on RAG.
Good open models exist
https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1
https://huggingface.co/mixedbread-ai/mxbai-colbert-large-v1
Does llama.cpp support this?

<!-- gh-comment-id:2198348053 --> @rjmalagon commented on GitHub (Jun 29, 2024): rerank and colbert would be wonderful, compliment embedding nicely on RAG. Good open models exist https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1 https://huggingface.co/mixedbread-ai/mxbai-colbert-large-v1 Does llama.cpp support this?
Author
Owner

@rjmalagon commented on GitHub (Jun 29, 2024):

I bring this as an idea on llama.cpp discussions.
https://github.com/ggerganov/llama.cpp/discussions/8216

<!-- gh-comment-id:2198360374 --> @rjmalagon commented on GitHub (Jun 29, 2024): I bring this as an idea on llama.cpp discussions. https://github.com/ggerganov/llama.cpp/discussions/8216
Author
Owner

@lucasdengcn commented on GitHub (Jul 10, 2024):

+1

<!-- gh-comment-id:2219782380 --> @lucasdengcn commented on GitHub (Jul 10, 2024): +1
Author
Owner

@queentrapwatchout commented on GitHub (Jul 10, 2024):

+1

<!-- gh-comment-id:2220623376 --> @queentrapwatchout commented on GitHub (Jul 10, 2024): +1
Author
Owner

@akan commented on GitHub (Jul 30, 2024):

+1

<!-- gh-comment-id:2257968295 --> @akan commented on GitHub (Jul 30, 2024): +1
Author
Owner

@hongliangdu commented on GitHub (Jul 30, 2024):

+1

<!-- gh-comment-id:2259025069 --> @hongliangdu commented on GitHub (Jul 30, 2024): +1
Author
Owner

@springrain commented on GitHub (Aug 20, 2024):

+1

<!-- gh-comment-id:2298104368 --> @springrain commented on GitHub (Aug 20, 2024): +1
Author
Owner

@Zakinator123 commented on GitHub (Aug 20, 2024):

+1

<!-- gh-comment-id:2298987442 --> @Zakinator123 commented on GitHub (Aug 20, 2024): +1
Author
Owner

@jmorganca commented on GitHub (Sep 2, 2024):

Merging with https://github.com/ollama/ollama/issues/3368

<!-- gh-comment-id:2325295498 --> @jmorganca commented on GitHub (Sep 2, 2024): Merging with https://github.com/ollama/ollama/issues/3368
Author
Owner

@netandreus commented on GitHub (Nov 29, 2024):

+1 We need this.

<!-- gh-comment-id:2507592619 --> @netandreus commented on GitHub (Nov 29, 2024): +1 We need this.
Author
Owner

@Kraego commented on GitHub (Dec 4, 2024):

+1

<!-- gh-comment-id:2517850168 --> @Kraego commented on GitHub (Dec 4, 2024): +1
Author
Owner

@bruno-oliveira commented on GitHub (Dec 5, 2024):

Any updates on this?

<!-- gh-comment-id:2520268036 --> @bruno-oliveira commented on GitHub (Dec 5, 2024): Any updates on this?
Author
Owner

@cope commented on GitHub (Dec 6, 2024):

Check #3749

Also: https://ollama.com/search?q=rerank

<!-- gh-comment-id:2523025292 --> @cope commented on GitHub (Dec 6, 2024): Check #3749 Also: https://ollama.com/search?q=rerank
Author
Owner

@netandreus commented on GitHub (Dec 12, 2024):

Unfortunatelly, this models cant work without API support from Ollama.

<!-- gh-comment-id:2537952190 --> @netandreus commented on GitHub (Dec 12, 2024): Unfortunatelly, this models cant work without API support from Ollama.
Author
Owner

@brightleo commented on GitHub (Dec 14, 2024):

Will Ollama support it in the future? We need support for the renank model.

<!-- gh-comment-id:2543031581 --> @brightleo commented on GitHub (Dec 14, 2024): Will Ollama support it in the future? We need support for the renank model.
Author
Owner

@Crimson-Hawk-1 commented on GitHub (Jan 7, 2025):

WE NEED rerank MODEL SUPPORT ASAP!!! PLEASE!!!!!!!!!

<!-- gh-comment-id:2575462611 --> @Crimson-Hawk-1 commented on GitHub (Jan 7, 2025): WE NEED rerank MODEL SUPPORT ASAP!!! PLEASE!!!!!!!!!
Author
Owner

@Rakhsan commented on GitHub (Jan 20, 2025):

HEll yeah pls do the thing. I need rerank models ASAP!

<!-- gh-comment-id:2602461249 --> @Rakhsan commented on GitHub (Jan 20, 2025): HEll yeah pls do the thing. I need rerank models ASAP!
Author
Owner

@Yak0vkaSup commented on GitHub (Feb 4, 2025):

+1

<!-- gh-comment-id:2633270354 --> @Yak0vkaSup commented on GitHub (Feb 4, 2025): +1
Author
Owner

@alekhed commented on GitHub (Feb 4, 2025):

+1

<!-- gh-comment-id:2635251703 --> @alekhed commented on GitHub (Feb 4, 2025): +1
Author
Owner

@Frank995 commented on GitHub (Feb 10, 2025):

+1

<!-- gh-comment-id:2647226685 --> @Frank995 commented on GitHub (Feb 10, 2025): +1
Author
Owner

@StanleyMXY commented on GitHub (Feb 11, 2025):

+1

<!-- gh-comment-id:2651113157 --> @StanleyMXY commented on GitHub (Feb 11, 2025): +1
Author
Owner

@adrianwysocki1993 commented on GitHub (Feb 11, 2025):

+1

<!-- gh-comment-id:2651269801 --> @adrianwysocki1993 commented on GitHub (Feb 11, 2025): +1
Author
Owner

@Leon371 commented on GitHub (Feb 12, 2025):

+1

<!-- gh-comment-id:2652482317 --> @Leon371 commented on GitHub (Feb 12, 2025): +1
Author
Owner

@virdb commented on GitHub (Feb 12, 2025):

+1

<!-- gh-comment-id:2653001644 --> @virdb commented on GitHub (Feb 12, 2025): +1
Author
Owner

@like-a-bause commented on GitHub (Feb 12, 2025):

+1

<!-- gh-comment-id:2653337086 --> @like-a-bause commented on GitHub (Feb 12, 2025): +1
Author
Owner

@torcoste commented on GitHub (Feb 12, 2025):

+1

<!-- gh-comment-id:2653416336 --> @torcoste commented on GitHub (Feb 12, 2025): +1
Author
Owner

@netandreus commented on GitHub (Feb 13, 2025):

+1

<!-- gh-comment-id:2656090163 --> @netandreus commented on GitHub (Feb 13, 2025): +1
Author
Owner

@ytfrdfiw commented on GitHub (Feb 15, 2025):

+1

<!-- gh-comment-id:2660848920 --> @ytfrdfiw commented on GitHub (Feb 15, 2025): +1
Author
Owner

@fernvenue commented on GitHub (Feb 16, 2025):

+1

<!-- gh-comment-id:2661599527 --> @fernvenue commented on GitHub (Feb 16, 2025): +1
Author
Owner

@rowstop commented on GitHub (Feb 19, 2025):

+1

<!-- gh-comment-id:2667299096 --> @rowstop commented on GitHub (Feb 19, 2025): +1
Author
Owner

@Doc-Robert commented on GitHub (Feb 19, 2025):

+1

<!-- gh-comment-id:2667386487 --> @Doc-Robert commented on GitHub (Feb 19, 2025): +1
Author
Owner

@cxhttt233 commented on GitHub (Feb 19, 2025):

+1

<!-- gh-comment-id:2668189107 --> @cxhttt233 commented on GitHub (Feb 19, 2025): +1
Author
Owner

@Yak0vkaSup commented on GitHub (Feb 19, 2025):

+1

<!-- gh-comment-id:2668797612 --> @Yak0vkaSup commented on GitHub (Feb 19, 2025): +1
Author
Owner

@Rakhsan commented on GitHub (Feb 19, 2025):

thing ain't even fixed why is this closed and everyone spamming +1

<!-- gh-comment-id:2669190373 --> @Rakhsan commented on GitHub (Feb 19, 2025): thing ain't even fixed why is this closed and everyone spamming +1
Author
Owner

@rjmalagon commented on GitHub (Feb 19, 2025):

The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this.

# use the /reranking endpoint
llama-server -m model.gguf --reranking
<!-- gh-comment-id:2669220752 --> @rjmalagon commented on GitHub (Feb 19, 2025): The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this. ``` # use the /reranking endpoint llama-server -m model.gguf --reranking ```
Author
Owner

@ytfrdfiw commented on GitHub (Feb 20, 2025):

The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this.

# use the /reranking endpoint
llama-server -m model.gguf --reranking

Is this true? Have you done any performance or burn-in testing? thank you!

<!-- gh-comment-id:2670875188 --> @ytfrdfiw commented on GitHub (Feb 20, 2025): > The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this. > > ``` > # use the /reranking endpoint > llama-server -m model.gguf --reranking > ``` Is this true? Have you done any performance or burn-in testing? thank you!
Author
Owner

@rjmalagon commented on GitHub (Feb 20, 2025):

The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this.

# use the /reranking endpoint
llama-server -m model.gguf --reranking

Is this true? Have you done any performance or burn-in testing? thank you!

I did not test it, my particular choice for reranker model use (jina-colbert-v2) is not enterely vainilla transformers and I suppose it will not work on llama.cpp, it seems too much effort to integrate the llama.cpp server for reranking on my mostly ollama pipeline to test it. But I think it is a first step that llama.cpp offer any reranking model support.

I use heavy embedding models (7B), slow but fortunately GPU accelerated by ollama, CPU reranking is just a little bottleneck for now for me.

<!-- gh-comment-id:2672653478 --> @rjmalagon commented on GitHub (Feb 20, 2025): > > The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this. > > ``` > > # use the /reranking endpoint > > llama-server -m model.gguf --reranking > > ``` > > Is this true? Have you done any performance or burn-in testing? thank you! I did not test it, my particular choice for reranker model use (jina-colbert-v2) is not enterely vainilla transformers and I suppose it will not work on llama.cpp, it seems too much effort to integrate the llama.cpp server for reranking on my mostly ollama pipeline to test it. But I think it is a first step that llama.cpp offer any reranking model support. I use heavy embedding models (7B), slow but fortunately GPU accelerated by ollama, CPU reranking is just a little bottleneck for now for me.
Author
Owner

@ytfrdfiw commented on GitHub (Feb 21, 2025):

The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this.

# use the /reranking endpoint
llama-server -m model.gguf --reranking

Is this true? Have you done any performance or burn-in testing? thank you!

I did not test it, my particular choice for reranker model use (jina-colbert-v2) is not enterely vainilla transformers and I suppose it will not work on llama.cpp, it seems too much effort to integrate the llama.cpp server for reranking on my mostly ollama pipeline to test it. But I think it is a first step that llama.cpp offer any reranking model support.

I use heavy embedding models (7B), slow but fortunately GPU accelerated by ollama, CPU reranking is just a little bottleneck for now for me.

ok! Thank you! I got it

<!-- gh-comment-id:2673109368 --> @ytfrdfiw commented on GitHub (Feb 21, 2025): > > > The llama.cpp project supports reranrikng, maybe it is not a priority for the Ollama project, but someone can offer the missing code to enable this. > > > ``` > > > # use the /reranking endpoint > > > llama-server -m model.gguf --reranking > > > ``` > > > > > > Is this true? Have you done any performance or burn-in testing? thank you! > > I did not test it, my particular choice for reranker model use (jina-colbert-v2) is not enterely vainilla transformers and I suppose it will not work on llama.cpp, it seems too much effort to integrate the llama.cpp server for reranking on my mostly ollama pipeline to test it. But I think it is a first step that llama.cpp offer any reranking model support. > > I use heavy embedding models (7B), slow but fortunately GPU accelerated by ollama, CPU reranking is just a little bottleneck for now for me. ok! Thank you! I got it
Author
Owner

@darkSuperman commented on GitHub (Feb 24, 2025):

I want to use ollama to manage all models in a unified way: LLM, Embedding, Rerank (Embedding), but Rerank is not supported yet. sad

<!-- gh-comment-id:2677451052 --> @darkSuperman commented on GitHub (Feb 24, 2025): I want to use `ollama `to manage all models in a unified way: **LLM,** **Embedding**, **Rerank** (Embedding), but **Rerank** is not supported yet. sad
Author
Owner

@dannyxu2015 commented on GitHub (Mar 5, 2025):

+1

<!-- gh-comment-id:2700648116 --> @dannyxu2015 commented on GitHub (Mar 5, 2025): +1
Author
Owner

@KinXdle commented on GitHub (Mar 6, 2025):

+1

<!-- gh-comment-id:2703074467 --> @KinXdle commented on GitHub (Mar 6, 2025): +1
Author
Owner

@gelonsoft commented on GitHub (Mar 6, 2025):

+1

<!-- gh-comment-id:2703513019 --> @gelonsoft commented on GitHub (Mar 6, 2025): +1
Author
Owner

@febef commented on GitHub (Mar 11, 2025):

+1

<!-- gh-comment-id:2712253232 --> @febef commented on GitHub (Mar 11, 2025): +1
Author
Owner

@rjmalagon commented on GitHub (Mar 11, 2025):

Mixedbread AI released their Qwen2 based rerankers, mxbai-rerank-base-v2 and mxbai-rerank-large-v2.
https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2
https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2
I think these high quality rerankers are supported by llama.cpp

<!-- gh-comment-id:2712366467 --> @rjmalagon commented on GitHub (Mar 11, 2025): Mixedbread AI released their Qwen2 based rerankers, mxbai-rerank-base-v2 and mxbai-rerank-large-v2. https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2 https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2 I think these high quality rerankers are supported by llama.cpp
Author
Owner

@TheNha commented on GitHub (Mar 13, 2025):

+1
@darkSuperman Me tooooooooooooo.

<!-- gh-comment-id:2720454737 --> @TheNha commented on GitHub (Mar 13, 2025): +1 @darkSuperman Me tooooooooooooo.
Author
Owner

@sean-liang commented on GitHub (Mar 15, 2025):

+1

<!-- gh-comment-id:2726115089 --> @sean-liang commented on GitHub (Mar 15, 2025): +1
Author
Owner

@RedamancyJ commented on GitHub (Mar 19, 2025):

+1

<!-- gh-comment-id:2735242986 --> @RedamancyJ commented on GitHub (Mar 19, 2025): +1
Author
Owner

@lilozz2 commented on GitHub (Mar 19, 2025):

+1

<!-- gh-comment-id:2735640015 --> @lilozz2 commented on GitHub (Mar 19, 2025): +1
Author
Owner

@jagga99 commented on GitHub (Mar 20, 2025):

+1

<!-- gh-comment-id:2741577464 --> @jagga99 commented on GitHub (Mar 20, 2025): +1
Author
Owner

@bennos1911 commented on GitHub (Mar 29, 2025):

+1

<!-- gh-comment-id:2763192748 --> @bennos1911 commented on GitHub (Mar 29, 2025): +1
Author
Owner

@boscar commented on GitHub (Apr 14, 2025):

+1

<!-- gh-comment-id:2801669708 --> @boscar commented on GitHub (Apr 14, 2025): +1
Author
Owner

@CheinTian commented on GitHub (Apr 16, 2025):

+1

<!-- gh-comment-id:2809675868 --> @CheinTian commented on GitHub (Apr 16, 2025): +1
Author
Owner

@JeeLin commented on GitHub (Apr 17, 2025):

+1

<!-- gh-comment-id:2811392113 --> @JeeLin commented on GitHub (Apr 17, 2025): +1
Author
Owner

@malaki12003 commented on GitHub (Apr 25, 2025):

+1

<!-- gh-comment-id:2829385477 --> @malaki12003 commented on GitHub (Apr 25, 2025): +1
Author
Owner

@ArnaudPannatier commented on GitHub (May 2, 2025):

+1

<!-- gh-comment-id:2847328675 --> @ArnaudPannatier commented on GitHub (May 2, 2025): +1
Author
Owner

@ypotsiah commented on GitHub (May 8, 2025):

+1

<!-- gh-comment-id:2862155945 --> @ypotsiah commented on GitHub (May 8, 2025): +1
Author
Owner

@ufosaga commented on GitHub (May 11, 2025):

+1

<!-- gh-comment-id:2869333086 --> @ufosaga commented on GitHub (May 11, 2025): +1
Author
Owner

@ptemnorusov commented on GitHub (Jun 28, 2025):

+1

<!-- gh-comment-id:3015975117 --> @ptemnorusov commented on GitHub (Jun 28, 2025): +1
Author
Owner

@MarSchra commented on GitHub (Jul 3, 2025):

+1

<!-- gh-comment-id:3033316745 --> @MarSchra commented on GitHub (Jul 3, 2025): +1
Author
Owner

@sinjab commented on GitHub (Jul 12, 2025):

+1

<!-- gh-comment-id:3064555145 --> @sinjab commented on GitHub (Jul 12, 2025): +1
Author
Owner

@Teayyyy commented on GitHub (Jul 23, 2025):

+1

<!-- gh-comment-id:3106413034 --> @Teayyyy commented on GitHub (Jul 23, 2025): +1
Author
Owner

@HardenGale commented on GitHub (Aug 12, 2025):

+1

<!-- gh-comment-id:3178239925 --> @HardenGale commented on GitHub (Aug 12, 2025): +1
Author
Owner

@xiexin36 commented on GitHub (Aug 26, 2025):

+1

<!-- gh-comment-id:3222829185 --> @xiexin36 commented on GitHub (Aug 26, 2025): +1
Author
Owner

@dengxifeng commented on GitHub (Sep 25, 2025):

+1

<!-- gh-comment-id:3332859757 --> @dengxifeng commented on GitHub (Sep 25, 2025): +1
Author
Owner

@Gwada commented on GitHub (Oct 12, 2025):

+1

<!-- gh-comment-id:3394112797 --> @Gwada commented on GitHub (Oct 12, 2025): +1
Author
Owner

@yen936 commented on GitHub (Jan 13, 2026):

+1

<!-- gh-comment-id:3745424955 --> @yen936 commented on GitHub (Jan 13, 2026): +1
Author
Owner

@TaylorPzreal commented on GitHub (Jan 31, 2026):

+1

llama.cpp supported.
download and start server
llama-server -hf model.GGUF --reranking

<!-- gh-comment-id:3828131270 --> @TaylorPzreal commented on GitHub (Jan 31, 2026): +1 llama.cpp supported. download and start server `llama-server -hf model.GGUF --reranking`
Author
Owner

@zanyar3 commented on GitHub (Feb 25, 2026):

+1

<!-- gh-comment-id:3961279494 --> @zanyar3 commented on GitHub (Feb 25, 2026): +1
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#28587