I have searched the existing issues and discussions.
Problem Description
In evaluation mode it seems models are selected completely randomly. So I have sometimes had three of three replies from the same model. This seems like a waste of electricity. If I wanted three results from Model A to pick the best I could just select it three times. In evaluation mode it seems pointless. Also it might be worth checking, if I don't realise all three are from the same model and upvote a reply, have I just given Model A one upvote and two downvotes?
When generating titles and tags the default is "Use Current". But in an evaluation set, "Current" is randomly picked instead of actually using whichever model was already loaded which is inefficient when testing models locally (unload model, load new model, generate title, unload model, load new model, generate tags).
Thanks for the work though, overall it's a useful frontend.
Desired Solution you'd like
Evaluation mode should attempt to pick different models for each reply rather than be completely random.
In Evaluation sets, "Use current" for title and other generations should use the currently loaded model, not randomly pick one from the evaluation set.
Alternatives Considered
No response
Additional Context
I'm not sure how the statistics work in some circumstances. Say the user has an evaluation set of 3 models and they request 4 replies. Two of the replies would have to be from the same model. So 1xA, 1xB, 2xC. If I upvote A am I downvoting C twice? If I upvote one of the C replies does it downvote itself as well as A & B?
It might be better to add a prompt when all available evaluation models have replied. Something like "All available models have replied. See Admin Panel > Settings > Evaluations to modify sets". instead of a reply from a model twice.
Originally created by @Fluffkin on GitHub (Oct 12, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/18264
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
### Problem Description
In evaluation mode it seems models are selected completely randomly. So I have sometimes had three of three replies from the same model. This seems like a waste of electricity. If I wanted three results from Model A to pick the best I could just select it three times. In evaluation mode it seems pointless. Also it might be worth checking, if I don't realise all three are from the same model and upvote a reply, have I just given Model A one upvote and two downvotes?
When generating titles and tags the default is "Use Current". But in an evaluation set, "Current" is randomly picked instead of actually using whichever model was already loaded which is inefficient when testing models locally (unload model, load new model, generate title, unload model, load new model, generate tags).
Thanks for the work though, overall it's a useful frontend.
### Desired Solution you'd like
Evaluation mode should attempt to pick different models for each reply rather than be completely random.
In Evaluation sets, "Use current" for title and other generations should use the currently loaded model, not randomly pick one from the evaluation set.
### Alternatives Considered
_No response_
### Additional Context
I'm not sure how the statistics work in some circumstances. Say the user has an evaluation set of 3 models and they request 4 replies. Two of the replies would have to be from the same model. So 1xA, 1xB, 2xC. If I upvote A am I downvoting C twice? If I upvote one of the C replies does it downvote itself as well as A & B?
It might be better to add a prompt when all available evaluation models have replied. Something like "All available models have replied. See Admin Panel > Settings > Evaluations to modify sets". instead of a reply from a model twice.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @Fluffkin on GitHub (Oct 12, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/18264
Check Existing Issues
Problem Description
In evaluation mode it seems models are selected completely randomly. So I have sometimes had three of three replies from the same model. This seems like a waste of electricity. If I wanted three results from Model A to pick the best I could just select it three times. In evaluation mode it seems pointless. Also it might be worth checking, if I don't realise all three are from the same model and upvote a reply, have I just given Model A one upvote and two downvotes?
When generating titles and tags the default is "Use Current". But in an evaluation set, "Current" is randomly picked instead of actually using whichever model was already loaded which is inefficient when testing models locally (unload model, load new model, generate title, unload model, load new model, generate tags).
Thanks for the work though, overall it's a useful frontend.
Desired Solution you'd like
Evaluation mode should attempt to pick different models for each reply rather than be completely random.
In Evaluation sets, "Use current" for title and other generations should use the currently loaded model, not randomly pick one from the evaluation set.
Alternatives Considered
No response
Additional Context
I'm not sure how the statistics work in some circumstances. Say the user has an evaluation set of 3 models and they request 4 replies. Two of the replies would have to be from the same model. So 1xA, 1xB, 2xC. If I upvote A am I downvoting C twice? If I upvote one of the C replies does it downvote itself as well as A & B?
It might be better to add a prompt when all available evaluation models have replied. Something like "All available models have replied. See Admin Panel > Settings > Evaluations to modify sets". instead of a reply from a model twice.