The current Arena Model implementation as described in #6538 doesn't make a ton of sense:
"It will randomly select a model from the collection to provide an answer to your query. Once you rank the response (use like or dislike button), the Arena model will reveal which specific model generated it, making this process a blind test. You can regenerate the response to obtain another one from a different model for comparison (note that the second response could come from the same model as the first)."
The leaderboard uses ELO to rank models, but they're not actually being compared to each other. Each response is evaluated in a vacuum and the user has to refer back across regenerated responses to dictate a winner. For instance, I generated a response twice to a question and liked both so I gave them both a thumbs up. However, now my leaderboard shows the model that was liked earlier as having "lost" which doesn't make a ton of sense.
It would make more sense from a user and UI perspective if when a prompt was given two models were picked at random to generate a response side by side, and the user clicked on which response they liked better or if it was a tie.
Originally created by @notasquid1938 on GitHub (Nov 1, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6643
The current Arena Model implementation as described in #6538 doesn't make a ton of sense:
"It will randomly select a model from the collection to provide an answer to your query. Once you rank the response (use like or dislike button), the Arena model will reveal which specific model generated it, making this process a blind test. You can regenerate the response to obtain another one from a different model for comparison (note that the second response could come from the same model as the first)."
The leaderboard uses ELO to rank models, but they're not actually being compared to each other. Each response is evaluated in a vacuum and the user has to refer back across regenerated responses to dictate a winner. For instance, I generated a response twice to a question and liked both so I gave them both a thumbs up. However, now my leaderboard shows the model that was liked earlier as having "lost" which doesn't make a ton of sense.
It would make more sense from a user and UI perspective if when a prompt was given two models were picked at random to generate a response side by side, and the user clicked on which response they liked better or if it was a tie.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @notasquid1938 on GitHub (Nov 1, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6643
The current Arena Model implementation as described in #6538 doesn't make a ton of sense:
"It will randomly select a model from the collection to provide an answer to your query. Once you rank the response (use like or dislike button), the Arena model will reveal which specific model generated it, making this process a blind test. You can regenerate the response to obtain another one from a different model for comparison (note that the second response could come from the same model as the first)."
The leaderboard uses ELO to rank models, but they're not actually being compared to each other. Each response is evaluated in a vacuum and the user has to refer back across regenerated responses to dictate a winner. For instance, I generated a response twice to a question and liked both so I gave them both a thumbs up. However, now my leaderboard shows the model that was liked earlier as having "lost" which doesn't make a ton of sense.
It would make more sense from a user and UI perspective if when a prompt was given two models were picked at random to generate a response side by side, and the user clicked on which response they liked better or if it was a tie.