WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Open accessAbstract
We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of ``slightly better/worse'' to ``tie'' if the winner response exceeds the loser one by more than $K$ characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study
Xu, L.-C., Zhao, S., Lin, Q.-M., Chen, L., Luo, Q.-Q., Wu, S.-S., Ye, X., Feng, H.-L., & Du, Z.-H. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2480268 · Cited by 40 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
“This is most closely related to other benchmarks with rubric-based grading, such as VISTA (Scale AI, 2025), PaperBench (Starace et al., 2025), WildBench (Lin et al., 2025), AMEGA (Fast et al., 2024), and MultiChallenge (Sirdeshmukh et al., 2025).”
The future of open human feedback
Don-Yehiya, S., Burtenshaw, B., Astudillo, R. F., Osborne, C., Jaiswal, M., Kuo, T.-S., Zhao, W., Shenfeld, I., Peng, A., Yurochkin, M., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01038-2 · Cited by 6 · see citing works
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
Liu, Y., Zhang, M. J. Q., & Choi, E. (2025) · arXiv · doi:10.48550/arxiv.2507.23158 · Cited by 3 · see citing works
“WildBench contains 1024 questions manually selected from the same source of WildChat .”
InfluentialDolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 6 · see citing works
Evidence from formal logical reasoning reveals that the language of thought is not natural language
Kean, H., Fung, A., Jaggers, P., Chen, J., Rule, J. S., Benn, Y., Tenenbaum, J. B., Piantadosi, S. T., Varley, R. A., & Fedorenko, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520095123 · Cited by 2 · see citing works
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
“Then, to evaluate these responses, a separate LLM (referred to as the judge model or LLM Judge ) is used to generate a quality score.”
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
“Additionally, inline citations were disabled for the search model in the Text Arena to avoid vote bias.”
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
Xie, S., Shi, Z., Shen, H., Wang, J., Zhong, S., Liu, M., Pan, C., Jia, P., Sun, B., Jing, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.21482
“B ENCH currently comprises 100 tasks spanning six domain categories: Technology (25), Energy & Materials (20), Industrials & Transport (18), Consumer (16), Finance (12), and Healthcare & Pharma (9).”
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Li, R., Du, M., Xu, B., Zhu, C., Wang, X., & Mao, Z. (2026) · arXiv · doi:10.48550/arxiv.2601.08536
“Recently, rubrics have been widely adopted for these types of evaluation tasks [Arora et al., 2025, Lin et al., 2024, Sirdeshmukh et al., 2025, Starace et al., 2025, Gou et al., 2025].”
- Authors
- Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.04770
- OpenAlex
- W4399511950
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- automated assessment, pairwise comparison, evaluation metrics, Elo rating, Chatbot Arena
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
