CoChat
Sign Up
PreprintCited by 271Artificial Intelligence in LawTopic ModelingEthics and Social Impacts of AI

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge

Open access
Li, Dawei, Jiang, Bohan, Huang, Liangjie, Beigi, Alimohammad, Zhao, Chengshuai, Tan, Zhen, Bhattacharjee, Amrita, Jiang, Yuxuan +5 more·2024·arXiv
DOI 10.48550/arxiv.2411.16594arXiv 2411.16594
Open this paper in CoChat — free accountOpen PDF

Abstract

Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP). Traditional methods, usually matching-based or small model-based, often fall short in open-ended and dynamic scenarios. Recent advancements in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs are leveraged to perform scoring, ranking, or selection for various machine learning evaluation scenarios. This paper presents a comprehensive survey of LLM-based judgment and assessment, offering an in-depth overview to review this evolving field. We first provide the definition from both input and output perspectives. Then we introduce a systematic taxonomy to explore LLM-as-a-judge along three dimensions: what to judge, how to judge, and how to benchmark. Finally, we also highlight key challenges and promising future directions for this emerging area. More resources on LLM-as-a-judge are on the website: https://llm-as-a-judge.github.io and https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.16594

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 386

  1. Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. 2024. Can we use large language models to fill relevance judgment holes? ArXiv preprint, abs/2405.05600. · arXiv
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. ArXiv preprint, abs/2303.08774. · arXiv
  3. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie CY Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In ICML 2024 Work- shop on In-Context Learning.
  4. Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. 2024. i-srt: Aligning large multimodal models for videos by it- erative self-retrospective judgment. ArXiv preprint, abs/2406.11280. · arXiv
  5. Marwah Alaofi, Negar Arabzadeh, Charles LA Clarke, and Mark Sanderson. 2024. Generative information retrieval evaluation. In Information Access in the Era of Generative AI, pages 135-159. Springer.
  6. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Hae- won Jeong, and 1 others. 2024. A survey on data selection for language models. ArXiv preprint, abs/2402.16827. · arXiv
  7. Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Tim Altendorf, Philipp Cimiano, and Benjamin Schiller. 2025. Argument summarization and its eval- uation in the era of large language models. arXiv preprint arXiv:2503.00847. · arXiv
  8. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791. · arXiv
  9. Negar Arabzadeh and Charles LA Clarke. 2025a. Benchmarking llm-based relevance judgment meth- ods. arXiv preprint arXiv:2504.12558. · arXiv
  10. Negar Arabzadeh and Charles LA Clarke. 2025b. A human-ai comparative analysis of prompt sensitiv- ity in llm-based relevance judgment. arXiv preprint arXiv:2504.12408. · arXiv
  11. Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, and Agha Ali Raza. 2024. The fellow- ship of the llms: Multi-agent workflows for synthetic preference optimization dataset generation. ArXiv preprint, abs/2408.08688. · arXiv
  12. Stuart Armstrong, Matija Franklin, Connor Stevens, and Rebecca Gorman. 2025. Defense against the dark prompts: Mitigating best-of-n jailbreaking with prompt evaluation. arXiv preprint arXiv:2502.00580. · arXiv
  13. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Repre- sentations.
  14. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  15. Sher Badshah and Hassan Sajjad. 2024. Reference- guided verdict: Llms-as-judges in automatic evaluation of free-form text. ArXiv preprint, abs/2408.09235. · arXiv
  16. Sher Badshah and Hassan Sajjad. 2025. Dafe: Llm-based evaluation through dynamic arbitration for free-form question-answering. arXiv preprint arXiv:2503.08542. · arXiv
  17. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. ArXiv preprint, abs/2212.08073. · arXiv
  18. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xi- aozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204. · arXiv
  19. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yi- jia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023a. Benchmarking foundation models with language-model-as-an-examiner. In Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
  20. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yi- jia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023b. Benchmarking foundation models with language-model-as-an-examiner. In Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
  21. Krisztian Balog, Donald Metzler, and Zhen Qin. 2025. Rankers, judges, and assistants: Towards understand- ing the interplay of llms in information retrieval eval- uation. arXiv preprint arXiv:2503.19092. · arXiv
  22. Chaithanya Bandi and Abir Harrasse. 2024. Ad- versarial multi-agent evaluation of large language models through iterative debates. arXiv preprint arXiv:2410.04663. · arXiv
  23. Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, and Begoña Altuna. 2025. Summarization metrics for spanish and basque: Do automatic scores and llm- judges correlate with humans? arXiv preprint arXiv:2503.17039. · arXiv
  24. Alimohammad Beigi, Bohan Jiang, Dawei Li, Tharindu Kumarage, Zhen Tan, Pouya Shaeri, and Huan Liu. 2024. Lrq-fact: Llm-generated relevant ques- tions for multimodal fact-checking. ArXiv preprint, abs/2410.04616. · arXiv
  25. Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadom- ski, Piotr Nyczyk, and Torsten Hoefler. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 17682-17690. AAAI Press. · DOI · Paper page
  26. Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Dax- iong Ji, Guozhou Zheng, and Huajun Chen. 2023. Oceangpt: A large language model for ocean science tasks. ArXiv preprint, abs/2310.02031. · arXiv
  27. Nathan Brake and Thomas Schaaf. 2024. Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency? In Find- ings of the Association for Computational Linguistics: NAACL 2024, pages 352-363.
  28. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  29. Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human anno- tators with llms. arXiv preprint arXiv:2501.10970. · arXiv
  30. Riccardo Cantini, Alessio Orsino, Massimo Ruggiero, and Domenico Talia. 2025. Benchmarking adver- sarial robustness to bias elicitation in large language models: Scalable automated assessment with llm-as- a-judge. arXiv preprint arXiv:2504.07887. · arXiv
  31. Hongliu Cao, Ilias Driouich, Robin Singh, and Eoin Thomas. 2025a. Multi-agent llm judge: automatic personalized llm judge design for evaluating natu- ral language generation applications. arXiv preprint arXiv:2504.02867. · arXiv
  32. Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Hao- ran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, and 1 others. 2025b. Video sim- pleqa: Towards factuality evaluation in large video language models. arXiv preprint arXiv:2503.18923. · arXiv
  33. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. In The Twelfth Interna- tional Conference on Learning Representations.
  34. Jiayi Chang, Mingqi Gao, Xinyu Hu, and Xiaojun Wan. 2025. Exploring the multilingual nlg evalu- ation abilities of llm-based evaluators. arXiv preprint arXiv:2503.04360. · arXiv
  35. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Tech- nology, 15(3):1-45.
  36. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision- language benchmark. In Forty-first International Conference on Machine Learning.
  37. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024a. MLLM-as- a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. In Forty-first Interna- tional Conference on Machine Learning.
  38. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024b. Humans or llms as the judge? a study on judgement biases. ArXiv preprint, abs/2402.10669. · arXiv
  39. Hongyu Chen and Seraphina Goldfarb-Tarrant. 2025. Safer or luckier? llms as safety evaluators are not robust to artifacts. arXiv preprint arXiv:2503.09347. · arXiv
  40. Junjie Chen, Weihang Su, Zhumin Chu, Haitao Li, Qinyao Ai, Yiqun Liu, Min Zhang, and Shaoping Ma. 2024c. An automatic and cost-efficient peer-review framework for language generation evaluation. arXiv preprint arXiv:2410.12265. · arXiv
  41. Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengx- iang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, and 1 others. 2024d. Au- tomated evaluation of large vision-language mod- els on self-driving corner cases. ArXiv preprint, abs/2404.10595. · arXiv
  42. Meilin Chen, Jian Tian, Liang Ma, Di Xie, Weijie Chen, and Jiang Zhu. 2025a. Unbiased evaluation of large language models from a causal perspective. arXiv preprint arXiv:2502.06655. · arXiv
  43. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025b. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050. · arXiv
  44. Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. 2025c. Do llm evaluators pre- fer themselves for a reason? arXiv preprint arXiv:2504.03846. · arXiv
  45. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, and 1 others. 2024e. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187. · arXiv
  46. Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, and 1 others. 2025d. Rm- r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387. · arXiv
  47. Yen-Shan Chen, Jing Jin, Peng-Ting Kuo, Chao-Wei Huang, and Yun-Nung Chen. 2024f. Llms are biased evaluators but not biased for retrieval augmented gen- eration. ArXiv preprint, abs/2410.20833. · arXiv
  48. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large lan- guage models for reference-free text quality evalua- tion: An empirical study. In Findings of the Associa- tion for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 361-374.
  49. Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, and 1 others. 2024g. Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation? ArXiv preprint, abs/2407.04842. · arXiv
  50. Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and 1 oth- ers. 2023. Evaluating hallucinations in chinese large language models. ArXiv preprint, abs/2310.03368. · arXiv
  51. Wayne Chi, Valerie Chen, Anastasios Nikolas An- gelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar. 2025. Copilot arena: A platform for code llm evaluation in the wild. arXiv preprint arXiv:2502.09328. · arXiv
  52. Cheng-Han Chiang and Hung-yi Lee. 2023a. Can large language models be an alternative to human evalua- tions? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15607-15631, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  53. Cheng-Han Chiang and Hung-yi Lee. 2023b. A closer look into automatic evaluation using large language models. arXiv preprint arXiv:2310.05657. · arXiv
  54. Cheng-Han Chiang, Hung-yi Lee, and Michal Lukasik. 2025. Tract: Regression-aware fine-tuning meets chain-of-thought reasoning for llm-as-a-judge. arXiv preprint arXiv:2503.04381. · arXiv
  55. Marianne Chuang, Gabriel Chuang, Cheryl Chuang, and John Chuang. 2025. Judging it, washing it: Scoring and greenwashing corporate climate disclo- sures using large language models. arXiv preprint arXiv:2502.15094. · arXiv
  56. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  57. Preetam Prabhu Srikar Dammu, Himanshu Naidu, and Chirag Shah. 2025. Dynamic-kgqa: A scalable framework for generating adaptive question answer- ing datasets. arXiv preprint arXiv:2503.05049. · arXiv
  58. Soumik Dey, Hansi Wu, and Binbin Li. 2025. To judge or not to judge: Using llm judgements for adver- tiser keyphrase relevance at ebay. arXiv preprint arXiv:2505.04209. · arXiv
  59. Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, and Nick Craswell. 2025. Llm- evaluation tropes: Perspectives on the validity of llm-evaluations. arXiv preprint arXiv:2504.19076. · arXiv
  60. Sumanth Doddapaneni, Mohammed Safi Ur Rah- man Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M Khapra. 2024a. Cross- lingual auto evaluation for assessing multilingual llms. arXiv preprint arXiv:2410.13394. · arXiv
  61. Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Sshubam Verma, and Mitesh M Khapra. 2024b. Finding blind spots in evaluator llms with inter- pretable checklists. ArXiv preprint, abs/2406.13439. · arXiv
  62. Tobias Domhan and Dawei Zhu. 2025. Same evalua- tion, more tokens: On the effect of input length for machine translation evaluation using large language models. arXiv preprint arXiv:2505.01761. · arXiv
  63. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2023. A survey on in-context learning. ArXiv preprint, abs/2301.00234. · arXiv
  64. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? ArXiv preprint, abs/2406.11657. · arXiv
  65. Yann Dubois, Balázs Galambosi, Percy Liang, and Tat- sunori B Hashimoto. 2024. Length-controlled al- pacaeval: A simple way to debias automatic evalua- tors. arXiv preprint arXiv:2404.04475. · arXiv
  66. Francisco Eiras, Eliott Zemour, Eric Lin, and Vaikkunth Mugunthan. 2025. Know thy judge: On the robust- ness meta-evaluation of llm safety judges. arXiv preprint arXiv:2503.04474. · arXiv
  67. Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Bodapati, and Dan Roth. 2024. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of auto- matic evaluation and llm-as-a-judge. arXiv preprint arXiv:2410.03775. · arXiv
  68. Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Pot- thast, Benno Stein, and 1 others. 2023. Perspec- tives on large language models for relevance judg- ment. In Proceedings of the 2023 ACM SIGIR Inter- national Conference on Theory of Information Re- trieval, pages 39-50.
  69. Zhiyuan Fan, Weinong Wang, Xing Wu, and De- bing Zhang. 2025. Sedareval: Automated evalu- ation using self-adaptive rubrics. arXiv preprint arXiv:2501.15595. · arXiv
  70. Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Per- ozzi. 2024. Test of time: A benchmark for evalu- ating llms on temporal reasoning. ArXiv preprint, abs/2406.09170. · arXiv
  71. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmark- ing legal knowledge of large language models. ArXiv preprint, abs/2309.16289. · arXiv
  72. Tao Feng, Yihang Sun, and Jiaxuan You. 2025. Graphe- val: A lightweight graph-based llm framework for idea evaluation. arXiv preprint arXiv:2503.12600. · arXiv
  73. Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu. 2024. M-mad: Multidimensional multi-agent debate for advanced machine translation evaluation. arXiv preprint arXiv:2412.20127. · arXiv
  74. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556-6576.
  75. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual llm-as-a-judge? arXiv preprint arXiv:2505.12201. · arXiv
  76. Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. ArXiv preprint, abs/2402.01383. · arXiv
  77. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. ArXiv preprint, abs/2304.02554. · arXiv
  78. Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. Trueteacher: Learn- ing factual consistency evaluation with large lan- guage models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pages 2053-2070.
  79. Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, and Asaf Yehudai. 2024. Justrank: Benchmarking llm judges for system ranking. arXiv preprint arXiv:2412.09569. · arXiv
  80. Shashwat Goel, Joschka Struber, Ilze Amanda Auz- ina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. 2025. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313. · arXiv
  81. Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, and Alexandra Choulde- chova. 2025. Validating llm-as-a-judge systems in the absence of gold labels. arXiv preprint arXiv:2503.05965. · arXiv
  82. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. · arXiv
  83. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, and 1 others. 2024. Direct language model alignment from online ai feedback. ArXiv preprint, abs/2402.04792. · arXiv
  84. Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choud- hury, Kalika Bali, and Sunayana Sitaram. 2024. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Findings of the Association for Computational Linguistics: EACL 2024, pages 1051-1070.
  85. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Rea- soning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154-8173, Singapore. Association for Com- putational Linguistics. · DOI · Paper page
  86. Hosein Hasanbeig, Hiteshi Sharma, Leo Betthauser, Fe- lipe Vieira Frujeri, and Ida Momennejad. 2023. Al- lure: auditing and improving llm-based evaluation of text using iterative in-context-learning. arXiv e- prints, pages arXiv-2309.
  87. Hangfeng He, Hongming Zhang, and Dan Roth. 2023. Socreval: Large language models with the so- cratic method for reference-free reasoning evaluation. arXiv preprint arXiv:2310.00074. · arXiv
  88. Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246. · arXiv
  89. Amey Hengle, Aswini Kumar, Anil Bandhakavi, and Tanmoy Chakraborty. 2025. Cseval: Towards automated, multi-dimensional, and reference-free counterspeech evaluation using auto-calibrated llms. arXiv preprint arXiv:2501.17581. · arXiv
  90. Xanh Ho, Jiahao Huang, Florian Boudin, and Akiko Aizawa. 2025. Llm-as-a-judge: Reassessing the per- formance of llms in extractive qa. arXiv preprint arXiv:2504.11972. · arXiv
  91. Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364-381.
  92. Aliyah R Hsu, James Zhu, Zhichao Wang, Bin Bi, Shub- ham Mehrotra, Shiva K Pentyala, Katherine Tan, Xiang-Bo Mao, Roshanak Omrani, Sougata Chaud- huri, and 1 others. 2024. Rate, explain and cite (rec): Enhanced explanation and attribution in automatic evaluation by large language models. arXiv preprint arXiv:2411.02448. · arXiv
  93. Aaron Hu. 2024. Developing an ai-based psychometric system for assessing learning difficulties and adaptive system to overcome: A qualitative and conceptual framework. ArXiv preprint, abs/2403.06284. · arXiv
  94. Renjun Hu, Yi Cheng, Libin Meng, Jiaxin Xia, Yi Zong, Xing Shi, and Wei Lin. 2025a. Training an llm-as-a- judge model: Pipeline, insights, and practical lessons. arXiv preprint arXiv:2502.02988. · arXiv
  95. Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024a. Are llm-based evaluators confusing nlg quality criteria? arXiv preprint arXiv:2402.12055. · arXiv
  96. Xinyu Hu, Mingqi Gao, Li Lin, Zhenghan Yu, and Xiaojun Wan. 2025b. A dual-perspective nlg meta-evaluation framework with automatic bench- mark and better interpretability. arXiv preprint arXiv:2502.12052. · arXiv
  97. Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xi- aojun Wan. 2024b. Themis: A reference-free nlg evaluation language model with flexibility and inter- pretability. ArXiv preprint, abs/2406.18365. · arXiv
  98. Hui Huang, Yingqi Qu, Hongli Zhou, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. 2024a. On the limi- tations of fine-tuned judge models for llm evaluation. arXiv preprint arXiv:2403.02839. · arXiv
  99. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023a. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, pages 1051-1068, Singa- pore. Association for Computational Linguistics. · DOI · Paper page
  100. Yue Huang, Qihui Zhang, Lichao Sun, and 1 others. 2023b. Trustgpt: A benchmark for trustworthy and responsible large language models. ArXiv preprint, abs/2306.11507. · arXiv
  101. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and 1 others. 2023. Llama guard: Llm-based input- output safeguard for human-ai conversations. ArXiv preprint, abs/2312.06674. · arXiv
  102. Andrés Isaza-Giraldo, Paulo Bala, Pedro F Campos, and Lucas Pereira. 2024. Prompt-gaming: A pilot study on llm-evaluating agent in a meaningful energy game. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1- 12.
  103. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720. · arXiv
  104. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023a. Multi-dimensional evaluation of text summarization with in-context learning. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 8487-8495, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  105. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023b. Multi-dimensional evaluation of text summarization with in-context learning. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 8487-8495, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  106. Yeonseok Jeong, Minsoo Kim, Seung-won Hwang, and Byung-Hak Kim. 2025. Agent-as-judge for factual summarization of long narratives. arXiv preprint arXiv:2501.09993. · arXiv
  107. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827-1843.
  108. Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. 2025. Safechain: Safety of language models with long chain-of-thought reasoning capa- bilities. arXiv preprint arXiv:2502.12025. · arXiv
  109. Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2024. Genres: Rethinking evaluation for generative relation extraction in the era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2820-2837.
  110. Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rag- rewardbench: Benchmarking reward models in re- trieval augmented generation for preference align- ment. arXiv preprint arXiv:2412.13746. · arXiv
  111. Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. 2024. Faithscore: Fine-grained evaluations of hallu- cinations in large vision-language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pages 5042-5063.
  112. Jaylen Jones, Lingbo Mo, Eric Fosler-Lussier, and Huan Sun. 2024. A multi-aspect framework for counter narrative evaluation using large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 147-168.
  113. Jaehun Jung, Faeze Brahman, and Yejin Choi. 2024. Trust or escalate: Llm judges with provable guar- antees for human agreement. ArXiv preprint, abs/2407.18370. · arXiv
  114. Nimit Kalra and Leonard Tang. 2025. Verdict: A li- brary for scaling judge-time compute. arXiv preprint arXiv:2502.18018. · arXiv
  115. Ivan Kartáč, Mateusz Lango, and Ondřej Dušek. 2025. Openlgauge: An explainable metric for nlg eval- uation with open-weights llms. arXiv preprint arXiv:2503.11858. · arXiv
  116. Akira Kawabata and Saku Sugawara. 2024. Rationale- aware answer verification by pairwise self-evaluation. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16178-16196.
  117. Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, and 1 others. 2024. Critiquellm: Towards an informative critique gener- ation model for evaluation of large language model generation. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13034-13054.
  118. Zachary Kenton, Noah Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, and 1 others. 2024. On scalable oversight with weak llms judging strong llms. Ad- vances in Neural Information Processing Systems, 37:75229-75276.
  119. Boshra Khalili and Andrew W Smyth. 2025. Autodrive- qa-automated generation of multiple-choice ques- tions for autonomous driving datasets using large vision-language models. arXiv preprint arXiv:2503.15778. · arXiv
  120. Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, and Alice Oh. 2024a. Llm-as-an-interviewer: Beyond static test- ing through dynamic llm evaluation. arXiv preprint arXiv:2412.10424. · arXiv
  121. Heegyu Kim, Taeyang Jeon, Seungtaek Choi, Ji Hoon Hong, Dong Won Jeon, Ga-Yeon Baek, Gyeong- Won Kwak, Dong-Hee Lee, Jisu Bae, Chihoon Lee, and 1 others. 2025. Towards fully-automated ma- terials discovery via large-scale synthesis dataset and expert-level llm-as-a-judge. arXiv preprint arXiv:2502.16457. · arXiv
  122. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and 1 others. Prometheus: Inducing fine-grained evalua- tion capability in language models. In The Twelfth International Conference on Learning Representa- tions.
  123. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. Prometheus 2: An open source language model specialized in evaluating other language mod- els. ArXiv preprint, abs/2405.01535. · arXiv
  124. Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. 2025. Revieweval: An evaluation framework for ai- generated reviews. arXiv preprint arXiv:2502.11736. · arXiv
  125. Tom Kocmi and Christian Federmann. 2023. Large lan- guage models are state-of-the-art evaluators of trans- lation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193-203, Tampere, Finland. Euro- pean Association for Machine Translation.
  126. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Bench- marking cognitive biases in large language models as evaluators. ArXiv preprint, abs/2309.17012. · arXiv
  127. Neema Kotonya, Saran Krishnasamy, Joel Tetreault, and Alejandro Jaimes. 2023. Little giants: Exploring the potential of small LLMs as evaluation metrics in summarization in the Eval4NLP 2023 shared task. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 202-218, Bali, Indonesia. Association for Computational Linguis- tics. · DOI
  128. Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. 2025. No free labels: Limitations of llm-as-a-judge without human ground- ing. arXiv preprint arXiv:2503.05061. · arXiv
  129. Abhishek Kumar, Sonia Haiduc, Partha Pratim Das, and Partha Pratim Chakrabarti. 2024a. Llms as evalua- tors: A novel approach to evaluate bug report sum- marization. arXiv preprint arXiv:2409.00630. · arXiv
  130. Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, and Lama Nachman. 2024b. Decoding biases: Automated methods and llm judges for gender bias detection in language models. ArXiv preprint, abs/2408.03907. · arXiv
  131. Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghaven- dra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen. 2023. Improving diversity of demo- graphic representation in large language models via collective-critiques and self-voting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10383-10405, Singapore. Association for Computational Linguis- tics. · DOI
  132. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, and 1 others. 2024. Rewardbench: Evaluating re- ward models for language modeling. arXiv preprint arXiv:2403.13787. · arXiv
  133. Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. Critice- val: Evaluating large-scale language model as critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  134. Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Min- joon Seo, Kang Min Yoo, and Youngjae Yu. 2024a. Aligning large language models by on-policy self- judgment. ArXiv preprint, abs/2402.11253. · arXiv
  135. Yebin Lee, Imseong Park, and Myungjoo Kang. 2024b. Fleur: An explainable reference-free evaluation met- ric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004. · arXiv
  136. Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bo- han Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025a. Preference leakage: A con- tamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.01534. · arXiv
  137. Dawei Li, Zhen Tan, and Huan Liu. 2024a. Exploring large language models for feature selection: A data- centric perspective. ArXiv preprint, abs/2408.12025. · arXiv
  138. Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sunkwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, and 1 others. 2024c. Dalk: Dynamic co-augmentation of llms and kg to answer alzheimer's disease questions with scientific literature. ArXiv preprint, abs/2405.04819. · arXiv
  139. Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yu- jia Zhou, Qian Dong, and Yiqun Liu. 2024d. Cali- braeval: Calibrating prediction distribution to miti- gate selection bias in llms-as-judges. ArXiv preprint, abs/2410.15393. · arXiv
  140. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. 2024e. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations.
  141. Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yi- fan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Su- jian Li, Bill Yuchen Lin, and 1 others. 2024f. Vl- rewardbench: A challenging benchmark for vision- language generative reward models. arXiv preprint arXiv:2411.17451. · arXiv
  142. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wang- meng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024g. Salad-bench: A hierarchical and compre- hensive safety benchmark for large language models. ArXiv preprint, abs/2402.05044. · arXiv
  143. Mingxuan Li, Hanchen Li, and Chenhao Tan. 2025b. Hypoeval: Hypothesis-guided evaluation for natural language generation. arXiv preprint arXiv:2504.07174. · arXiv
  144. Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy F Chen, and Min-Yen Kan. 2024h. Decom- pose and aggregate: A step-by-step interpretable eval- uation framework. arXiv preprint arXiv:2405.15329. · arXiv
  145. Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023a. Exploring the reliability of large language models as customized evaluators for diverse nlp tasks. arXiv preprint arXiv:2310.19740. · arXiv
  146. Renhao Li, Minghuan Tan, Derek F Wong, and Min Yang. 2024i. Coevol: Constructing better responses for instruction finetuning through multi-agent coop- eration. ArXiv preprint, abs/2406.07054. · arXiv
  147. Ruosen Li, Ruochen Li, Barry Wang, and Xinya Du. 2024j. Iqa-eval: Automatic evaluation of human- model interactive question answering. Advances in Neural Information Processing Systems, 37:109894- 109921.
  148. Ruosen Li, Teerth Patel, and Xinya Du. 2023b. Prd: Peer rank and discussion improve large lan- guage model based evaluations. ArXiv preprint, abs/2307.02762. · arXiv
  149. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024k. From crowdsourced data to high- quality benchmarks: Arena-hard and benchbuilder pipeline. ArXiv preprint, abs/2406.11939. · arXiv
  150. Xiaomin Li, Mingye Gao, Zhiwei Zhang, Chang Yue, and Hong Hu. 2024l. Rule-based data se- lection for large language models. arXiv preprint arXiv:2410.04715. · arXiv
  151. Xiaonan Li and Xipeng Qiu. 2023a. MoT: Memory-of- thought enables ChatGPT to self-improve. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 6354- 6374, Singapore. Association for Computational Lin- guistics. · DOI
  152. Xiaonan Li and Xipeng Qiu. 2023b. Mot: Memory-of- thought enables chatgpt to self-improve. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6354-6374.
  153. Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yon- grui Chen, Wenhao Xu, Guilin Qi, and Dehai Min. 2024m. Mateval: A multi-agent discussion frame- work for advancing open-ended text evaluation. In International Conference on Database Systems for Advanced Applications, pages 415-426. Springer.
  154. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. In The Twelfth International Conference on Learning Representa- tions.
  155. Yuran Li, Jama Hussein Mohamud, Chongren Sun, Di Wu, and Benoit Boulet. 2025c. Leveraging llms as meta-judges: A multi-agent framework for evaluating llm judgments. arXiv preprint arXiv:2504.17087. · arXiv
  156. Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024o. Split and merge: Aligning position biases in llm- based evaluators. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 11084-11108.
  157. Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuanjing Huang, and Zhongyu Wei. 2024a. Debatrix: Multi-dimensinal debate judge with iter- ative chronological analysis based on llm. arXiv preprint arXiv:2403.08010. · arXiv
  158. Sirui Liang, Baoli Zhang, Jun Zhao, and Kang Liu. 2024b. Abseval: An agent-based framework for script evaluation. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 12418-12434.
  159. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. ArXiv preprint, abs/2305.19118. · arXiv
  160. Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, and 1 others. 2024c. I-sheep: Self-alignment of llm from scratch through an iter- ative self-enhancement paradigm. ArXiv preprint, abs/2408.08072. · arXiv
  161. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050. · arXiv
  162. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth Interna- tional Conference on Learning Representations.
  163. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chan- dra Bhagavatula, and Yejin Choi. 2023. The unlock- ing spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations.
  164. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  165. Yen-Ting Lin and Yun-Nung Chen. 2023a. LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language mod- els. In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), pages 47- 58, Toronto, Canada. Association for Computational Linguistics. · DOI
  166. Yen-Ting Lin and Yun-Nung Chen. 2023b. LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language mod- els. In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), pages 47- 58, Toronto, Canada. Association for Computational Linguistics. · DOI
  167. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2025. Evaluating text-to-visual generation with image-to-text generation. In European Confer- ence on Computer Vision, pages 366-384. Springer.
  168. Beiming Liu, Zhizhuo Cui, Siteng Hu, Xiaohua Li, Haifeng Lin, and Zhengxin Zhang. 2025a. Llm eval- uation based on aerospace manufacturing expertise: Automated generation and multi-model question an- swering. arXiv preprint arXiv:2501.17183. · arXiv
  169. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics. · DOI
  170. Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eu- nah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. 2024a. X-eval: Generalizable multi-aspect text evaluation via augmented instruction tuning with auxiliary evaluation aspects. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Pa- pers), pages 8560-8579, Mexico City, Mexico. Asso- ciation for Computational Linguistics.
  171. Rundong Liu, Andre Frade, Amal Vaidya, Maxime Labonne, Marcus Kaiser, Bismayan Chakrabarti, Jonathan Budd, and Sean Moran. 2025b. On iter- ative evaluation and enhancement of code quality using gpt-4o. arXiv preprint arXiv:2502.07399. · arXiv
  172. Shuliang Liu, Xinze Li, Zhenghao Liu, Yukun Yan, Cheng Yang, Zheni Zeng, Zhiyuan Liu, Maosong Sun, and Ge Yu. 2025c. Judge as a judge: Improv- ing the evaluation of retrieval-augmented generation through the judge-consistency of large language mod- els. arXiv preprint arXiv:2502.18817. · arXiv
  173. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023a. Statistical rejection sampling improves preference optimization. ArXiv preprint, abs/2309.06657. · arXiv
  174. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: NLG evaluation using gpt-4 with better human align- ment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511-2522, Singapore. Association for Com- putational Linguistics. · DOI · Paper page
  175. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2025d. Pairwise rm: Perform best-of-n sampling with knockout tournament. arXiv preprint arXiv:2501.13007. · arXiv
  176. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2024b. Aligning with human judgement: The role of pair- wise preference in large language model evaluators. arXiv preprint arXiv:2403.16950. · arXiv
  177. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2023c. Llms as narcissistic evaluators: When ego inflates evaluation scores. ArXiv preprint, abs/2311.09766. · arXiv
  178. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024c. Calibrating LLM- based evaluator. In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 2638-2656, Torino, Italia. ELRA and ICCL.
  179. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024d. Hd-eval: Align- ing large language model evaluators through hi- erarchical criteria decomposition. arXiv preprint arXiv:2402.15754. · arXiv
  180. Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. 2024e. Meta ranking: Less capable language models are capable for single re- sponse judgement. ArXiv preprint, abs/2402.12146. · arXiv
  181. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025e. Inference-time scaling for generalist reward model- ing. arXiv preprint arXiv:2504.02495. · arXiv
  182. Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. LLM comparative assessment: Zero-shot NLG eval- uation through pairwise comparisons using large lan- guage models. In Proceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139-151, St. Julian's, Malta. Association for Computational Linguistics.
  183. Adian Liusie, Potsawee Manakul, and Mark JF Gales. 2023. Zero-shot nlg evaluation through pair- ware comparisons with llms. ArXiv preprint, abs/2307.07889. · arXiv
  184. Edoardo Loru, Jacopo Nudo, Niccolò Di Marco, Matteo Cinelli, and Walter Quattrociocchi. 2025. Decoding ai judgment: How llms assess news credibility and bias. arXiv preprint arXiv:2502.04426. · arXiv
  185. Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zam- brano, Karolina Stańczak, Peter Shaw, Christopher J Pal, and Siva Reddy. 2025. Agentrewardbench: Eval- uating automatic evaluations of web agent trajecto- ries. arXiv preprint arXiv:2504.08942. · arXiv
  186. Yi-Fan Lu, Xian-Ling Mao, Tian Lan, Heyan Huang, Chen Xu, and Xiaoyan Gao. 2024a. Beyond ex- act match: Semantically reassessing event extrac- tion by large language models. arXiv preprint arXiv:2410.09418. · arXiv
  187. Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. 2024b. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Informa- tion Processing Systems, 36.
  188. Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024a. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. ArXiv preprint, abs/2406.07070. · arXiv
  189. Ziyang Luo, Haoning Wu, Dongxu Li, Jing Ma, Mohan Kankanhalli, and Junnan Li. 2024b. Videoautoarena: An automated arena for evaluating large multimodal models in video analysis through user simulation. arXiv preprint arXiv:2411.13281. · arXiv
  190. Shengjie Ma, Chong Chen, Qi Chu, and Jiaxin Mao. 2024. Leveraging large language models for rel- evance judgments in legal case retrieval. ArXiv preprint, abs/2403.18405. · arXiv
  191. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. ArXiv preprint, abs/2305.02156. · arXiv
  192. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singa- pore. Association for Computational Linguistics. · DOI · Paper page
  193. Philipp Mondorf and Barbara Plank. 2024. Beyond accuracy: Evaluating the reasoning behavior of large language models-a survey. ArXiv preprint, abs/2404.01869. · arXiv
  194. Behrad Moniri, Hamed Hassani, and Edgar Dobriban. 2024. Evaluating the performance of large language models via debates. ArXiv preprint, abs/2406.11044. · arXiv
  195. Wenhan Mu, Ling Xu, Shuren Pei, Le Mi, and Huichi Zhou. 2025. Evaluate-and-purify: Fortifying code language models against adversarial attacks using llm-as-a-judge. arXiv preprint arXiv:2504.19730. · arXiv
  196. Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Carina Suzana Negreanu, Chris Parnin, and Advait Sarkar. 2024. Evaluating the evaluator: Measuring llms' adher- ence to task evaluation instructions. ArXiv preprint, abs/2408.08781. · arXiv
  197. Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, and Arman Cohan. 2024. On evaluat- ing the integration of reasoning and action in LLM agents with database question answering. In Find- ings of the Association for Computational Linguis- tics: NAACL 2024, pages 4556-4579, Mexico City, Mexico. Association for Computational Linguistics.
  198. Ali Naseh and Niloofar Mireshghallah. 2025. Syn- thetic data can mislead evaluations: Membership inference as machine text detection. arXiv preprint arXiv:2501.11786. · arXiv
  199. Kun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao, Zhenhui Liu, Yu Wang, Ming Pang, and Li Yuan. 2024. Pico: Peer review in llms based on the consis- tency optimization.
  200. Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. arXiv preprint arXiv:2402.15987. · arXiv

200 of 386 shown

Cited by

Sorted by citations

Authors
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., Shu, K., Cheng, L., & Liu, H.
Affiliations
  • University of Maryland, Baltimore County (US)
  • University of Illinois Chicago (US)
  • Northwestern University (US)
  • Emory University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
2757-2791
DOI
10.48550/arxiv.2411.16594
arXiv
2411.16594
OpenAlex
W4416037109
Cited by
271
References
3
Citations by year
2026: 160 · 2025: 111
FWCI
1019.40
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2312862, CAREER, 2440542)
  • U.S. Department of Homeland Security (17STQAC00001-08-00)
  • Cisco Systems
  • National Institutes of Health
  • National Institute on Aging (R01AG091762)
Topics
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, LLM-as-a-judge, automated assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review