Can Large Language Models Be an Alternative to Human Evaluations?
Open accessAbstract
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans.However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GPT-3.5 has a positive correlation with human annotations.
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Chiang and Lee, 2023a
Section 3.3.1 reports statistically significant positive correlations between text-davinci-003 ratings and English teachers’ ratings of stories. The copy does not identify text-davinci-003 as GPT-3.5, and the measured correlation is specific to those story ratings rather than human annotations generally. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLLM-as-a-judge evaluations show parity with human judgment.
“Recent efforts using LLMs as evaluators (LLM-as-a-judge (Zheng et al., 2023)) show parity with human judgment (Chiang and Lee, 2023b)”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Chiang and Lee, 2023b
Section 3.3.1 reports statistically significant but weak-to-strong positive correlations between LLM and English-teacher ratings of stories. Section 3.3 also says their absolute scores differ. The findings support some alignment in judgment, but not the broader claim of parity with human judgment. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLLM ratings of text quality track human ratings.
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Chiang and Lee, 2023
The remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation.
“Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation [51, 190, 314].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6311 · Cited by 1,108 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Generating Natural Language Adversarial Examples
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M. B., & Chang, K.-W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1316 · Cited by 895 · see citing works
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Ren, S., Deng, Y., He, K., & Che, W. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1103 · Cited by 666 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
BAE: BERT-based Adversarial Examples for Text Classification
Garg, S., & Ramakrishnan, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.498 · Cited by 518 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Huang, F., Kwak, H., & An, J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587368 · Cited by 247 · see citing works
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
Guan, J., Huang, F., Zhao, Z., Zhu, X., & Huang, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00302 · Cited by 230 · see citing works
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · see citing works
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180 · see citing works
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · see citing works
Reevaluating Adversarial Examples in Natural Language
Morris, J. X., Lifland, E., Lanchantin, J., Ji, Y., & Qi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.341 · Cited by 96 · see citing works
Non-expert evaluation of summarization systems is risky
Gillick, D., & Liu, Y. (2010) · Cited by 71 · see citing works
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
The evaluation of text quality: expert-focused and reader-focused methods compared
Lentz, L. R., & de Jong, M. D. T. (1997) · IEEE Transactions on Professional Communication · doi:10.1109/47.649557 · Cited by 51 · see citing works
Improving Neural Story Generation by Targeted Common Sense Grounding
Mao, H. H., Majumder, B. P., McAuley, J., & Cottrell, G. W. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1615 · Cited by 50 · see citing works
Rethinking the Agreement in Human Evaluation Tasks
Amidei, J., Piwek, P., & Willis, A. (2018) · Open Research Online (The Open University) · Cited by 40 · see citing works
Yoo, K., Kim, J., Jang, J., & Kwak, N. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.289 · Cited by 33 · see citing works
Agreement is overrated: A plea for correlation to assess human evaluation reliability
Amidei, J., Piwek, P., & Willis, A. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8642 · Cited by 27 · see citing works
How do Humans Evaluate Machine Translation
Guzmán, F., Abdelalí, A., Temnikova, I. P., Sajjad, H., & Vogel, S. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3059 · Cited by 15 · see citing works
BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text ClassificationPreprint
Hauser, J., Zhao, M., Pascual, D., & Wattenhofer, R. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07403 · Cited by 14 · see citing works
Are Synonym Substitution Attacks Really Synonym Substitution Attacks?
Chiang, C.-H., & Lee, H.-Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.117 · Cited by 10 · see citing works
Self-citationBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Aligning AI With Shared Human Values
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2008.02275
Flagged first, then by citations
Research works printed but not linked: 3
- R Botsch. 2011. Chapter 12: Significance and mea- sures of association. Scopes and Methods of Political Science.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 429 · see citing works
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201 · see citing works
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
Jiang, H., Zhang, X., Cao, X., Breazeal, C., Roy, D. K., & Kabbara, J. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.229 · Cited by 201 · see citing works
Wang, A., Morgenstern, J., & Dickerson, J. P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00986-z · Cited by 192 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Large Language Models can Accurately Predict Searcher Preferences
Thomas, P., Spielman, S., Craswell, N., & Mitra, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657707 · Cited by 168 · see citing works
Gaber, F., Shaik, M., Allega, F., Bilecz, A. J., Busch, F., Goon, K., Franke, V., & Akalin, A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01684-1 · Cited by 165 · see citing works
Dennstädt, F., Zink, J., Putora, P. M., Hastings, J., & Čihorić, N. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02575-4 · Cited by 165 · see citing works
LLM-based NLG Evaluation: Current Status and Challenges
Gao, M., Hu, X., Yin, X., Ruan, J., Pu, X., & Wan, X. (2025) · Computational Linguistics · doi:10.1162/coli_a_00561 · Cited by 155 · see citing works
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.669 · Cited by 152 · see citing works
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
Lu, W., Luu, R. K., & Buehler, M. J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01564-y · Cited by 126 · see citing works
Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models
Chen, Y., Fu, Q., Yuan, Y., Wen, Z., Fan, G., Liu, D., Zhang, D., Li, Z., & Xiao, Y. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614905 · Cited by 103 · see citing works
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y., & Kim, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642216 · Cited by 102 · see citing works
Cheng, L., Li, X., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.637 · Cited by 100 · see citing works
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94 · see citing works
Exploring the frontiers of LLMs in psychological applications: a comprehensive review
Ke, L., Tong, S., Cheng, P., & Peng, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11297-5 · Cited by 92 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 88 · see citing works
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.969 · Cited by 87 · see citing works
Laskar, M. T. R., Alqahtani, S., Bari, M. S., Rahman, M., Khan, M. A. M., Khan, H., Jahan, I., Bhuiyan, A., Tan, C. W., Parvez, M. R., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.764 · Cited by 87 · see citing works
Honeybee: Locality-Enhanced Projector for Multimodal LLM
Cha, J., Kang, W., Mun, J., & Roh, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01311 · Cited by 88 · see citing works
Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706527 · Cited by 84 · see citing works
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Yuan, M., Bao, P., Yuan, J., Shen, Y., Chen, Z., Xie, Y., Zhao, J., Li, Q., Chen, Y., Zhang, L., et al. (2024) · Medicine Plus · doi:10.1016/j.medp.2024.100030 · Cited by 82 · see citing works
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79 · see citing works
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 71 · see citing works
Optimization-based Prompt Injection Attack to LLM-as-a-Judge
Shi, J., Yuan, Z., Liu, Y., Huang, Y., Zhou, P., Sun, L., & Gong, N. Z. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3690291 · Cited by 70 · see citing works
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Al-Rashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70879-4_6 · Cited by 70 · see citing works
The Illusion of Artificial Inclusion
Agnew, W. S., Bergman, A. S., Chien, J., Díaz, M., El-Sayed, S., Pittman, J., Mohamed, S., & McKee, K. R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642703 · Cited by 69 · see citing works
Wang, Y., Luo, H., & Fang, W. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103246 · Cited by 69 · see citing works
Knowledge graph validation by integrating LLMs and human-in-the-loop
Tsaneva, S., Dessı̀, D., Osborne, F., & Sabou, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104145 · Cited by 69 · see citing works
LEVA: Using Large Language Models to Enhance Visual Analytics
Zhao, Y., Zhang, Y., Zhang, Y., Zhao, X., Wang, J., Shao, Z., Turkay, Ç., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3368060 · Cited by 65 · see citing works
QA-RAG: Exploring LLM Reliance on External Knowledge
Mansurova, A., Mansurova, A., & Nugumanova, A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8090115 · Cited by 63 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
A Closer Look into Using Large Language Models for Automatic Evaluation
Chiang, C.-H., & Lee, H.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.599 · Cited by 59 · see citing works
Self-citationThe simulation of judgment in LLMs
Loru, E., Nudo, J., Di Marco, N., Santirocchi, A., Atzeni, R., Cinelli, M., Cestari, V., Rossi-Arnaud, C., & Quattrociocchi, W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2518443122 · Cited by 57 · see citing works
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 52 · see citing works
Sottana, A., Liang, B., Zou, K., & Yuan, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.543 · Cited by 51 · see citing works
Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy
Seo, H., Hwang, T., Jung, J., Kang, H., Namgoong, H., Lee, Y. H., & Jung, S. (2025) · Applied Sciences · doi:10.3390/app15020671 · Cited by 51 · see citing works
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 51 · see citing works
Large language model for knowledge synthesis and AI-enhanced biomanufacturing
Li, W., Mao, Z., Xiao, Z., Liao, X., Koffas, M. A. G., Chen, Y., Ma, H., & Tang, Y. (2025) · Trends in biotechnology · doi:10.1016/j.tibtech.2025.02.008 · Cited by 48 · see citing works
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47 · see citing works
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 47 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 47 · see citing works
The Next Chapter: A Study of Large Language Models in Storytelling
Xie, Z., Cohn, T., & Lau, J. H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.23 · Cited by 41 · see citing works
RecExplainer: Aligning Large Language Models for Explaining Recommendation Models
Lei, Y., Lian, J., Yao, J., Huang, X., Lian, D., & Xie, X. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671802 · Cited by 41 · see citing works
Semantic Sleuth: Identifying Ponzi Contracts via Large Language Models
Wu, C., Chen, J., Wang, Z., Liang, R., & Du, R. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695055 · Cited by 41 · see citing works
VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction
Gorniak, J., Kim, Y., Wei, D., & Kim, N. W. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676414 · Cited by 40 · see citing works
DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows
Patel, A., Raffel, C., & Callison-Burch, C. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.208 · Cited by 39 · see citing works
LAMD: Context-Driven Android Malware Detection and Classification with LLMs
Qian, X., Zheng, X., He, Y., Yang, S., & Cavallaro, L. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00017 · Cited by 39 · see citing works
Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025) · Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2025.ijcnlp-long.18 · Cited by 39 · see citing works
López-Gazpio, I. (2025) · Information · doi:10.3390/info16060473 · Cited by 38 · see citing works
Bevilacqua, M., Oketch, K., Qin, R., Stamey, W., Zhang, X., Gan, Y., Yang, K., & Abbasi, A. (2024) · ACM Transactions on Information Systems · doi:10.1145/3702639 · Cited by 37 · see citing works
Simulacrum of Stories: Examining Large Language Models as Qualitative Research Participants
Kapania, S., Agnew, W. S., Eslami, M., Heidari, H., & Fox, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713220 · Cited by 37 · see citing works
Weng, L., Wang, X., Lu, J., Feng, Y., Liu, Y., Feng, H., Huang, D., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3567131 · Cited by 37 · see citing works
Cognitive Agents Powered by Large Language Models for Agile Software Project Management
Cinkusz, K., Chudziak, J. A., & Niewiadomska‐Szynkiewicz, E. (2024) · Electronics · doi:10.3390/electronics14010087 · Cited by 36 · see citing works
Hadi, M. U., Tashi, Q. A., Qureshi, R., Shah, A., Muneer, A., Irfan, M. I., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2025) · doi:10.36227/techrxiv.23589741.v8 · Cited by 36 · see citing works
Synthetic Test Collections for Retrieval Evaluation
Rahmani, H. A., Craswell, N., Yılmaz, E., Mitra, B., & Campos, D. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657942 · Cited by 34 · see citing works
Improving the Validity of Automatically Generated Feedback via Reinforcement Learning
Scarlatos, A., Smith, D., Woodhead, S., & Lan, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64302-6_20 · Cited by 32 · see citing works
Kholodna, N., Julka, S., Khodadadi, M., Gumus, M. N., & Granitzer, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70381-2_25 · Cited by 32 · see citing works
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28 · see citing works
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 28 · see citing works
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI
Pangakis, N., & Wolken, S. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35883 · Cited by 27 · see citing works
Advancing health coaching: A comparative study of large language model and health coaches
Ong, Q. C., Ang, C., Chee, D. Z. Y., Lawate, A., Sundram, F., Dalakoti, M., Pasalic, L., To, D., Fox, T. E., Bojić, I., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103004 · Cited by 27 · see citing works
Which is better? Exploring Prompting Strategy For LLM-based Metrics
Kim, J., Lee, S., Han, S. H., Park, S.-Y., Lee, J., Jeong, K., & Kang, P. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.14 · Cited by 26 · see citing works
METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Hyun, S., Guo, M., & Babar, M. A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00019 · Cited by 26 · see citing works
Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
Palo, F. D., Singhi, P., & Fadlallah, B. H. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.215 · Cited by 26 · see citing works
Can large language models meet the challenge of generating school-level questions?
Maity, S., Deroy, A., & Sarkar, S. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100370 · Cited by 26 · see citing works
FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models
Li, Y., Liu, X., Wang, X., Lee, B., Wang, S., Rocha, A. D. R., & Lin, W. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3597211 · Cited by 26 · see citing works
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI
Ladapo, O. O., Jooda, D., Dosunmu, A. A., & Abolaji, T. O. (2024) · International Journal of Multidisciplinary Futuristic Development · doi:10.54660/ijmfd.2024.5.1.81-95 · Cited by 26 · see citing works
Leveraging foundation and large language models in medical artificial intelligence
Wong, I. N., Monteiro, O., Baptista‐Hon, D. T., Wang, K., Lu, W.-Y., Sun, Z., Nie, S., & Yin, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003302 · Cited by 25 · see citing works
Hong, J., Seto, C., Fan, A., & Maciejewski, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3536358 · Cited by 25 · see citing works
Detecting mental disorder on social media: A ChatGPT-augmented explainable approach
Belcastro, L., Cantini, R., Marozzo, F., Talia, D., & Paolo Trunfio, A. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100321 · Cited by 24 · see citing works
AutoAD III: The Prequel - Back to the Pixels
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01720 · Cited by 24 · see citing works
Leveraging large language models for thematic analysis: a case study in the charity sector
Wen, C., Clough, P., Paton, R., & Middleton, R. (2025) · AI & Society · doi:10.1007/s00146-025-02487-4 · Cited by 23 · see citing works
Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
Chen, B., Wang, G., Guo, H., Wang, Y., & Yan, Q. (2023) · International Symposium on Research in Attacks, Intrusions and Defenses (RAID) · doi:10.1145/3607199.3607237 · Cited by 22 · see citing works
MARKMyWORDS: Analyzing and Evaluating Language Model Watermarks
Piet, J., Sitawarin, C., Fang, V., Mu, N., & Wagner, D. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00012 · Cited by 22 · see citing works
GPT-4 as an X data annotator: Unraveling its performance on a stance classification task
Liyanage, C. R., Gokani, R., & Mago, V. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307741 · Cited by 21 · see citing works
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance
Xu, J., Wei, T., Hou, B., Orzechowski, P., Yang, S., Jin, R., Paulbeck, R., Wagenaar, J., Demiris, G., & Shen, L. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737393 · Cited by 21 · see citing works
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges
Ahsan, H., McInerney, D. J., Kim, J., Potter, C., Young, G. S., Amir, S., & Wallace, B. (2023) · PubMed · doi:10.48550/arxiv.2309.04550 · Cited by 21 · see citing works
Are Large Language Models Reliable Argument Quality Annotators?
Mirzakhmedova, N., Gohsen, M., Chang, C. H., & Stein, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63536-6_8 · Cited by 21 · see citing works
Causality-Aided Evaluation and Explanation of Large Language Model-Based Code Generation
Ji, Z., Ma, P., Li, Z., Wang, Z., & Wang, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728938 · Cited by 21 · see citing works
Li, D., Zhao, Y. C., Wang, Z., Jung, C., & Zhang, Z. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13110405 · Cited by 20 · see citing works
Strengths and Weaknesses of LLM-Based and Rule-Based NLP Technologies and Their Potential Synergies
Karanikolas, N. Ν., Manga, E., Samaridi, N. E., Stergiopoulos, V., Tousidou, E., & Vassilakopoulos, M. G. (2025) · Electronics · doi:10.3390/electronics14153064 · Cited by 20 · see citing works
Can Large Language Models Serve as Evaluators for Code Summarization?
Wu, Y., Wan, Y., Chu, Z., Zhao, W., Liu, Y., Zhang, H., Shi, X., Jin, H., & Yu, P. S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3595283 · Cited by 20 · see citing works
SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation
Bougie, N., & Watanabe, N. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-industry.5 · Cited by 19 · see citing works
To eat or to feed : can large language models provide useful feedback in translation education?
Jiao, H., Hu, W. Q., & Xiao-jun, Z. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2533074 · Cited by 19 · see citing works
Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?
Kim, S., & Oh, D. (2025) · Applied Sciences · doi:10.3390/app15062971 · Cited by 18 · see citing works
KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models
Yu, Z., Gao, C., Yao, W., Wang, Y., Ye, W., Wang, J., Xie, X., Zhang, Y., & Zhang, S. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.325 · Cited by 18 · see citing works
ReZG: Retrieval-augmented zero-shot counter narrative generation for hate speech
Jiang, S., Tang, W., Chen, X., Tang, R., Wang, W., & Wang, W. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129140 · Cited by 18 · see citing works
Priv-IQ: A Benchmark and Comparative Evaluation of Large Multimodal Models on Privacy Competencies
Shahriar, S., & Dara, R. A. (2025) · AI · doi:10.3390/ai6020029 · Cited by 18 · see citing works
Yao, Z., Zhang, Z., Tang, C., Bian, X., Zhao, Y., Yang, Z., Wang, J., Zhou, H., Jang, W. S., Ouyang, F., et al. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.292 · Cited by 18 · see citing works
Belcastro, L., Carlucci, C., Cosentino, C., Lió, P., & Marozzo, F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108160 · Cited by 17 · see citing works
GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks
Krechetova, V., & Kochedykov, D. (2025) · doi:10.1145/3764915.3770721 · Cited by 17 · see citing works
LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?
Zhao, F., Chen, J., Lim, L., Ahmad, I., Agrawal, D., & Abbadi, A. E. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401595 · Cited by 17 · see citing works
- Authors
- Chiang, C.-H., & Lee, H.-Y.
- Affiliations
- National Taiwan University (TW)
- Corresponding author
- Lee, Hung-yi
- ORCID
- Lee, Hung-yi
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 15607-15631
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571232
- Cited by
- 331
- References
- 40
- Citations by year
- 2026: 90 · 2025: 130 · 2024: 80 · 2023: 31
- FWCI
- 58.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Delta Electronics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, LLM evaluation, human evaluation, text quality assessment, adversarial attacks, assessment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
