The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Can Large Language Models Be an Alternative to Human Evaluations?
Open accessAbstract
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans.However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Chiang and Lee, 2023a
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent efforts using LLMs as evaluators (LLM-as-a-judge (Zheng et al., 2023)) show parity with human judgment (Chiang and Lee, 2023b)”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Chiang and Lee, 2023b
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Chiang and Lee, 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation [51, 190, 314].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Research works printed but not linked: 3
- R Botsch. 2011. Chapter 12: Significance and mea- sures of association. Scopes and Methods of Political Science.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Cited by
Sorted by citations
Items 101–200 of 477
Priv-IQ: A Benchmark and Comparative Evaluation of Large Multimodal Models on Privacy Competencies
Shahriar, S., & Dara, R. A. (2025) · AI · doi:10.3390/ai6020029 · Cited by 20
Yao, Z., Zhang, Z., Tang, C., Bian, X., Zhao, Y., Yang, Z., Wang, J., Zhou, H., Jang, W. S., Ouyang, F., et al. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.292 · Cited by 49
Belcastro, L., Carlucci, C., Cosentino, C., Lió, P., & Marozzo, F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108160 · Cited by 20
GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks
Krechetova, V., & Kochedykov, D. (2025) · doi:10.1145/3764915.3770721 · Cited by 29
LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?
Zhao, F., Chen, J., Lim, L., Ahmad, I., Agrawal, D., & Abbadi, A. E. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401595 · Cited by 29
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16
Qin, H. X., Zhu, G., Fan, M., & Hui, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713569 · Cited by 16
Li, J., Li, X., Steinberg, J., Choube, A., Yao, B., Xu, X., Wang, D., Mynatt, E. D., & Mishra, V. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749508 · Cited by 28
A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
Arabzadeh, N., & Clarke, C. L. A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730159 · Cited by 30
Ayanwale, M. A., Adelana, O. P., Bamiro, N. B., Olatunbosun, S. O., Idowu, K. O., & Adewale, K. A. (2025) · F1000Research · doi:10.12688/f1000research.161637.1 · Cited by 15
Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
Kapania, S., Ballard, S. A., Kessler, A., & Vaughan, J. W. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732005 · Cited by 19
DataLab: A Unified Platform for LLM-Powered Business Intelligence
Weng, L., Tang, Y., Feng, Y., Chang, Z., Chen, R., Feng, H., Hou, C., Huang, D., Li, Y., Rao, H., et al. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00326 · Cited by 20
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Leiter, C., Opitz, J., Deutsch, D., Gao, Y., Dror, R., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.10 · Cited by 19
Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets
Moghe, N., Fazla, A., Amrhein, C., Kocmi, T., Steedman, M. J., Birch, A., Sennrich, R., & Guillou, L. (2024) · Computational Linguistics · doi:10.1162/coli_a_00537 · Cited by 15
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
Zhang, L. L., Wang, S., Jia, X., Zheng, Z., Yan, Y., Gao, L., Li, Y., & Xu, M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676382 · Cited by 15
Goodbye human annotators? Content analysis of social policy debates using ChatGPT
Gielens, E., Sowula, J., & Leifeld, P. (2025) · Journal of Social Policy · doi:10.1017/s0047279424000382 · Cited by 18
Towards Rationality in Language and Multimodal Agents: A Survey
Jiang, B., Xie, Y., Wang, X., Yuan, Y., Hao, Z., Bai, X., Su, W., Taylor, C. J., & Mallick, T. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.186 · Cited by 26
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References
Tang, T., Lu, H., Jiang, Y., Huang, H., Zhang, D., Zhao, X., Kocmi, T., & Wei, F. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.367 · Cited by 15
A scalable framework for evaluating health language models
Mallinar, N., Heydari, A. A., Liu, X., Faranesh, A. Z., Winslow, B. D., Hammerquist, N., Graef, B., Speed, C., Malhotra, M., Patel, S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02492-x · Cited by 23
Democratizing Reasoning Ability: Tailored Learning from Large Language Model
Wang, Z., Huang, S., Liu, Y., Wang, J., Song, M., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.120 · Cited by 22
GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
Hamid, A., Samidi, H. R., Pappachan, P., Finin, T., & Yus, R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0081 · Cited by 21
Prompt Engineering for Applied Linguistics: Elements, Examples, Techniques, and Strategies
Lin, Z. (2024) · English Language Teaching · doi:10.5539/elt.v17n9p14 · Cited by 14
Media coverage of climate activist groups in Germany
Dablander, F., Wimmer, S., & Haslbeck, J. M. B. (2025) · Climatic Change · doi:10.1007/s10584-025-03959-8 · Cited by 17
The Impostor is Among Us: Can Large Language Models Capture the Complexity of Human Personas?
Lazik, C. K., Katins, C., Kauter, C., Jakob, J., Jay, C., Grunske, L., & Kosch, T. (2025) · Mensch und Computer (MuC) · doi:10.1145/3743049.3743057 · Cited by 14
Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs
Upadhayay, B., & Behzadan, V. (2024) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2024.trustnlp-1.18 · Cited by 13
FunQA: Towards Surprising Video Comprehension
Xie, B., Zhang, S., Zhou, Z., Li, B., Zhang, Y., Hessel, J., Yang, J., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73232-4_3 · Cited by 13
Maity, S., Deroy, A., & Sarkar, S. (2024) · doi:10.36227/techrxiv.171665626.60610675/v1 · Cited by 14
Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context
Ruan, K., Wang, X., Hong, J., Wang, P., Liu, Y., & Sun, H. (2026) · Nature Communications · doi:10.1038/s41467-026-70245-1 · Cited by 50
CoDefeater: Using LLMs To Find Defeaters in Assurance Cases
Gohar, U., Hunter, M. C., Lutz, R. R., & Cohen, M. B. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695296 · Cited by 12
Assessing and Verifying Task Utility in LLM-Powered Applications
Arabzadeh, N., Huo, S., Mehta, N., Wu, Q., Wang, C., Awadallah, A. H., Clarke, C. L. A., & Kiseleva, J. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1219 · Cited by 23
“They are uncultured”: Unveiling Covert Harms and Social Threats in LLM Generated Conversations
Dammu, P. P. S., Jung, H., Singh, A. A. K., Choudhury, M., & Mitra, T. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1134 · Cited by 17
Xiong, Y., Tu, X., & Zhao, W. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104232 · Cited by 13
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Malaviya, C., Chang, J. C., Roth, D., Iyyer, M., Yatskar, M., & Lo, K. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.24 · Cited by 26
Revisiting Meta-evaluation for Grammatical Error Correction
Kobayashi, M., Mita, M., & Komachi, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00676 · Cited by 12
Lee, D., Hwang, Y., Kim, Y., Park, J., & Jung, K. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.452 · Cited by 11
Trustworthiness of Children Stories Generated by Large Language Models
Bhandari, P., & Brennan, H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.24 · Cited by 17
Ranking Large Language Models without Ground Truth
Dhurandhar, A., Nair, R., Singh, M., Daly, E., & Ramamurthy, K. N. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.143 · Cited by 11
Liu, Y., Wen, Z., Weng, L., Woodman, O., Yang, Y., & Chen, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3410523 · Cited by 15
On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments
Fang, J., Aréchiga, N., Namikoshi, K., Bravo, N. S., Hogan, C., & Shamma, D. A. (2024) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3652988.3673967 · Cited by 14
LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives
Shimabucoro, L., Ruder, S., Kreutzer, J., Fadaee, M., & Hooker, S. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.521 · Cited by 24
He, Z., Naphade, S., & Huang, T.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714319 · Cited by 10
Geyer, W., He, J., Sarkar, D., Brachman, M., Hammond, C., Heins, J., Ashktorab, Z., Rosemberg, C., & Hill, C. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729200 · Cited by 10
GPT-4 shows comparable performance to human examiners in ranking open-text answers
Zubaer, A. A., Granitzer, M., Geschwind, S., Lambsdorff, J. G., & Voss, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-21572-8 · Cited by 12
LLMChain: Blockchain-Based Reputation System for Sharing and Evaluating Large Language Models
Bouchiha, M. A., Telnoff, Q., Bakkali, S., Champagnat, R., Rabah, M., Coustaty, M., & Ghamri-Doudane, Y. M. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00067 · Cited by 10
Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation
Zhang, R., Ouni, J., & Eger, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00519 · Cited by 15
Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games
Shapira, E., Madmon, O., Reichart, R., & Tennenholtz, M. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21102 · Cited by 28
IdeaBench: Benchmarking Large Language Models for Research Idea Generation
Guo, S., Shariatmadari, A. H., Xiong, G., Huang, A., Kim, M., Williams, C. M., Bekiranov, S., & Zhang, A. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3737419 · Cited by 9
An Empirical Analysis on Large Language Models in Debate Evaluation
Liu, X., Liu, P., & He, H. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-short.44 · Cited by 20
Nonomura, R., & Mori, H. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1582287 · Cited by 15
Benchmarking Large Language Models on Homework Assessment in Circuit Analysis
Chen, L., Qin, Z., Guo, Y., Rohde, J. A., & Zhang, Y. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00501-w · Cited by 11
Large language models in human-robot collaboration: A systematic review, trends, and challenges
Yuan, G., Liu, X., Xiao, M., Xiao, J., & Wang, L. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.01.011 · Cited by 9
Akinseloyin, O., Jiang, X., & Palade, V. (2026) · Biology Methods and Protocols · doi:10.1093/biomethods/bpag006 · Cited by 10
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Shashidhar, S., Chinta, A., Sahai, V., Wang, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.608 · Cited by 9
Kuan, C.-Y., Yang, C.-K., Huang, W., Lu, K.-H., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832184 · Cited by 13
Self-citationTowards Better Evaluation of Instruction-Following: A Case-Study in Summarization
Skopek, O., Aralikatte, R., Gooding, S., & Cărbune, V. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.16 · Cited by 17
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.363 · Cited by 8
CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation
Bougie, N., & Watanabe, N. (2025) · arXiv · doi:10.48550/arxiv.2506.21805 · Cited by 8
Governing Open Vocabulary Data Leaks Using an Edge LLM through Programming by Example
Li, Q., Wen, J., & Jin, H. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699760 · Cited by 8
A Survey on Quality Evaluation of Instruction Fine-tuning Datasets for Large Language Models
Luo, Y., Liu, Y., Zhang, L., Gao, F., & Gu, J. (2025) · Data Intelligence · doi:10.3724/2096-7004.di.2025.0021 · Cited by 11
Emerging Data Practices: Data Work in the Era of Large Language Models
Garcia, A. A., Candello, H. C. D. S. P., Badillo-Urquiola, K. A., & Wong-Villacrés, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714069 · Cited by 15
Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation
Epure, E., Deldjoo, Y., Sguerra, B., Schedl, M., & Moussallam, M. (2026) · ACM Transactions on Recommender Systems · doi:10.1145/3844509 · Cited by 11
Grading scale impact on LLM-as-a-judge: human-LLM alignment is highest on 0–5 grading scale
Li, W., Zhao, M., Dong, W., Cai, J., Wei, Y., Pocress, M., Li, Y., Yuan, W., Wang, X., Hou, R., et al. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00153-8 · Cited by 8
What Matters in a Measure? A Perspective from Large-Scale Search Evaluation
Thomas, P., Kazai, G., Craswell, N., & Spielman, S. E. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657845 · Cited by 8
DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation
Kim, A., Kim, K., & Yoon, S. W. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.112 · Cited by 14
Zhang, F., Li, J., Li, Y.-C., Zhang, Z., Yu, Y., & Ye, D. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3590731 · Cited by 14
Tan, T. F., Elangovan, K., Ong, J. C. L., Lee, A., Shah, N. H., Sung, J. J. �., Wong, T. Y., Lan, X., Liu, N., Wang, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5029562 · Cited by 8
Enhancing memory retrieval in generative agents through LLM-trained cross attention networks
Hong, C., & He, Q. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1591618 · Cited by 7
Beyond Binary Classification: A Fine-Grained Safety Dataset for Large Language Models
Yu, J., Li, L., & Lan, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3393245 · Cited by 9
Taveekitworachai, P., Dewantoro, M. F., Xia, Y., Suntichaikul, P., & Thawonmas, R. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3529117 · Cited by 7
PaRT: Enhancing Proactive Social Chatbots with Personalized Real-Time Retrieval
Niu, Z., Xie, Z., Cao, S., Lu, C., Ye, Z., Xu, T., Liu, Z., Gao, Y. F., Chen, J., Xu, Z., et al. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3731946 · Cited by 10
Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains
Chu, X., Tan, Z., Xue, H., Wang, G., Mo, T., & Li, W. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.171 · Cited by 15
Njuguna, G. N., & Qingfei, M. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103080 · Cited by 7
An Automotive Fault Diagnosis Framework Based on Knowledge Graphs and Large Language Models
Lin, W., & Miao, K. (2025) · Electronics · doi:10.3390/electronics14214180 · Cited by 8
MCeT: Behavioral Model Correctness Evaluation using Large Language Models
Ahmed, K. A. M., Song, J., Chen, B., Ou, W., & Zheng, B. (2025) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models67397.2025.00014 · Cited by 7
Bianchi, O., Willey, M., Alvarado, C. X., Danek, B. P., Khani, M., Kuznetsov, N., Dadu, A., Shah, S. M. T., Koretsky, M. J., Makarious, M. B., et al. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100943 · Cited by 8
Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set
Kohli, G. S., Parida, S., Sekhar, S., Saha, S., Nair, N. B., Agarwal, P., Khosla, S., Patiyal, K., & Dhal, D. (2023) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3639856.3639890 · Cited by 12
Zhang, Y., Huang, B., Yuan, W., Jiang, Z., Peng, L., Chen, S., & Tan‐Soo, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104090 · Cited by 7
Enhancing Text Annotation Through Rationale-Driven Collaborative Few-Shot Prompting
Wu, J., Wang, X., & Weijia Jia, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889661 · Cited by 10
GeoFactory: an LLM performance enhancement framework for geoscience factual and inferential tasks
Chen, Z., Wang, X., Zhang, X., Lin, M., Liao, Y., Li, J., & Bai, Y. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2506291 · Cited by 8
Learning Evaluation Models From Large Language Models for Sequence Generation
Wang, C., Zhou, H., Chang, K., Liu, T., Zhang, C., Yang, M., Du, Q., Xiao, T., Zhang, Y., & Zhu, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587460 · Cited by 11
Automatic Legal Writing Evaluation of LLMs
Pires, R., Malaquias, R. D., & Nogueira, R. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769227 · Cited by 15
GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification TaskPreprint
Liyanage, C., Gokani, R., & Mago, V. (2023) · doi:10.36227/techrxiv.24143706.v1 · Cited by 7
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9
Optimizing context-based location extraction by tuning open-source LLMs with RAG
Wang, Z., Masri, Y., Malarvizhi, A. S., Stover, T., Ahmed, S., Wong, D. W. S., Jiang, Y., Li, Y., Bere, M., Rothbart, D., et al. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2521786 · Cited by 8
EvalAssist: Insights on Task-Specific Evaluations and AI-Assisted Judgment Strategy Preferences
Ashktorab, Z., Desmond, M. J., Pan, Q., Johnson, J. M., Cooper, M. S., Daly, E., Nair, R., Pedapati, T., Do, H. J., & Geyer, W. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747740 · Cited by 6
Lazik, C. K., Kauter, C., Nunes, I., Ziglowski, A., Pryma, A., Katins, C., Grunske, L., & Kosch, T. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00031 · Cited by 6
A Cost-Effective Framework to Evaluate LLM-Generated Relevance Judgements
Merlo, S., Marchesin, S., Faggioli, G., & Ferro, N. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761200 · Cited by 7
Aligning LLMs for the Classroom with Knowledge-Based Retrieval: A Comparative RAG Study
Jain, A. D., Cui, L., & Chen, S. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346682 · Cited by 8
LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Kucia, F. J., Chakraborty, A., & Wróblewska, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29918-5_38 · Cited by 8
Large Language Models as SocioTechnical Systems
Dhole, K. (2023) · doi:10.18653/v1/2023.bigpicture-1.6 · Cited by 7
Intelligent Agents for Data Exploration
Amer-Yahia, S. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3685800.3685913 · Cited by 6
A Novel Approach to Password Strength Evaluation Using ChatGPT-Based Prompt Metrics
Kim, S. J., & Lee, B. M. (2024) · IEEE Access · doi:10.1109/access.2024.3503653 · Cited by 6
Spoken Stereoset: on Evaluating Social Bias Toward Speaker in Speech Large Language Models
Lin, Y., Chen, W.-C., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832259 · Cited by 6
Self-citationSun, J., Shi, W., Shen, X., Liu, S., Wei, L., & Wan, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103037 · Cited by 9
Beyond Retrieval: Generating Narratives in Conversational Recommender Systems
Sayana, K., Vasudeva, R., Vasilevski, Y., Su, K., Hebert, L., Pine, J., Pham, H., Jash, A., & Sodhi, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717531 · Cited by 8
ScamFerret: Detecting Scam Websites Autonomously with Large Language Models
Nakano, H., Koide, T., & Chiba, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97620-9_1 · Cited by 8
Alipour, S., Sen, I., Samory, M., & Mitra, T. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1136 · Cited by 8
AI‐Generated Essays: Characteristics and Implications on Automated Scoring and Academic Integrity
Zhong, Y., Hao, J., Fauß, M., Li, C., & Wang, Y. (2026) · Educational Measurement Issues and Practice · doi:10.1111/emip.70013 · Cited by 10
Tu, S., Powers, A. D., Doogan, S., & Choi, J. D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag050 · Cited by 11
Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 7
Self-citation
Metadata
- Authors
- Chiang, C.-H., & Lee, H.-Y.
- Affiliations
- National Taiwan University (TW)
- Corresponding author
- Lee, Hung-yi
- ORCID
- Lee, Hung-yi
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 15607-15631
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571232
- Cited by
- 331
- References
- 40
- Citations by year
- 2026: 90 · 2025: 130 · 2024: 80 · 2023: 31
- FWCI
- 58.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Delta Electronics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, LLM evaluation, human evaluation, text quality assessment, adversarial attacks, assessment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
