The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Can Large Language Models Be an Alternative to Human Evaluations?
Open accessAbstract
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans.However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Chiang and Lee, 2023a
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent efforts using LLMs as evaluators (LLM-as-a-judge (Zheng et al., 2023)) show parity with human judgment (Chiang and Lee, 2023b)”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Chiang and Lee, 2023b
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Chiang and Lee, 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation [51, 190, 314].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Research works printed but not linked: 3
- R Botsch. 2011. Chapter 12: Significance and mea- sures of association. Scopes and Methods of Political Science.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Cited by
Sorted by citations
Items 201–300 of 477
Dolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 8
DeltaScore: Fine-Grained Story Evaluation with Perturbations
Xie, Z., Li, M., Cohn, T., & Lau, J. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.353 · Cited by 14
Brake, N., & Schaaf, T. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.25 · Cited by 6
TAKAGI, H., Moriya, S., Sato, T., Nagao, M., & Higuchi, K. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712119 · Cited by 6
Factuality Guided Diffusion-Based Abstractive Summarization
Shin, J., Park, H., & Song, H.-J. (2024) · IEEE Access · doi:10.1109/access.2024.3467063 · Cited by 5
Liu, H., García, M. B., & Korkakakis, N. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00064 · Cited by 5
CluSanT: Differentially Private and Semantically Coherent Text Sanitization
Awon, A. M., Lu, Y., Potka, S., & Thomo, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.187 · Cited by 5
An Empirical Study of Evaluating Long-form Question Answering
Xian, N., Fan, Y., Zhang, R., de Rijke, M., & Guo, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729895 · Cited by 5
LLM-Based Automated Generation and Tri-Modal Representation of Cyber Attack Scenario
Roh, S.-H., & Kim, T.-S. (2025) · IEEE Access · doi:10.1109/access.2025.3597850 · Cited by 5
Patel, K., Lin, Y.-Z., Raul, G., Shih, B. P.-J., Redondo, M. W., Latibari, B. S., Pacheco, J., Salehi, S., & Satam, P. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328318 · Cited by 5
Zhang, H., Song, J., Zhu, J., Wu, Y., Zhang, T., & Niu, C. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.553 · Cited by 15
Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendation
Shimizu, R., Wada, T., Wang, Y., Kruse, J., O’Brien, S., HtaungKham, S., Song, L., Yoshikawa, Y., Saito, Y., Tsung, F., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714583 · Cited by 5
FutureGen: A RAG-based Approach to Generate the Future Work of Scientific Article
Azher, I. A., Mokarrama, M. J., Guo, Z., Choudhury, S. R., & Alhoori, H. (2025) · IEEE International Conference on eScience · doi:10.1109/escience65000.2025.00087 · Cited by 4
FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models
Yu, Z., Gao, C., Yao, W., Wang, Y., Zeng, Z., Ye, W., Wang, J., Zhang, Y., & Zhang, S. (2024) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2024.emnlp-demo.1 · Cited by 5
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4
HSG-RAG: Hierarchical Knowledge Base Construction for Embedded System Development
Lu, Z., Xu, H., Chen, A. Q., Tang, S., Zhang, J., Feng, Y., Pan, W., & Huang, J. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3731680 · Cited by 4
How Do Hackathons Foster Creativity? Towards Automated Evaluation of Creativity at Scale
Falk, J., Chen, Y., Rafner, J. F., Zhang, M., Bjerva, J., & Nolte, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713447 · Cited by 6
Megahed, F. M., Chen, Y.-J., Jones‐Farmer, L. A., Lee, Y., Wang, J., & Zwetsloot, I. M. (2026) · Journal of Business Analytics · doi:10.1080/2573234x.2026.2652281 · Cited by 6
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4
Gender inclusive language generation framework: A reasoning approach with RAG and CoT
Chinnan, S. P. M., Drury‐Grogan, M. L., & Chakravarthi, B. R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114092 · Cited by 4
Núñez‐Regueiro, F., Falcón, S., & Bressoux, P. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13764-6 · Cited by 5
Sucal, V., Jullien, M., Njifenjou, A., & Lefèvre, F. (2025) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3765766.3765797 · Cited by 4
SUVA: A Probabilistic Framework for Auditing LLMs with an Application to Social Preferences
Leng, Y., & Yuan, Y. (2026) · Information Systems Research · doi:10.1287/isre.2024.0857 · Cited by 4
Revising with AI: aligning feedback technologies with learners’ revision processes
Helm, G., & Hesse, F. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1702406 · Cited by 4
A Human-in/on-the-Loop Framework for Accessible Text Generation
Moreno, L., & Martínez, P. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2gtngp2nmx63 · Cited by 4
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
Li, M., Li, H., & Tan, C. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1963 · Cited by 5
Large language models in manufacturing: a comprehensive review
Maghanaki, M., Shahin, M., & Chen, F. F. (2026) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-026-18711-4 · Cited by 4
Li, K., Qian, C., & Yang, X. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12851-4 · Cited by 6
Leveraging LLM generated labels to reduce bad matches in job recommendations
Pei, Y., Pang, Y. W., Cai, W., Sengupta, N., & Toshniwal, D. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688043 · Cited by 4
Harmful Prompt Classification for Large Language Models
Gupta, O., Lozano, M., Busalim, A. H., Jaiswal, R. R., & Quille, K. (2024) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3701268.3701271 · Cited by 4
Atalan, A., Keskin, A., & Özer, S. (2024) · International Journal of Healthcare Management · doi:10.1080/20479700.2024.2444102 · Cited by 4
SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions
Wang, Y., & Xu, H. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00046 · Cited by 5
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Shaaban, M. A., Saleem, T. J., Papineni, V. R. K., & Yaqub, M. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04978-0_44 · Cited by 6
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Fang, B., Dai, X., & Karimi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.578 · Cited by 4
Embracing Foundation Models for Advancing Scientific Discovery
Guo, S., Shariatmadari, A. H., Xiong, G., & Zhang, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825618 · Cited by 5
Large language model AI-guided creative writing co-creation in secondary schools
Spitsberg, T., Kettler, T., & McKamie, J. (2025) · Theory Into Practice · doi:10.1080/00405841.2025.2528551 · Cited by 3
VAP3: Variation-Aware Prompt Performance Prediction
Arabzadeh, N., & Ebrahim Bagheri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730264 · Cited by 3
PhoniTale: Phonologically Grounded Mnemonic Generation for Typologically Distant Language Pairs
Kang, S., Gwon, M., Kwon, S., Lee, J., Lan, A., Raj, B., & Singh, R. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1299 · Cited by 3
A Multi-LLM Pipeline for Retrieval-Grounded, Bloom’s Taxonomy-Aligned Question Generation
Abdullah, M., Mansoor, I., Rey, V. F., & Fraz, M. M. (2025) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt269104.2025.11360748 · Cited by 3
Systematic validation of LLM-generated structured data — A design space and remaining challenges
Sachdeva, M., Narayanan, C., Wiedenkeller, M., Sedláková, J., & Bernard, J. (2026) · Computers & Graphics · doi:10.1016/j.cag.2026.104545 · Cited by 3
Evaluating AI-assisted customer verification for synthetic nucleic acid screening
Acelas, A., Pálya, H., Flyangolts, K., Fady, P.-E., & Nelson, C. (2026) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2026.1819556 · Cited by 3
Azad, S. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-8129-8_10 · Cited by 3
Li, J., Hu, J., & Zhang, G. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.050005 · Cited by 3
LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval
Rahmani, H. A., Siro, C., Aliannejadi, M., Craswell, N., Clarke, C. L. A., Faggioli, G., Mitra, B., Thomas, P., & Yılmaz, E. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3705706 · Cited by 3
NarrAD: Automatic Generation of Audio Descriptions for Movies with Rich Narrative Context
Park, J., Ye, J., Lee, S., Ka, H. W., & Han, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00050 · Cited by 3
Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrieval
Li, Y., Eustratiadis, P., Lupart, S., & Kanoulas, E. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730110 · Cited by 3
Signé, Q., Boughanem, M., Moreno, J. G., & Belkacem, T. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744624 · Cited by 6
Annotation-Free Human Sketch Quality Assessment
Yang, L., Pang, K., Zhang, H., & Song, Y.-Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02001-1 · Cited by 3
Kotonya, N., Krishnasamy, S., Tetreault, J., & Jaimes, A. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.17 · Cited by 6
Student Answer Script Evaluation Using Large Language Model
Joseph, H. M., Aishwarya, S., Sriga, & Khumar, H. (2024) · International Conference on Electronics, Computing, Communication and Control Technology (ICECCC) · doi:10.1109/iceccc61767.2024.10593995 · Cited by 3
Ma, Z., Pan, M., Hou, Y., Yang, L., & Wang, W. W. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3484460 · Cited by 3
Wu, H. R., Hong, H., Sun, L., Bai, X., & Pu, M. (2024) · Electronics · doi:10.3390/electronics13234581 · Cited by 4
Beyond Labels: Aligning Large Language Models with Human-Like Reasoning
Kabir, M. R., Sultan, R. M., Asif, I. H., Ahad, J. I., Rahman, F., Amin, M. R., Mohammed, N., & Rahman, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78172-8_16 · Cited by 7
GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification TaskPreprint
Liyanage, C., Gokani, R., & Mago, V. (2023) · doi:10.36227/techrxiv.24143706 · Cited by 3
Scaling Point-in-Time Language Models
Kelly, B., Malamud, S., Schwab, J., & Xu, T. A. (2026) · National Bureau of Economic Research · doi:10.3386/w35247 · Cited by 4
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2
A Comparative Analysis of GPT-3.5, GPT-4 and GPT-4.o in Heart Failure
Günay, Ş., & Sığırlı, D. (2024) · Uludağ Üniversitesi Tıp Fakültesi Dergisi · doi:10.32708/uutfd.1543370 · Cited by 2
Huang, C.-Y., Shih, M.-H., Lu, K.-H., Hsiao, C.-Y., & Lee, H.-Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889205 · Cited by 2
Self-citationWu, H. R., Hong, H., Mao, J., Yin, Z., Wu, Y., Bai, X., Sun, L., Pu, M., Liu, J., & Li, Y. (2025) · Applied Sciences · doi:10.3390/app15095041 · Cited by 2
Li, S., Li, J., Qu, Y., Shi, X., Guo, Y., He, Z., Wang, Y., & Tan, W. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.477 · Cited by 2
Baazeem, I. S., Al-Khalifa, H. S., & Al‐Salman, A. S. (2025) · Computation · doi:10.3390/computation13110258 · Cited by 2
Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Alymova, Y. A., Альфредовна, А. Д., Федоровна, Ш. Ю., Kodenko, M. R., Blokhin, I. A., & Reshetnikov, R. V. (2025) · AI · doi:10.3390/ai6110287 · Cited by 3
Enhancement Large Language Models Domain Through Ontology-Based Retrieval-Augmented Generation
Clarizia, F., Santo, M. D., Gaeta, R., & Loffredo, R. (2025) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.392507 · Cited by 2
LangPTune: Optimizing Language-based User Profiles for Recommendation
Gao, Z., Zhou, J., Dai, Y., & Joachims, T. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761369 · Cited by 6
BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents
Choi, Y., Kim, E., Kim, H., Park, D., Lee, H., Kim, J. Y., & Kim, J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747677 · Cited by 6
Self-citationReasoning About Human Values in GitHub Issues: What Can a Large Language Model Reveal?
Mougouei, D., Haque, A., Nurwidyantoro, A., Mougouei, E., Fahmideh, M., Shahin, M., Dharmawan, A., Widodo, T. W., & Dam, H. K. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2594137 · Cited by 2
Pan, Y., Duan, C., Du, J., Zhang, J., Du, K., Zhang, C., Liu, Z., Zhang, W., Wang, B., Ren, Y., et al. (2026) · Annals of Surgical Oncology · doi:10.1245/s10434-026-19176-1 · Cited by 2
Chen, N., Tang, L., Liu, Y., Lin, C., Li, Z., Shi, C., Shi, C., Xia, M., Gašević, D., Gašević, D., et al. (2026) · JMIR Medical Education · doi:10.2196/84347 · Cited by 2
Knowledge-Driven Generative Design of Role-Playing Game Scenarios
Owczarek, W., Wróbel, J., & Pęszor, D. (2026) · Applied Sciences · doi:10.3390/app16062966 · Cited by 2
Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios
Lee, S., Kim, H. S., & Kim, H. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i45.41226 · Cited by 2
Fu, Y., Li, H., You, M., Wang, L., Liu, W., Zhou, K., Wang, L., Chen, X., & Chen, G. (2026) · Healthcare · doi:10.3390/healthcare14152389 · Cited by 2
Zeng, X., Liu, Y., Meng, F., & Zhou, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.710 · Cited by 2
Berger, M., & Liu, S. (2024) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv64461.2024.00012 · Cited by 2
Explain further: multi-level explanations for fake news detection using large language models
Sallami, D., & Aı̈meur, E. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02640-8 · Cited by 2
Design of Photovoltaic Cleaning Robots Assisted by Large Language Models
Chai, S., Ying, W., & Wan, R. (2023) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid59865.2023.00054 · Cited by 2
Aligning Large Language Models via Fine-grained Supervision
Xu, D., Qiu, L., Kim, M., Ladhak, F., & Do, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-short.62 · Cited by 7
Human Evaluation in Large Language Model Testing
Dharmendra, H., Raghunandan, G. C., Sindhu, A. N., Samanvitha, C., Nethravathi, N., & Elango, D. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-5380-6.ch022 · Cited by 2
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset
Chehbouni, K., Carr, J. C., More, Y., Cheung, J. C., & Farnadi, G. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.596 · Cited by 5
DCA-Bench: A Benchmark for Dataset Curation Agents
Huang, B., Yu, Y., Huang, J., Zhang, X., & Ma, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737422 · Cited by 8
Moon, H., Lee, J., Eo, S., Park, C., Seo, J., & Lim, H. (2024) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2024.findings-eacl.145 · Cited by 2
DeepTagger: Knowledge Enhanced Named Entity Recognition for Web-Based Ads Queries
Zuo, S., Tang, P., Hu, X., Lou, Q., Jiao, J., & Charles, D. X. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615467 · Cited by 2
Paek, E. S., Fan, T., Finch, J. D., & Choi, J. D. (2024) · Information · doi:10.3390/info15090580 · Cited by 2
Distilling Event Sequence Knowledge From Large Language Models
Wadhwa, S., Hassanzadeh, O., Bhattacharjya, D., Barker, K., & Ni, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77844-5_13 · Cited by 2
Evaluating Large Language Models in Psychological Research: A Guide for ReviewersPreprint
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ag7hy · Cited by 3
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Lee, K.-H., Chen, X., Furuta, H., Canny, J., & Fischer, I. (2024) · arXiv · doi:10.48550/arxiv.2402.09727 · Cited by 1
From Precision to Perception: User Surveys in the Evaluation of Keyword Extraction Algorithms
Cai, J., Leckner, S., & Björklund, J. (2025) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3736733.3736741 · Cited by 1
Anemeta, D. D., & Rzepka, R. (2025) · Algorithms · doi:10.3390/a18080507 · Cited by 1
Mandapati, S. V., Kummetha, V. C., Concas, S., & Staes, L. (2025) · Electronics · doi:10.3390/electronics15010152 · Cited by 1
A White-Box Benchmark for Intrinsic Hallucination Detection in Turkish LLMs
Bayramoglu, B., Ficici, M., Bildirici, F., Kızıltepe, R. S., & Karakuş, M. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418463 · Cited by 1
Xiang, Y., Lu, J., Wei, J., & Hu, Y. (2026) · Mathematics · doi:10.3390/math14060950 · Cited by 1
Similarity-Based Prompt Construction for Large Language Model in Medical Tasks
Liu, G., Pan, M., Ma, Z., Gu, M., Yang, L., & Qin, J. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1717-0_6 · Cited by 1
HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants
Gritta, M., Λάμπουρας, Γ., & Iacobacci, I. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.456 · Cited by 1
Revolutionizing Wireless Modeling and Simulation with Network-Oriented LLMs
Liu, J., Peng, Z., Xu, D., & Liu, Y. (2024) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/ipccc59868.2024.10850042 · Cited by 1
Graded Suspiciousness of Adversarial Texts to Humans
Tonni, S. M., Faustini, P., & Dras, M. (2025) · Computational Linguistics · doi:10.1162/coli_a_00555 · Cited by 1
MoDeST: A dataset for Multi Domain Scientific Title Generation
Bölücü, N., Bilge, Y. C., Çetintaş, D., & Yücel, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113557 · Cited by 2
LLM-Human Pipeline for Cultural Grounding of Conversations
Pujari, R., & Goldwasser, D. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.48 · Cited by 2
Likelihood-based Mitigation of Evaluation Bias in Large Language Models
Ohi, M., Kaneko, M., Koike, R., Loem, M., & Okazaki, N. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.480 · Cited by 1
Multi-Facet Blending for Faceted Query-by-Example Retrieval
Do, H., Ryu, S., Kim, J., & Lee, G. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1388 · Cited by 1
Enhancing Vietnamese Automatic Question Generation with Synthetic Data and Preference Optimization
Nguyen, D. H., Tran, C., & Tran, H. T. (2025) · doi:10.1109/icnlp65360.2025.11108542 · Cited by 1
Gazzo, A. M., Berardelli, S., Biancospino, M., Cuollo, L., Zotti, F. D., Ferraro, E., Marra, A., Tartarotti, E., & Magni, P. (2026) · Genome biology · doi:10.1186/s13059-026-04237-1 · Cited by 1
Metadata
- Authors
- Chiang, C.-H., & Lee, H.-Y.
- Affiliations
- National Taiwan University (TW)
- Corresponding author
- Lee, Hung-yi
- ORCID
- Lee, Hung-yi
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 15607-15631
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571232
- Cited by
- 331
- References
- 40
- Citations by year
- 2026: 90 · 2025: 130 · 2024: 80 · 2023: 31
- FWCI
- 58.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Delta Electronics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, LLM evaluation, human evaluation, text quality assessment, adversarial attacks, assessment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
