The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Can Large Language Models Be an Alternative to Human Evaluations?
Open accessAbstract
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans.However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Chiang and Lee, 2023a
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent efforts using LLMs as evaluators (LLM-as-a-judge (Zheng et al., 2023)) show parity with human judgment (Chiang and Lee, 2023b)”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Chiang and Lee, 2023b
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Chiang and Lee, 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation [51, 190, 314].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Research works printed but not linked: 3
- R Botsch. 2011. Chapter 12: Significance and mea- sures of association. Scopes and Methods of Political Science.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Cited by
Sorted by citations
Items 301–400 of 477
From Reviews to Results: Generative AI for Review-Driven Product and Service Comparisons
Cosentino, C., Gündüz-Cüre, M., Marozzo, F., & Birim, Ş. Ö. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05461-6_6 · Cited by 1
InfAL: Inference Time Adversarial Learning for Improving Research Ideation
Guo, S., Shariatmadari, A. H., Wang, P., Huang, A., & Zhang, A. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.667 · Cited by 1
Uncovering Argumentative Flow: A Question-Focus Discourse Structuring Framework
Wang, Y., Zhou, X., Zheng, S., Huang, L., Luo, Z., Luo, W., & Bai, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.416 · Cited by 1
Leveling the playing field? Palestinian digital diplomacy in the 2023–2024 Gaza conflict
Kazzou, K., & Lim, S. (2025) · Place Branding and Public Diplomacy · doi:10.1057/s41254-025-00422-4 · Cited by 1
ECGText: Maximizing GPT Capabilities for Enhanced Text Generation and Emotional Insight
Roy, B. N., Maity, S., Singh, K. R., Chowdhury, P., Das, A., & Saha, D. (2025) · SN Computer Science · doi:10.1007/s42979-025-04564-3 · Cited by 1
How LLMs Influence Perceived Bias in Journalism
Kaeberlein, A., & Alikanhi, M. (2025) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-098-4-059 · Cited by 1
Beyond tickets and texts: Addressing challenges in AI-driven customer service
Fodor, S., Csáki, C., & Kő, A. (2026) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2026.100395 · Cited by 1
LLMs as NAO Robot 3D Motion Planners
Catalini, R., Salici, G., Biagi, F., Borghi, G., Biagiotti, L., & Vezzani, R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00258 · Cited by 1
Wei, Y., Li, X., Huang, Z., Lin, Z., & Zhang, N. (2026) · Computers in Industry · doi:10.1016/j.compind.2026.104462 · Cited by 1
Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
Liu, Z., Khajavi, S. H., Jiang, G., & Liu, X. W. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i44.41104 · Cited by 1
Reducing Human Effort to Validate LLM Relevance Judgements via Stratified Sampling
Merlo, S., Marchesin, S., Faggioli, G., & Ferro, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_27 · Cited by 1
H2A-BPMN: A Hierarchical and Hybrid Agent Framework for Industrial BPMN Automated Modeling
Xu, Q., Zhang, Y., Tang, H., Ran, Q., Gao, H., & Xu, Y. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-21171-2_10 · Cited by 1
Cross-Domain Fake News Detection on Unseen Domains via LLM-Based Domain-Aware User Modeling
Yang, X., Wang, Y., Zhu, J., Ding, P., Liu, H., Zhang, X., & Liu, H. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792990 · Cited by 1
Barghorn, J., Sotnikova, A., Friedli, S., & Bosselut, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791236 · Cited by 2
iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision
Bai, J., Cheong, W. S., Müller, P., & Lim, B. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790539 · Cited by 4
A Graph-Enhanced Defense Framework for Explainable Fake News Detection with LLM
Wang, B., Ma, J., Lin, H., Yang, Z., Yang, R., Tian, Y., & Chang, Y. (2026) · ACM Transactions on Information Systems · doi:10.1145/3810244 · Cited by 1
Feng, B., Liu, C.-F., Liang, Y.-H. L., Yang, C. S., Fu, S., Chen, Z., Lu, K.-H., Huang, S.-F., Yang, C. S., Wang, Y.-C. F., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464094 · Cited by 1
Self-citationRule-Based Explanations for Retrieval-Augmented LLM Systems
Rorseth, J., Godfrey, P., Golab, L., Srivastava, D., & Szlichta, J. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10737-z · Cited by 1
The Crowd Without People: Multi-agent Large Language Models as Idea Evaluators
Probost, F., Schoch, M., Voucko-Glockner, H., Gimpel, H., & Laubacher, R. (2026) · Group Decision and Negotiation · doi:10.1007/s10726-026-09993-w · Cited by 1
Pan, Y.-L., Yu, P.-F., Yin, X.-L., Hong-sheng, Y., Long, X., & Tong, Q. (2026) · Educational Technology Research and Development · doi:10.1007/s11423-026-10659-2 · Cited by 1
AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation
Bougie, N., Marconi, G. M., Ye, X., & Watanabe, N. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.747 · Cited by 13
Auto-Judge: A Cross-Task Benchmark for Comparing LLM Judges for Citation-Grounded RAG Systems
Farzi, N., Hagen, T. H., Yang, E., Fröbe, M., Pradeep, R., Rahmani, H. A., Wang, X., Zendel, O., Potthast, M., & Dietz, L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808601 · Cited by 2
TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models
Li, C., Liu, X., Song, Z., Chi, C., Shi, B., Zhao, C., Chang, G., Wang, Z., Yang, K., Wang, X., et al. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808617 · Cited by 5
Aspect-Based Semantic Textual Similarity for Educational Test Items
Do, H., & Lee, G. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64299-9_30 · Cited by 2
Accuracy and creativity analysis of ChatGPT in quantitative aptitude
Mane, P. C. (2025) · International Journal of Information and Learning Technology · doi:10.1108/ijilt-02-2024-0019 · Cited by 1
DHP Benchmark: Are LLMs Good NLG Evaluators?
Wang, Y., Yuan, J., Chuang, Y.-N., Wang, Z., Liu, Y., Cusick, M., Kulkarni, P., Ji, Z., Ibrahim, Y., & Hu, X. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.451 · Cited by 1
RAGProbe: Breaking RAG Pipelines with Evaluation Scenarios
Sivasothy, S., Barnett, S., Kurniawan, S., Rasool, Z., & Vasa, R. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00015 · Cited by 1
Zhao, K., Yang, B., Tang, C., Dai, S., Tang, H., Lin, C., & Zhan, L. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109574 · Cited by 2
Image Embedding Sampling Method for Diverse Captioning
Waheed, S., & An, N. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.156 · Cited by 1
LTRC_IIITH’s 2023 Submission for Prompting Large Language Models as Explainable Metrics Task
Baswani, P., Mukherjee, A., & Shrivastava, M. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.13 · Cited by 1
Promptor: An Interactive and Conversational Prompting Agent for Intelligent Text Entry TechniquesPreprint
Shen, J., Dudley, J. J., Zheng, J., Byrne, B., & Kristensson, P. O. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4690829 · Cited by 1
Survey-Based Evaluation of Keyword Extraction Algorithms for Internet-Scale Contextual AdvertisingPreprint
Cai, J., Leckner, S., & Björklund, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4883750 · Cited by 1
Theory of Mind Imitation by LLMs for Physician-Like Human EvaluationPreprint
Awasthi, R., Mishra, S. V., Raghu, C., Auron, M., Atreja, A., Mahapatra, D., Singh, N., Khanna, A. K., Cywiński, J. B., Maheshwari, K. K., et al. (2025) · medRxiv · doi:10.1101/2025.03.01.25323142 · Cited by 1
DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-AnsweringPreprint
Badshah, S., & Sajjad, H. (2025) · Qeios · doi:10.32388/b69sky · Cited by 1
Kuerbanjiang, W., Wang, X., Jiamaliding, Y., Maimaitiaili, M., & Yi, Y. (2025) · medRxiv · doi:10.1101/2025.10.30.25339199 · Cited by 1
Edge-Deployed Context-Aware LLM Framework for Low-Latency Bi-Directional Gaze-Speech HRIPreprint
Zhai, Y., & Chian, Y.-T. (2026) · Preprints.org · doi:10.20944/preprints202602.0308.v1 · Cited by 1
Experimental Evidence on the Learning Impact of Generative AI
Contractor, Z., & Reyes, G. (2026) · arXiv · doi:10.48550/arxiv.2607.08849
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2604.22891
SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval
Derehag, J., Calva, C., & Ghiurau, T. (2026) · arXiv · doi:10.48550/arxiv.2603.15599
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709 · Cited by 51
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Zhao, J., Shin, C., Huang, T.-H., GNVV, S. S. S. N., & Sala, F. (2026) · arXiv · doi:10.48550/arxiv.2603.00039
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
Generative Reviewer Agents: Scalable Simulacra of Peer Review
Bougie, N., & Watanabe, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2025.emnlp-industry.8
EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media
Kolagar, Z., Steindl, S., & Zarcone, A. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.4
Survey-Based Evaluation of Keyword Extraction Algorithms for Internet-Scale Contextual AdvertisingPreprint
Cai, J., Leckner, S., & Björklund, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4898318
ZhuJiu-Knowledge: A Fairer Platform for Evaluating Multiple Knowledge Types in Large Language Models
Du, P., Liang, S., Zhang, B., Cao, P., Chen, Y., Liu, K., & Zhao, J. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-demo.20 · Cited by 1
Retrieval Augmented Spoken Language Generation for Transport Domain
Srinivasagan, G., & Georges, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70566-3_1
Kim, S., Kim, J., & Oh, D. (2024) · Journal of Digital Contents Society · doi:10.9728/dcs.2024.25.9.2479
Limitations of Coding LLM'sPreprint
Ignise, A., & Vahi, Y. (2024) · doi:10.36227/techrxiv.172902881.12882316/v1
OpenLEAF: A Novel Benchmark for Open-Domain Interleaved Image-Text Generation
An, J., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, L., & Luo, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3685511
Anaguchi, F., Chakraborty, S., Morita, T., Egami, S., Ugai, T., & Fukuda, K. (2024) · International Symposium on Computing and Networking · doi:10.1109/candar64496.2024.00010
Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and ChallengesPreprint
Hisham, M. (2025) · Qeios · doi:10.32388/z95x8o
Koyuncu, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3736408 · Cited by 1
Hennekeuser, D., Vaziri, D. D., Golchinfar, D., & Stevens, G. (2025) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-025-01169-5
Human Evaluation of Some English-Hindi Machine Translation Systems
Lonchenpa, R., Krishnia, A., Katyayan, P., Vizo, M., & Joshi, N. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-91331-0_7
Fan, Z., He, J., Li, X., Kuang, S., Song, K., Zhou, Y., & Qiu, X. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.438
Methods to Evaluate and Improve the Quality of Text Summarization
Nguyen, H. (2025) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2443390
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
Tavakoli, L., & Zamani, H. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744574 · Cited by 4
Exploring the Potential of Foundation Models as Reliable AI Contact Centers
Byun, H., Park, M., Kim, S., Kim, E., & Song, K. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737420
How Aligned are Humans and Large Language Models in Evaluating Computational Thinking Tasks?
Asgari, M., Mannila, L., & Strömbäck, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01222-7_9
Mowafy, A., Talebi-Kalaleh, M., Sabek, M., Peng, H., Aqib, M., Adeeb, S., Elgammal, M. A., & Pettit, C. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--55492 · Cited by 1
Reliability and Limitations of GPT-Based Automated Evaluation of Creative Writing
Kim, S. (2025) · Journal of Digital Contents Society · doi:10.9728/dcs.2025.26.8.2141
Evaluating Large Language Models Against Human in Stage Play TranslationPreprint
Su, C., Zhao, E., & Li, Z. (2025) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/j6xb4_v1
Machines flunking an exam: Evaluating large language models on course-related open questionsPreprint
Huang, J., Wei, Y., Zhang, L., Lai, R., Lai, F., & Zheng, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6679662/v1
Towards Reliable Statistical Guarantees for LLM Alignment EvaluationPreprint
Jin, G., Lin, Y., Liu, S., Du, Y., Huang, T., Hu, L., Wang, D., & Huang, X. (2025) · doi:10.36227/techrxiv.176053298.85586285/v1
Financial News Summarization: Can Extractive Methods Still Offer a True Alternative to LLMs?
Reche, N., Pontes, E. L., & Torres‐Moreno, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09037-9_14
Baradziej, S. (2025) · TransNav the International Journal on Marine Navigation and Safety of Sea Transportation · doi:10.12716/1001.19.03.16
UTMath: A Benchmark for Math Evaluation with Unit Test
Yang, B. K., Yang, Q., Ma, Y., & Liu, R. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.315
Ding, Y., Jiang, X., & Akinseloyin, O. (2025) · medRxiv · doi:10.1101/2025.11.08.25339805
An Automated Pipeline for Constructing a Vietnamese VQA-NLE Dataset
Duong, T.-B., Tran, H.-M., Le-Nguyen, B.-N., & Duong, D.-T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-95-1746-6_18
Why Are You Wrong? Counterfactual Explanations for Language Grounding with 3D Objects
Preintner, T., Yuan, W., Huang, Q., König, A., Bäck, T., Raponi, E., & van Stein, B. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227256
Human Evaluation of Some English-Hindi Machine Translation Systems
Lonchenpa, R., Krishnia, A., Vizo, M., & Joshi, N. (2025) · Rupkatha Journal on Interdisciplinary Studies in Humanities · doi:10.21659/rupkatha.v17n3.04g
MicroTraitLLM: A RAG-based, Microbe-focused LLM for Researchers
Rogers, G., Brown, J. R., Sokhansanj, B. A., Mell, J. C., & ROSEN, G. L. (2025) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3765612.3767201
Revolutionizing Language Learning and Proficiency With Adaptive Assessments and AI-Driven System
Nurfadhilah, E., Afra, D. I. N., Putra, P. K., Untariyati, N. A., Fajri, R., Pebiana, S., & Riza, H. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0275-1.ch010
Javadi, V. S., Abedin, Z. U., & Flek, L. (2025) · Workshop on Perspectivist Approaches to NLP (NLPerspectives) · doi:10.18653/v1/2025.nlperspectives-1.2
A False Sense of Privacy: Evaluating the Limitsof Textual Data Sanitization for Privacy ProtectionPreprint
Bhuekar, A. (2025) · Preprints.org · doi:10.20944/preprints202512.2058.v1
Can Vision-Language Models See the World Like You Do?
He, Z., He, J., & Zhang, X. (2025) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss55030.2025.11313959
Lukoyanova, M. A., Danilov, A., Zaripova, R. R., Salekhova, L., & Batrova, N. (2025) · Science for Education Today · doi:10.15293/2658-6762.2506.07
KGEval: Evaluating Scientific Knowledge Graphs with Large Language Models
Nechakhin, V., D’Souza, J., Eger, S., & Auer, S. (2026) · Information · doi:10.3390/info17010035
Literary Language Mashup: Curating Fictions with Large Language Models
Manzanarez, G. A., Monroy, R., Flores, J. G., & Calvo, H. (2026) · Mathematics · doi:10.3390/math14020210
GuardianMind: A Multi-Modal Enhanced Large Language Model for Smart City Emergency ResponsePreprint
Zhao, T., & Wu, L. (2026) · Preprints.org · doi:10.20944/preprints202601.0211.v1
Methods for assessing the quality of experts in the verification of large language models
Teterevenkov, D. (2025) · Neurocomputers · doi:10.18127/j19998554-202506-07
Towards Better Image-Text Matching: Concept-Guided Alignment for Vision-Language Models
Wang, X., Zhang, H., Dong, J., Lin, Y., & Liu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5567-3_36
Enhancing Debunking Effectiveness Through LLM-Based Personality Adaptation
Dell'Oglio, P., Bondielli, A., Marcelloni, F., & Passaro, L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_23
Mou, B., & Lou, R. (2026) · ChemRxiv · doi:10.26434/chemrxiv.15000021/v1
Large Language Model Judged Self-Training for Named Entity Recognition
Chen, S., Wang, J., Yang, C., Xiao, Y., Li, Z., & Lin, X. A. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3778009
Optimizing Chain of Thought Generation Through Hierarchical Dynamic Reasoning
Chen, Y., Chen, H., Shao, P., & Liu, X. (2025) · International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI) · doi:10.1109/ricai68060.2025.11385108
The UNLP 2024 Shared Task on Fine-Tuning Large Language Models for Ukrainian
Romanyshyn, M., Syvokon, O., & Kyslyi, R. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/57pbivswb65o
Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP) @ LREC-COLING 2024
(2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5bwu58575ghh
Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024
(2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4upp3i6m57nt
Lou, Z., & Qi, H. (2026) · Preprints.org · doi:10.20944/preprints202603.0231.v1
Kharlamova, D., & Trnavac, R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6344118
Generative Simulation and Summarization of Neonatal Patient Data
Levine, J., Riarh, G., & Green, J. R. (2026) · Information · doi:10.3390/info17030261
An Iterative Multi-Agent Analysis for Automated Evaluation in NLG Tasks
Alhawasi, H., Shan, R., & Youssef, A. S. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00012
Han, L., Chen, C., & Zheng, C. (2025) · New frontiers in translation studies · doi:10.1007/978-981-95-4798-2_8
Xin, S., Li, H., Yang, Z., Zhou, Y., Dong, L., & Chen, C. L. P. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423063
Борисов, А. А., Burtsev, T., Kosov, P., Bobrovskaya, T. V., Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Pamova, A. P., & Arzamasov, K. M. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1768005
When LLMs Annotate: Reliability Challenges in Low-Resource NLI
Panahi, S., Kelleher, J., & Nedumpozhimana, V. (2026) · Workshop on Language Models for Low-Resource Languages (LoResLM) · doi:10.18653/v1/2026.loreslm-1.17
Metadata
- Authors
- Chiang, C.-H., & Lee, H.-Y.
- Affiliations
- National Taiwan University (TW)
- Corresponding author
- Lee, Hung-yi
- ORCID
- Lee, Hung-yi
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 15607-15631
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571232
- Cited by
- 331
- References
- 40
- Citations by year
- 2026: 90 · 2025: 130 · 2024: 80 · 2023: 31
- FWCI
- 58.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Delta Electronics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, LLM evaluation, human evaluation, text quality assessment, adversarial attacks, assessment consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
