State of What Art? A Call for Multi-Prompt LLM Evaluation
Open accessAbstract
Recent advances in LLMs have led to an abundance of evaluation benchmarks, which typically rely on a single instruction template per task. We create a large-scale collection of instruction paraphrases and comprehensively analyze the brittleness introduced by single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. We find that different instruction templates lead to very different performance, both absolute and relative. Instead, we propose a set of diverse metrics on multiple instruction paraphrases, specifically tailored for different use cases (e.g., LLM vs. downstream development), ensuring a more reliable and meaningful assessment of LLM capabilities. We show that our metrics provide new insights into the strengths and limitations of current LLMs.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LMs are not robust: the precise string used for a given prompt, even down to whitespace usage, can significantly impact performance.
“LMs are not robust, often in seemingly unintuitive ways: for example, the precise string used for a given prompt, even down to whitespace usage, can significantly impact performance (Weber et al., 2023; Sclar et al., 2023; Mizrahi et al., 2024; Alzahrani et al., 2024; Webson and Pavlick, 2022).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Mizrahi et al., 2024
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Kendall, M. G., & Smith, B. B. (1939) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177732186 · Cited by 992 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Nonparametric Statistics for Non‐Statisticians
Corder, G. W., & Foreman, D. I. (2009) · Wiley eBooks · doi:10.1002/9781118165881 · Cited by 971 · see citing works
THE TREATMENT OF TIES IN RANKING PROBLEMS
KENDALL, M. G. (1945) · Biometrika · doi:10.1093/biomet/33.3.239 · Cited by 715 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 2× · see citing works
“Various LLM evaluation benchmarks, such as BIG-bench and HELM, use a single instruction template per task, evaluating all models against it (Srivastava et al., 2023a; Liang et al., 2023).”
BackgroundMethod(2021) · International Conference on Learning Representations · Cited by 435 · see citing works
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.806 · Cited by 160 · see citing works
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151 · see citing works
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., & Zhou, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.183 · Cited by 130 · see citing works
Demystifying Prompts in Language Models via Perplexity Estimation
Gonen, H., Iyer, S., Blevins, T., Smith, N., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.679 · Cited by 109 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., & Li, B. (2021) · arXiv · doi:10.48550/arxiv.2111.02840 · Cited by 88 · see citing works
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., Yang, L., Huang, H., Ye, W., Geng, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12095 · Cited by 86 · see citing works
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Honovich, O., Shaham, U., Bowman, S. R., & Levy, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.108 · Cited by 66 · see citing works
Guo, J., Lv, S., Wang, S., Wei, S.-M., Guo, Y., & Yang, L.-Y. (2022) · Learning and Instruction · doi:10.1016/j.learninstruc.2022.101692 · Cited by 40 · see citing works
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Voronov, A. Y., Wolf, L., & Ryabinin, M. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.375 · Cited by 33 · see citing works
Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks
Rao, A., Vashistha, S., Naik, A., Aditya, S., & Choudhury, M. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/2ivv5y3ufu9y · Cited by 19 · see citing works
Evaluating the Zero-shot Robustness of Instruction-tuned Language ModelsPreprint
Sun, J., Shaib, C., & Wallace, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.11270 · Cited by 13 · see citing works
LMentry: A Language Model Benchmark of Elementary Language Tasks
Efrat, A., Honovich, O., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.666 · Cited by 13 · see citing works
Weber, L., Bruni, E., & Hupkes, D. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.20 · Cited by 9 · see citing works
Efficient Benchmarking (of Language Models)
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., & Choshen, L. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.139 · Cited by 7 · see citing works
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
4 of the 39 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Homogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188 · see citing works
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185 · see citing works
Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective
Lavrinovics, E., Biswas, R., Bjerva, J., & Hose, K. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100844 · Cited by 111 · see citing works
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024) · arXiv · doi:10.48550/arxiv.2410.12405 · Cited by 78
“(Mizrahi et al., 2024) demonstrated that model robustness leads to cherry-picking of model performance.”
BackgroundAnalysis of LLMs for educational question classification and generation
Faraby, S. A., Romadhony, A., & Adiwijaya, A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100298 · Cited by 69 · see citing works
Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models
Zhang, T., Irsan, I. C., Thung, F., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697009 · Cited by 48 · see citing works
POSIX: A Prompt Sensitivity Index For Large Language Models
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.852 · Cited by 34 · see citing works
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Voronov, A. Y., Wolf, L., & Ryabinin, M. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.375 · Cited by 33 · see citing works
Social Bias Evaluation for Large Language Models Requires Prompt Variations
Hida, R., Kaneko, M., & Okazaki, N. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.783 · Cited by 29 · see citing works
Kastrati, M., Imran, A. S., Hashmi, E., Kastrati, Z., Daudpota, S. M., & Biba, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110136 · Cited by 24 · see citing works
Ceron, T., Falk, N., Barić, A., Nikolaev, D., & Padó, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00710 · Cited by 23 · see citing works
Are Large Language Models Consistent over Value-laden Questions?
Moore, J., Deshpande, T., & Yang, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.891 · Cited by 21 · see citing works
Windisch, P. Y., Dennstädt, F., Koechli, C., Schröder, C., Aebersold, D. M., Forster, R. E., & Zwahlen, D. R. (2024) · Cureus · doi:10.7759/cureus.75748 · Cited by 20 · see citing works
Computational hermeneutics: evaluating generative AI as a cultural technology
Kommers, C., Ahnert, R., Antoniak, M., Benetos, E., Benford, S. A., Bunz, M., Caramiaux, B., Concannon, S. J., Disley, M., Dobson, J. E., et al. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1753041 · Cited by 19 · see citing works
Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks
Siska, C., Marazopoulou, K., Ailem, M., & Bono, J. W. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.560 · Cited by 19 · see citing works
Exploring the Impact of Large Language Models on Disease Diagnosis
Almubark, I. (2025) · IEEE Access · doi:10.1109/access.2025.3527025 · Cited by 18 · see citing works
An Investigation of Prompt Variations for Zero-Shot LLM-Based Rankers
Sun, S., Zhuang, S., Wang, S., & Zuccon, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_12 · Cited by 18 · see citing works
Confidence in the Reasoning of Large Language Models
Yudi Pawitan, Y., & Holmes, C. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.b033a087 · Cited by 17 · see citing works
Hallucinations in medical devices
Granstedt, J., Prabhat, K. C., Deshpande, R., Garcia, V. L., & Badano, A. G. (2025) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2025.100145 · Cited by 15 · see citing works
Pecher, B., Srba, I., & Bieliková, M. (2024) · ACM Computing Surveys · doi:10.1145/3691339 · Cited by 15 · see citing works
Viswanathan, S., Ibrahim, S. B., Shankar, R., Binns, R., Kleek, M. G. V., & Slovák, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714088 · Cited by 15 · see citing works
Has the creativity of large-language models peaked?
Haase, J., Hanel, P. H. P., & Pokutta, S. (2025) · Journal of Creativity · doi:10.1016/j.yjoc.2025.100113 · Cited by 13 · see citing works
Wu, D., Yang, J., & Wang, K. (2024) · Patterns · doi:10.1016/j.patter.2024.101030 · Cited by 11 · see citing works
MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models
Chen, Z., Liu, Y., Shi, L. Q., Wang, Z.-J., Chen, X., Zhao, Y., & Ren, F. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714674 · Cited by 11 · see citing works
From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
Ohmer, X. I., Bruni, E., & Hupkes, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00529 · Cited by 11 · see citing works
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Haldar, R., & Hockenmaier, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1361 · Cited by 11 · see citing works
“…bias (Zheng et al., 2023; Li et al., 2025), uncer-tainity in judgments (Wagner et al., 2024), adversarial vulnerability (Gu et al., 2025), and domain-specific performance gaps (Tseng et al., 2024), in addition to existing problems of LLM generations like prompt sensitivity (Mizrahi et al., 2024).”
JuStRank: Benchmarking LLM Judges for System Ranking
Gera, A., Boni, O., Perlitz, Y., Bar-Haim, R., Eden, L., & Yehudai, A. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.34 · Cited by 10 · see citing works
Schofield, A., Wu, S., de Volo, T. B., Kuze, T., Gomez, A., & Sultana, S. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701201 · Cited by 9 · see citing works
Sun, L., An, Y.-H., Song, X., & Liu, J. (2025) · Electronic Commerce Research · doi:10.1007/s10660-025-09990-2 · Cited by 9 · see citing works
Developing high-quality, practical, and ethical automated L2 speaking assessments
de Jong, N. H., Raaijmakers, S. A., & Tigelaar, D. E. H. (2025) · System · doi:10.1016/j.system.2025.103796 · Cited by 9 · see citing works
Bandeira, A., Gonçalves, L. H. D. N., Holl, F., Shaibu, J. U., Gonçalves, M. L., Payinda, R., Paudel, S., Berionni, A., WFPHA, Y., Purnat, T. D., et al. (2025) · JMIR Infodemiology · doi:10.2196/69474 · Cited by 9 · see citing works
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Waldis, A., Perlitz, Y., Choshen, L., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 9 · see citing works
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8 · see citing works
“…a single model is often performed many times to monitor checkpoints during pre-training (Biderman et al., 2023a; Liu et al., 2023) and to explore different prompting strategies or a wider range of hyperparameters (Weber et al., 2023b; Mizrahi et al., 2023; Sclar et al., 2023; Voronov et al., 2024).”
BackgroundChain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs
Ngassom, S. K., Dakhel, A. M., Tambon, F., & Khomh, F. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664772 · Cited by 8 · see citing works
Guiding Generative Storytelling with Knowledge Graphs
Pan, Z., Andronis, A., Hayek, E., Wilkinson, O. A. P., Lasy, I., Parry, A., Gadney, G., Smith, T. J., & Grierson, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2603634 · Cited by 8 · see citing works
Othman, A., Chemnad, K., Tlili, A., Da, T., Wang, H., & Huang, R. (2024) · Discover Global Society · doi:10.1007/s44282-024-00113-0 · Cited by 8 · see citing works
Leiter, C., & Eger, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.641 · Cited by 8 · see citing works
Enhancing healthcare resource allocation through large language models
Wan, F. J., Wang, K., Wang, T., Qin, H., Fondrevelle, J., & Duclos, A. (2025) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2025.101859 · Cited by 7 · see citing works
How do LLMs perform on Turkish? A multi-faceted multi-prompt evaluation
Topal, M. B., Bozanta, A., & Başar, A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127421 · Cited by 7 · see citing works
Is This the Best Prompt? Scoring Prompts for Arabic NLP Across LLMs
Refai, D., Al-shaibani, M. S., & Ahmad, I. (2025) · IEEE Access · doi:10.1109/access.2025.3616181 · Cited by 7 · see citing works
A Scoping Review of Large Language Models: Architecture and Applications
Moujahid, H., Boutahar, K., Gannour, O. E., Saleh, S., Cherradi, B., & Abbassi, A. E. (2024) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset60544.2024.10549006 · Cited by 7 · see citing works
Deep learning application for marketing engagement – its thematic evolution
Yu, B. T. W., & Liu, S. (2024) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-08-2024-0371 · Cited by 7 · see citing works
Zeng, Y., Brown, C., Byari, M., Raymond, J., Schmitt, T. M., & Rounsevell, M. (2025) · Geoscientific model development · doi:10.5194/gmd-18-4983-2025 · Cited by 7 · see citing works
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models
Nalbandyan, G., Shahbazyan, R., & Bakhturina, E. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-industry.39 · Cited by 6 · see citing works
HuggingGraph: Understanding the Supply Chain of LLM Ecosystem
Rahman, M. S., Gao, P., & Ji, Y. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761510 · Cited by 6 · see citing works
When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
Seleznyov, M., Chaichuk, M., Ershov, G., Panchenko, A., Tutubalina, E. V., & Somov, O. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1109 · Cited by 6 · see citing works
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
“This typically requires a reimplementation step , where a variety of new models or codebases must interface with the new benchmark in order for it to be widely run.”
Position: Benchmarking is Broken - Don't Let AI be Its Own Judge
Cheng, Z., Wohnig, S., Gupta, R., Alam, S., Abdullahi, T., Alves, J., Nielsen-Garcia, C., Mir, S., Li, S., Orender, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-2433 · Cited by 5 · see citing works
Identifying and Mitigating API Misuse in Large Language Models
Zhuo, T. Y., He, J., Sun, J., Xing, Z., Lo, D., Grundy, J., & Du, X. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3651566 · Cited by 5 · see citing works
Topal, M. B., Bozanta, A., & Basar, A. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10838185 · Cited by 5 · see citing works
Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions
Li, Y., Miao, Y., Ding, X., Krishnan, R., & Padman, R. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.347 · Cited by 5 · see citing works
LLMs prompted for graphs: hallucinations and generative capabilities
Richardeau, G., Chali, S., Merrer, E. L., Penzo, C., & Trédan, G. (2025) · Applied Network Science · doi:10.1007/s41109-025-00754-3 · Cited by 5 · see citing works
Su, H., Lei, S., Chen, Y., Lv, Y., & Zhang, Q. (2025) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.5c16158 · Cited by 4 · see citing works
Endless forms most similar: the dearth of the author in AI-supported art
Kreminski, M. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02326-6 · Cited by 4 · see citing works
Fajčík, M., Dočekal, M., Doležal, J., Ondrej, K., Beneš, K., Kapsa, J., Smrž, P., Polok, A., Hradiš, M., Nevěřilová, Z., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.32 · Cited by 4 · see citing works
Jin, R., Zhao, M., Niu, C., Xia, Y., Zhou, H., & Liu, N. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13774-4 · Cited by 4 · see citing works
Benchmarking generative AI tools for interpretation of the WHO TB mutation catalogue
Moreno-Molina, M., Suresh, A., Colman, R. E., & Rodwell, T. C. (2026) · BMC Digital Health · doi:10.1186/s44247-025-00225-w · Cited by 4 · see citing works
Wang, R., Zhang, X., Liu, Q., Su, J., Zhang, Y., & Ma, Y. (2026) · npj Heritage Science · doi:10.1038/s40494-026-02384-z · Cited by 4 · see citing works
Engel, C., Hermstrüwer, Y., & Kim, A. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09517-6 · Cited by 4 · see citing works
A Large Model Assisted Remote Sensing Image Scene Understanding Algorithm Based on Object Detection
Wang, Z., Xu, Z., Yang, W., Chen, W., & Yang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5597-4_5 · Cited by 4 · see citing works
Gupta, R., Tiwari, S. M., & Chaudhary, P. (2025) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-82062-5_7 · Cited by 4 · see citing works
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3 · see citing works
“For the EvalJudge Human An-notation task, we recruited native English speakers through Amazon Mechanical Turk 3 (MTurk”
Zambrano, A. F., Singhal, S., Pankiewicz, M., Baker, R. S., Porter, C., & Liu, X. (2025) · Information and Learning Sciences · doi:10.1108/ils-11-2024-0156 · Cited by 3 · see citing works
Groen, E. C., Dalpiaz, F., van Vliet, M., Winter, B., Doerr, J., & Brinkkemper, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112533 · Cited by 3 · see citing works
THE COST OF EXPERTISE: PERFORMANCE TRADEOFFS IN LLMs FOR SYSTEMS ENGINEERING
Wach, P., Bell, R., Jugan, B., Longshore, R., & Madachy, R. J. (2025) · INCOSE International Symposium · doi:10.1002/iis2.70078 · Cited by 3 · see citing works
VLM@school: Evaluation of AI Image Understanding on German Middle School Knowledge
Peinl, R., & Tischler, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07986-2_41 · Cited by 3 · see citing works
Okeukwu-Ogbonnaya, A., Amatapu, R., Bergtold, J. B., & Amariucai, G. T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13513-1_22 · Cited by 3 · see citing works
🧑🍳 Cooking Up Creativity : Enhancing LLM Creativity through Structured Recombination
Mizrahi, M., Shani, C., Stanovsky, G., Jurafsky, D., & Shahaf, D. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.638 · Cited by 3 · see citing works
Self-citationLabel-Efficient Model Selection for Text Generation
Ashury-Tahan, S., Gera, A., Sznajder, B., Choshen, L., Ein‐Dor, L., & Shnarch, E. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.456 · Cited by 3 · see citing works
Moore, K., Roberts, J., Pham, T., Ewaleifoh, O., & Fisher, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.126 · Cited by 3 · see citing works
Saarthi: Ensemble AI4Bharat IndicTrans2 and Gemini for Multilingual Indian Chatbot
Jain, A., Tiwari, A., Kumar, P., Londhe, K., Kumar, S., & Chaturvedi, A. (2025) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1109/aimlsystems67835.2025.11330894 · Cited by 3 · see citing works
LEAF: Predicting the Environmental Impact of Food Products based on their Name
Krahmer, B. (2024) · Workshop on Natural Language Processing Meets Climate Change (ClimateNLP) · doi:10.18653/v1/2024.climatenlp-1.10 · Cited by 2 · see citing works
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios
Ackerman, S., Rabinovich, E., Farchi, E., & Tavor, A. A. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.158 · Cited by 2 · see citing works
Masterpiece Creation: An AI-powered Robotic Calligraphy Creation System
Hu, S., Wang, N., Zhao, K., Jing, Y., Zhang, Y., Wang, Z., Guo, B., & Yu, Z. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749957 · Cited by 2 · see citing works
Xue, H., Zhao, B., Xie, H., & Sun, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-2725-0_12 · Cited by 2 · see citing works
Reducing Complexity in Surgical Consents: The Role of AI in Patient Communication
Ramanathan, R., Kelly, R., Shaw, J. D., Gopakumar, A., Shannon, M. F., Gonzalez, C., Weinberg, J., Bonamer, J. P., Wawrose, R. A., Spitnale, M. J., et al. (2025) · Medical Research Archives · doi:10.18103/mra.v13i12.7178 · Cited by 2 · see citing works
Trabelsi, Z., Ali, M., Qayyum, T., & Tariq, A. (2026) · Scientific Reports · doi:10.1038/s41598-026-39791-y · Cited by 2 · see citing works
ZeroDL: Zero-shot Distribution Learning for Text Clustering via Large Language Models
Jo, H., Lee, H., Yoo, K. M., & Park, T. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1005 · Cited by 2 · see citing works
TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
Tambon, F., Nikanjam, A., Zid, C., Khomh, F., & Antoniol, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773285 · Cited by 2 · see citing works
The Screen Decides the Verdict: Operating Characteristics of Model-Exclusion Rules in LLM EvaluationPreprint
Boullineau, E., & Arciniegas, J. D. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7490919 · Cited by 2 · see citing works
SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation
Shen, M., Chang, N., Liu, S., & Álvarez, J. M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709417 · Cited by 1 · see citing works
Fu, Z.-X., & Lee, C.-H. (2024) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3711542.3711561 · Cited by 1 · see citing works
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1 · see citing works
Tackling Distribution Shift in LLM via KILO: Knowledge-Instructed Learning for Continual Adaptation
Muttakhiroh, L., & Fevens, T. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204312 · Cited by 1 · see citing works
On the Effect of Instruction Tuning Loss on Generalization
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.42 · Cited by 1 · see citing works
Bañuls-Lapuerta, F. A., Marti-Miralles, V., González-García, R. J., & Rico, G. M. (2026) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/86786 · Cited by 1 · see citing works
FermBench: A new benchmark for measuring the capabilities of LLMs on fermentation knowledge
Caccavale, F., Aouichaoui, A. R. N., Krühne, U., Gernaey, K. V., & Gargalo, C. L. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100577 · Cited by 1 · see citing works
Novák, A., Oblaković, G., Njavro, M., & Ehler, M. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12062-7 · Cited by 1 · see citing works
CATS : An enhanced framework in LLM-based tabular data synthesis by correlation augmentation
Chen, L., Jiang, M., Dai, Z., Liu, S., & Chai, H. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132645 · Cited by 1 · see citing works
LLM-Enabled Knowledge Transfer: Modeler to SME
Anderson, C., Topcu, T. G., Jugan, B., Nerayo, M., & Wach, P. (2026) · Conference on systems engineering research series · doi:10.1007/978-3-032-12309-1_36 · Cited by 1 · see citing works
A large language model framework for sample-free population synthesis
Jones, M., Dawson, R., & Mills, J. P. (2026) · PLoS ONE · doi:10.1371/journal.pone.0341704 · Cited by 1 · see citing works
Xie, Q., Liang, Z., Wu, J., Chen, Y., Wang, W., Ma, W., Ming, Z., Yang, H., & Wu, K. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.2084 · Cited by 1 · see citing works
Skill Learning Using Process Mining for Large Language Model Plan Generation
Redis, A. C., Sani, M. F., Zarrin, B., & Burattin, A. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-82225-4_48 · Cited by 1 · see citing works
Evaluation and Prototype of Health Counseling LLM Using the Turing TestPreprint
Nakamura, K. D., Tatsuoka, H., Miyakawa, T., Ishii, H., Onishi, M., & Ohyama, Y. (2024) · Preprints.org · doi:10.20944/preprints202408.1614.v1 · Cited by 1 · see citing works
Ding, Y., Han, X., Yang, J., Wang, T., Bi, Z., Song, X., Hao, J., Song, J., Ge, E., Peng, B., et al. (2026) · doi:10.36227/techrxiv.176784493.36112281/v1 · Cited by 1 · see citing works
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv
“At a broader level, Mizrahi et al. (2024) provide a comprehensive survey of robustness in LLMs, organizing findings across adversarial, out-of-distribution, and prompt-sensitivity dimensions, and concluding that robustness remains an open challenge even for frontier models.”
Zh-Paral: Benchmark Dataset for Comprehension of Chinese Paralinguistic Speech
Chen, D., & Wu, Z. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800328
Bias Mitigation and Shifting Ideologies of LLM by Prompting
Ueda, Y., & Suwa, H. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94171-9_41
On the Attitudes of GameDev Industry Artists towards GenAI. Preliminary Results
Sikorski, Ł., Matulewski, J., & Czerwonka, M. (2025) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/3716489.3728432
- Authors
- Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G.
- Affiliations
- Hebrew University of Jerusalem (IL)
- University of Haifa (IL)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 933-949
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4401208776
- Cited by
- 113
- References
- 39
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 19
- FWCI
- 16.68
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- European Commission (HORIZON2020)
- Hebrew University of Jerusalem
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- LLM evaluation, LLM evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
