Large Language Models are Inconsistent and Biased Evaluators
Open accessAbstract
The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Scores vary with prompt wording, anchors, familiarity, and repeated sampling, while judges may recognize and favor outputs from their own model family.
“Scores also vary with prompt wording, anchors, familiarity, and repeated sampling, while judges may recognize and favor outputs from their own model family(Stureborg et al., 2024;Panickssery et al., 2024).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Stureborg et al., 2024
Section 3.4 (page unavailable) reports that scores can vary with slight prompt changes and repeated sampling; sections 3.3 and 3.1 report anchoring effects and familiarity bias. The paper mentions preference for self-generated text in its discussion of prior work, but our copy does not establish that judges recognize outputs from their own model family. (LLM, conf=0.87)
Experimental — accuracy not yet measuredSycophancy is a special case of Q_i, with Q_i = δ_{K•1}, near-maximum scores assigned indiscriminately.
“sycophancy (Q i = δ K•1 , near-maximum scores assigned indiscriminately, Wang et al., 2023;Stureborg et al., 2024);”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Stureborg et al., 2024
Section 3.2 (page unavailable) reports that scores on a 1–100 scale cluster between 70 and 100, with outsized probabilities at 90 and 95. That supports a tendency toward high scores, but the paper does not characterize it as sycophancy or say that near-maximum scores are assigned indiscriminately, as the point-mass notation implies. (LLM, conf=0.82)
Experimental — accuracy not yet measuredLLM judges exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.
“LLM judges are shown to exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Stureborg et al., 2024
LLMs fail to produce stable preferences across reruns.
“Studies have also shown that LLMs favor their own generations (Koo et al., 2023), or fail to produce stable preferences across reruns (Stureborg et al., 2024).”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Stureborg et al., 2024
LLM-based evaluations are sensitive to prompt phrasing, label order, response length, and position.
“LLM-based evaluations are sensitive to prompt phrasing (Zhou et al., 2023), label order (Wang et al., 2023), response length (Stureborg et al., 2024), and position (Shi et al., 2024;Kudugunta et al., 2023).”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Stureborg et al., 2024
LLM-as-a-judge has limitations such as self-preference bias, verbosity bias, and sensitivity to prompt phrasing.
“Despite its advantages, LLM-as-a-judge has limitations such as self-preference bias, where an LLM tends to favour its own responses; verbosity bias, where some LLMs may prefer longer, more detailed responses (Zheng et al., 2023;Stureborg et al., 2024); or sensitivity to prompt phrasing (Verga et al., 2024).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Stureborg et al., 2024
The corruption sweep is synthetic injection at a single rubric serialisation and temperature 0, so it does not probe prompt-format sensitivity.
“the corruption sweep is synthetic injection at a single rubric serialisation and temperature 0, so it does not probe prompt-format sensitivity”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Stureborg et al., 2024
Section 2.1.4 describes an evaluation prompt, and the Implementation of Previous Work reports one method run at temperature 0. Neither passage identifies a synthetic-injection corruption sweep or says whether such a sweep tested prompt-format sensitivity. Searches for corruption, injection, and prompt-format variations found no passage addressing that claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe unrestricted corruption component Q_i admits, as special cases, every LLM-judge failure mode reported in the single-judge bias literature.
“The unrestricted Q i admits, as special cases, every LLM-judge failure mode reported in the single-judge bias literature:”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Stureborg et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 26
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
- Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq, Pengfei Liu, and Graham Neubig. 2020. Re- evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Cheng-Han Chiang and Hung-yi Lee. 2023. A closer look into using large language models for automatic evaluation. In Findings of the Association for Com- putational Linguistics: EMNLP 2023, pages 8928- 8942, Singapore. Association for Computational Lin- guistics. · DOI
- Nikolas Coupland. 2011. How frequent are numbers? Language & Communication, 31(1):27-37.
- Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of gen- erated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10960-10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409. · DOI · Paper page
- Erick Fonseca, Lisa Yankovskaya, André F. T. Mar- tins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Pa- pers, Day 2), pages 1-10, Florence, Italy. Association for Computational Linguistics. · DOI
- Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summariza- tion. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Compu- tational Linguistics. · DOI · Paper page
- Hidehito Honda, Rina Kagawa, and Masaru Shirasuna. 2022. On the round number bias and wisdom of crowds in different response formats for numerical estimation. Scientific Reports, 12(1):8167.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Liny- ong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023d. Revisiting the gold standard: Ground- ing summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4140-4170, Toronto, Canada. Association for Computational Linguistics. · DOI
- Annie Louis and Ani Nenkova. 2013. Automatically as- sessing machine summary content without a gold standard. Computational Linguistics, 39(2):267- 300. · DOI
- Tomoko Nemoto and David Beglar. 2014. Likert-scale questionnaires. In JALT 2013 conference proceed- ings, pages 1-8.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
- Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Pro- ceedings of the Sixth Conference on Machine Trans- lation, pages 1030-1040, Online. Association for Computational Linguistics.
- Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
- Thomas Scialom, Sylvain Lamprier, Benjamin Pi- wowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
- Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics. · DOI · Paper page
- Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language mod- els are not yet human-level evaluators for abstrac- tive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215-4233, Singapore. Association for Computa- tional Linguistics. · DOI
- Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Characterizing the confidence of large language model-based automatic evaluation metrics. In Proceedings of the 18th Conference of the Euro- pean Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 76-89, St. Julian's, Malta. Association for Computational Linguistics.
- Manoj Thomas and Vicki Morwitz. 2009. Heuristics in numerical cognition: Implications for pricing. In Handbook of pricing research in marketing, pages 132-149.
- Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131. · DOI · Paper page
- Oleg Vasilyev, Vedant Dharnidharka, and John Bohan- non. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11-20, Online. Association for Computational Linguistics. · DOI
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837.
- Robert B Zajonc. 1968. Attitudinal effects of mere ex- posure. Journal of personality and social psychology, 9(2p2):1.
- Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Lin- guistics. · DOI · Paper page
Cited by
Sorted by citations
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
LLM based QA chatbot builder: A generative AI-based chatbot builder for question answering
Salim, M. S., Hossain, S. I., Jalal, T., Bose, D. K., & Basher, M. J. I. (2024) · SoftwareX · doi:10.1016/j.softx.2024.102029 · Cited by 26 · see citing works
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11 · see citing works
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 6 · see citing works
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4 · see citing works
Pumpo, M. D., Villani, L., Gualano, M. R., et al. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1874389 · Cited by 3 · see citing works
“In our study, the presence of both substantial human–human agreement and measurable expert disagreement supports a cautious interpretation: LLMs may provide useful complementary evaluations, but their ratings should not be treated as substitutes for expert judgement (17).”
SkillAggregation: Reference-free LLM-Dependent Aggregation
Sun, G., Kagrecha, A., Manakul, P., Woodland, P., & Gales, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.756 · Cited by 3 · see citing works
Eliciting problem specifications for LLM-Modulo cognitive systems
Wray, R. E., Kirk, J. R., & Laird, J. E. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101409 · Cited by 3 · see citing works
Automata-Based Steering of Large Language Models for Diverse Structured Generation
Luan, X., Wei, Z., Zhang, Y., & Sun, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4213-0_2 · Cited by 1 · see citing works
Solar Photovoltaic Assessment with Large Language ModelPreprint
Guo, M., & Weng, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5362724 · Cited by 2 · see citing works
Rueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., et al. (2026) · JMIR Medical Education · doi:10.2196/88580 · Cited by 1 · see citing works
Norman, J. D., Rivera, M. U., & Hughes, D. A. (2026) · arXiv · doi:10.48550/arxiv.2606.19544 · Cited by 1 · see citing works
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709
“Furthermore, Stureborg et al. [2024] find that LLM-as-a-judge tends to assign higher scores to completions with lower perplexity—a phenomenon often referred to as familiarity bias.”
Suhara, Y. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1382
Self-citationThe use of artificial intelligence in the assessment of live subtitling quality: the NER Buddy
Romero-Fresco, P., Amigo, Ó. A., & Bacigalupe, L. A. (2024) · Tradumàtica tecnologies de la traducció · doi:10.5565/rev/tradumatica.408 · Cited by 1 · see citing works
Pinuer, L. P., Zapata, I. T., & Quintanilla, H. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-09074-4_42
Pan, Y.-L., Yu, P.-F., Yin, X.-L., Hong-sheng, Y., Long, X., & Tong, Q. (2026) · Educational Technology Research and Development · doi:10.1007/s11423-026-10659-2 · Cited by 1 · see citing works
CODE-GEN: A RAG-Based Agentic AI System for Multiple-choice Question Generation
Duan, X., Nwanganga, F., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_10
RAG as a Content-Analysis Assistant: Auditing SDG Discourse in Online Videos
Safta, C., Bros, V., & Gática-Pérez, D. (2026) · ICDAR@ICMR · doi:10.1145/3810987.3815539
“Third, LLM-as-judge scores are subjective and can be brittle, so we interpret only aggregate trends [26].”
Tchuindjang, M., Duran, N., Legg, P., & Medjek, F. (2026) · Cybersecurity · doi:10.1186/s42400-026-00633-z
“Such discrepancies are especially problematic in cybersecurity settings, where false negatives can directly translate into undetected harmful outputs, undermining trust in automated evaluation pipelines (Koo et al. 2023; Liu et al. 2023; Stureborg et al. 2024; Eiras et al. 2025).”
Granof, M., Ledley, K. T., Xia, W., Myrick, K. J., & Torous, J. B. (2026) · Journal of Mental Health · doi:10.1080/09638237.2026.2732871
Wilhelmi, T., Bartsch, V., Platt, M., Rashid, A., Hornig, J., Krusche, M., Hueber, A. J., Fink, D., Pfeil, A., Dischereit, G., et al. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02470-6
“This pattern is consistent with reported central tendency bias in LLM-based ordinal scoring, where models disproportionately avoid extreme ratings compared to human raters [ 36 , 37 ].”
Multimodal LLMs for Visual Style Evaluation
Pascual, R., Aguirre, I., Sesma‐Sara, M., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39388-3_15
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
RoPoLL: Robust Panel of LLM Judges
Acharya, A., Pan, K. W., & Verkhovsky, B. (2026) · arXiv · doi:10.48550/arxiv.2606.30931
- Authors
- Stureborg, R., Alikaniotis, D., & Suhara, Y.
- Affiliations
- Duke University (US)
- Nvidia (United Kingdom) (GB)
- ORCID
- Stureborg, Rickard
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2405.01724
- OpenAlex
- W4396777837
- Semantic Scholar
- 53d212ea368e75cc152c3cb287343da22849915e
- Cited by
- 15
- References
- 0
- Citations by year
- 2026: 9 · 2025: 3 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
- Keywords
- LLM-based evaluation, evaluation bias, familiarity bias, anchoring effect, prompt sensitivity, zero-shot evaluation, text quality assessment, bias mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
