CoChat
Sign Up
PreprintCited by 9Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge

Open access
Schroeder, Kayla, Wood-Doughty, Zach·2024·arXiv
DOI 10.48550/arxiv.2412.12509arXiv 2412.12509
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have become increasingly powerful and ubiquitous, but their stochastic nature poses challenges to the reliability of their outputs. While deterministic settings can improve consistency, they do not guarantee reliability, as a single sample from the model's probability distribution can still be misleading. Building upon the concept of LLM-as-a-judge, we introduce a novel framework for rigorously evaluating the reliability of LLM judgments, leveraging McDonald's omega. We evaluate the reliability of LLMs when judging the outputs of other LLMs on standard single-turn and multi-turn benchmarks, simultaneously investigating the impact of temperature on reliability. By analyzing these results, we demonstrate the limitations of fixed randomness and the importance of considering multiple samples, which we show has significant implications for downstream applications. Our findings highlight the need for a nuanced understanding of LLM reliability and the potential risks associated with over-reliance on single-shot evaluations. This work provides a crucial step towards building more trustworthy and reliable LLM-based systems and applications.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces a framework using McDonald’s omega to measure internal consistency reliability of LLM-as-a-judge outputs across repeated judgments that vary only the random seed. It finds that LLM judgment reliability is often questionable and sensitive to model, task, and temperature, while high reliability is associated with stable evaluation in a simpler Head-to-Tail application.

Key findings

  • Inter-rater reliability across three LLM evaluators varied drastically across 100 replications, ranging from 0.167 to 1.00, showing it is highly sensitive to random seed variation.
  • McDonald’s omega reliability varied by judge, benchmark, and temperature: at temperature 0 all judges scored 1, but at nonzero temperatures many values fell below acceptable thresholds; Starling-LM-7B-beta and Meta-Llama-3-8B-Instruct showed questionable reliability across benchmarks, with SQuAD and MT-Bench often below acceptable.
  • The reliability ranking of judges (Gemma-1.1-7b-it > Starling-LM-7B-beta > Meta-Llama-3-8B-Instruct) inverted their Chatbot Arena performance ranking, suggesting a possible performance-reliability trade-off.
  • SQuAD had higher answer accuracy but lower judgment reliability than BBH, which the authors attribute to difficulty distinguishing among multiple highly accurate responses; in Head-to-Tail, judges showed very high omega reliability (0.9893–0.9901) and stable ALM estimates (0.0395–0.0396 vs. 0.039 reported by Sun et al., 2024).

Methods

  • The framework used BBH, SQuAD, and MT-Bench, selecting one question from each category for 55 total questions; five responses per question were generated with Chain-of-Thought prompting from LLaMA-3-8B, Vicuna-7B-v1.5, Gemma-7B, Phi-2, and Falcon-7b using top-k 50 and temperature 0.75.
  • Judgment LLMs were Starling-LM-7B-beta, Gemma-1.1-7b-it, and Meta-Llama-3-8B-Instruct; each was prompted 100 times per question, varying only the replication/random seed, at temperatures 1.0, 0.75, 0.5, 0.25, and 0, with top-k fixed at 50.
  • Reliability was calculated with McDonald’s omega using the reliabilPy package; a separate application replicated the Head-to-Tail benchmark’s “head” Academic-domain questions with Vicuna-7B and the same three judges.

Limitations

  • The paper states that the work is focused on single- and multi-turn benchmarks and that future research should extend to additional datasets and benchmarks and develop domain-specific guidelines for interpreting reliability metrics.
  • The paper states that calculated reliability is likely an optimistic upper bound because judgments that did not select a response were treated as a single “Non Response” category, ignoring variability in non-responses.
  • Evident from the text: the study uses only three judge models, all below 10B parameters, and a limited set of benchmarks; it isolates seed variation while holding other parameters fixed, so it does not test all sources of stochasticity or broader model/task coverage.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.12509

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 10 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    Our replicated A LM values closely agree with the original results reported by Sun et al. (2023).

    “Table 5 shows close agreement between our replicated A LM values and the original results, confirming the reliability and reproducibility of LLM-based evaluations in this structured task.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sun et al., 2023 source
  • Findingown data

    Our LLM judges had remarkably high reliability scores across Head-to-Tail replications.

    “Table 6 shows remarkably high reliability scores for our LLM judges across Head-to-Tail replications.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    SQuAD, despite its higher overall answer accuracy, exhibits lower judgment reliability than BBH.

    “SQuAD, despite its higher overall answer accuracy (Table 4), paradoxically exhibits lower judgment reliability than BBH (Table 2).”
    Quote in the paperSubstantial claim
  • Findingown data

    Minimal variability exists across replications in the A LM estimates.

    “Similarly, it is clear from the standard errors in the A LM estimates in Table 5 that minimal variability exists across replications.”
    Quote in the paperSubstantial claim
  • Findingown data

    The ideal temperature for reliable LLM judgment is not universal; it varies depending on the specific LLM and the task.

    “First, we observed that the ideal temperature for reliable LLM judgment is not universal; it varies depending on the specific LLM and the task, as revealed in Table 2.”
    Quote in the paperSubstantial claim
  • May combine two claims. 11
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 7

  • May combine two claims.

    Interpretationown data

    LLM-based evaluations in the Head-to-Tail structured task are reliable and reproducible.

    “Table 5 shows close agreement between our replicated A LM values and the original results, confirming the reliability and reproducibility of LLM-based evaluations in this structured task.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Sun et al., 2023 source

      Section 2.2 (Metrics; page unavailable) reports 98% agreement between LLM-based judgments and human judgments and calls the metrics reliable. Section 3.5 (Robustness; page unavailable) reports that regenerated ChatGPT answers differed for only 1% of sampled questions with the specified prompt, but it does not establish that the evaluation results can be independently replicated. The close agreement between replicated A_LM values and the originals is the citing authors’ own finding, not one reported by this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • The quote only partly states this claim.

    Findingown data

    Inter-rater reliability across three LLM evaluators on BBH judgments varied from 0.167 to 1.00 across 100 replications.

    “Table 1 details the drastic variation in inter-rater reliability, ranging from 0.167 to 1.00.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The LLM-as-a-judge paradigm relies upon the internal consistency reliability of LLM judges.

    “The LLM-as-a-judge paradigm, coined by Zheng et al. (2023), relies upon the internal consistency reliability of LLM judges.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zheng et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Our findings reinforce reliability's importance in trustworthy LLM evaluation and highlight the potential for reliable LLM judgment in well-defined tasks.

    “Our findings reinforce reliability's importance in trustworthy LLM evaluation and highlight the potential for reliable LLM judgment in well-defined tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: LLM judgment reliability is important for trustworthy LLM evaluation.

  • May combine two claims.

    Premisecitation

    A single LLM output is a draw from a probabilistic distribution; even with deterministic settings, it remains subject to inherent randomness.

    “A singular output from an LLM is the result of a draw from a probabilistic distribution (Vaswani, 2017).”
    • Also stated in: § 1 Introduction · plain
      “Even with deterministic settings, a single LLM output remains a sample from the model's probability distribution, subject to inherent randomness.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Vaswani, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hellrich and Hahn, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The application of internal consistency to the LLM-as-a-judge paradigm, particularly with a focus on seed variation, has been largely overlooked.

    “While internal consistency is a well-established concept, its application to the LLM-as-a-judge paradigm, particularly with a focus on seed variation, has been largely overlooked.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    This work pioneers the systematic investigation of the variability of an LLM evaluator's judgments solely by varying the random seed, holding all other parameters constant.

    “Critically, this work pioneers the systematic investigation of the variability of an LLM evaluator's judgments solely by varying the random seed, holding all other parameters constant.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    A single best response should exist in response options [A]-[E] for a judge to uncover because each of the five responses contains reasoning in addition to the question response.

    “Given the employment of Chain-of-Thought in the initial responses to a given benchmark question, a single best response should exist in response options [A] -[E] for a judge to uncover because each of the five responses contains reasoning in addition to the question response.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zheng et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Highly reliable LLM judges produce consistent and trustworthy evaluations.

    “This demonstrates that highly reliable LLM judges produce consistent and trustworthy evaluations, further supporting the validity of reliability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Head-to-Tail employment of LLM-as-a-judge is much more simplistic than our experiments and would be expected to show a higher reliability.

    “Thus, this employment of LLM-as-a-judge is much more simplistic than our experiments and would be expected to show a higher reliability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Inter-rater reliability is highly sensitive to random seed variation, making it an unreliable indicator of true judgment quality.

    “This disconcertingly wide range demonstrates that interrater reliability is highly sensitive to random seed variation, making it an unreliable indicator of true judgment quality.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The five responses to each benchmark question were included in the prompt in a random order to prevent judgment bias.

    “To prevent judgment bias, the five responses to the respective benchmark question are included in the prompt in a random order as response options [A] -[E].”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Head-to-Tail reliability is much higher than that of our experiments because the LLM judgments in the Head-to-Tail paradigm only have two options ('correct' or 'incorrect') and are provided with a ground truth in the prompt.

    “It is unsurprising that the Head-to-Tail reliability is much higher than that of our experiments given that the LLM judgments in the Head-to-Tail paradigm only have two options ('correct' or 'incorrect') and are provided with a ground truth in the prompt.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    All judgments that did not provide a judgment of best response were treated as a 'Non Response' category, so the calculated reliability serves as an upper bound on reliability because variability in non-response judgments is ignored.

    “It is important to note that in calculating the reliability, all judgments that did not provide a judgment of best response were treated as belonging to the same category (”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Our results validate our framework's importance.

    “Our results validate our framework's importance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Internal consistency reliability is a well-established statistical concept and the most common type of reliability employed across domains.

    “We define reliability in this context as internal consistency reliability, a well-established statistical concept and the most common type of reliability employed across domains Henson (2001).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Henson (2001)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    LLM judgment reliability is a significant concern, with several models exhibiting questionable reliability across benchmarks like SQuAD and MT-Bench.

    “LLM judgment reliability is a significant concern, with several models exhibiting questionable reliability across benchmarks like SQuAD and MT-Bench.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    McDonald's omega provides a more robust measure of internal consistency and is more robust to deviations from assumptions than Cronbach's alpha.

    “Ultimately, omega provides a more robust measure of internal consistency and has been shown to be more robust to deviations from assumptions than alpha (Stensen and Lydersen, 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Stensen and Lydersen, 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

18 of 18

  • The authors state this as an assumption.

    Additive measurement error: error in each LLM judgment replication is independent of the true judgment value.

    “1. Additive Measurement Error: Error in each LLM judgment replication is independent of the true judgment value.”

  • The authors state this as an assumption.

    Uncorrelated errors: errors in different LLM judgment replications are independent of each other.

    “2. Uncorrelated Errors: Errors in different LLM judgment replications are independent of each other.”

  • The authors state this as an assumption.

    Single latent trait: all LLM judgments are attempting to measure the same underlying 'true' judgment.

    “3. Single Latent Trait: All LLM judgments are attempting to measure the same underlying”

  • The paper relies on this without support.

    LLM judgment reliability is important for trustworthy LLM evaluation.

    “Our findings reinforce reliability's importance in trustworthy LLM evaluation and highlight the potential for reliable LLM judgment in well-defined tasks.”

  • The paper relies on this without support.

    The Chatbot Arena performance ranking is a valid measure of LLM performance.

    “The ranking of LLM judgment reliability (Gemma-1.1-7b-it> Starling-LM-7B-beta > Meta-Llama-3-8B-Instruct) inverts their Chatbot Arena performance ranking.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting year for the Head-to-Tail source

    Minor

    The text identifies the source of the replicated A LM metric as Sun et al. (2023), but Table 5 attributes the original result to Sun et al., 2024.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 27

  1. Aaron A Agbo. 2010. Cronbach's alpha: Review of limitations and associated recommendations. Journal of Psychology in Africa, 20(2):233-239. · DOI
  2. Sivakumar Alagumalai and David D Curtis. 2005. Clas- sical test theory. Springer. · DOI
  3. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2019. Agreement is overrated: A plea for correlation to assess human evaluation reliability. In Proceedings of the 12th International Conference on Natural Lan- guage Generation, pages 344-354. · DOI
  4. Shreya Bhandari, Yunting Liu, Yerin Kwak, and Zachary A Pardos. 2024. Evaluating the psycho- metric properties of chatgpt-generated questions. Computers and Education: Artificial Intelligence, 7:100284. · DOI
  5. Sairavi Kiran Biri, Subir Kumar, Muralidhar Panigrahi, Shaikat Mondal, Joshil Kumar Behera, and Himel Mondal. 2023. Assessing the utilization of large language models in medical education: Insights from undergraduate medical students. Cureus, 15(10). · DOI · Paper page
  6. Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika, 16(3):297-334. · DOI · Paper page
  7. Joshua Davis, Liesbet Van Bulck, Brigitte N Durieux, Charlotta Lindvall, et al. 2024. The temperature feature of chatgpt: modifying creativity for clinical research. JMIR Human Factors, 11(1):e53559. · DOI
  8. Michael Desmond, Zahra Ashktorab, Qian Pan, Casey Dugan, and James M Johnson. 2024. Evalullm: Llm assisted evaluation of generative outputs. In Compan- ion Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 30-32. · DOI
  9. Xue-Yong Fu, Md Tahmid Rahman Laskar, Cheng Chen, and Shashi Bhushan Tn. 2023b. Are large language models reliable judges? a study on the factuality eval- uation capabilities of LLMs. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 310-316, Sin- gapore. Association for Computational Linguistics.
  10. Kevin A Hallgren. 2012. Computing inter-rater relia- bility for observational data: an overview and tuto- rial. Tutorials in quantitative methods for psychology, 8(1):23. · DOI · Paper page
  11. Johannes Hellrich and Udo Hahn. 2016. Bad com- pany-neighborhoods in neural embedding spaces considered harmful. In Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers, pages 2785-2796.
  12. Robin K Henson. 2001. Understanding internal consis- tency reliability estimates: A conceptual primer on coefficient alpha. Measurement and evaluation in counseling and development, 34(3):177-189. · DOI
  13. Lisa Kollitsch, Klaus Eredics, Martin Marszalek, Michael Rauchenwald, Sabine D Brookman-May, Maximilian Burger, Katharina Körner-Riffard, and Matthias May. 2024. How does artificial intelligence master urological board examinations? a comparative analysis of different large language models' accuracy and reliability in the 2022 in-service assessment of the european board of urology. World Journal of Urology, 42(1):20. · DOI
  14. Yan Liu, Amery D Wu, and Bruno D Zumbo. 2010. The impact of outliers on cronbach's coefficient al- pha estimate of reliability: Ordinal/rating scale item responses. Educational and Psychological Measure- ment, 70(1):5-21. · DOI
  15. Camila Paola Malkewitz, Philipp Schwall, Christian Meesters, and Jochen Hardt. 2023. Estimating reli- ability: A comparison of cronbach's α, mcdonald's ωt and the greatest lower bound. Social Sciences & Humanities Open, 7(1):100368. · DOI
  16. Roderick P McDonald. 2013. Test theory: A unified treatment. psychology press.
  17. Brent Mittelstadt, Sandra Wachter, and Chris Russell. 2023. To protect science, we must use llms as zero-shot translators. Nature Human Behaviour, 7(11):1830-1832. · DOI · Paper page
  18. Stefanie Nowak and Stefan Rüger. 2010. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image an- notation. In Proceedings of the international con- ference on Multimedia information retrieval, pages 557-566. · DOI
  19. Jonas Rieger, Carsten Jentsch, and Jörg Rahnenführer. 2024. Ldaprototype: A model selection algorithm to improve reliability of latent dirichlet allocation. PeerJ Computer Science, 10:e2279. · DOI
  20. Ruoxi Shang, Gary Hsieh, and Chirag Shah. 2024. Trusting your ai agent emotionally and cognitively: Development and validation of a semantic differential scale for ai trust. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1343-1356. · DOI
  21. Kenneth Stensen and Stian Lydersen. 2022. Internal consistency: from alpha to omega. Tidsskrift for den Norske Laegeforening: Tidsskrift for Praktisk Medicin, ny Raekke, 142(12). · DOI
  22. Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Gai. 2024. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729-9750. · DOI
  23. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, and et al. 2024b. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295. · arXiv
  24. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.
  25. Sagar K Wadkar, Khajan Singh, Ritu Chakravarty, and Shivaji D Argade. 2016. Assessing the reliability of attitude scale by cronbach's alpha. Journal of Global Communication, 9(2):113-117. · DOI
  26. Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. 2024. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. npj Digital Medicine, 7(1):41. · DOI
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38-45. · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Schroeder, K., & Wood-Doughty, Z.
Affiliations
Northwestern University (US)
ORCID
Wood-Doughty, Zach
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.12509
arXiv
2412.12509
OpenAlex
W4405562067
Cited by
8
References
0
Citations by year
2026: 4 · 2025: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
LLM-as-a-judge, large language models, McDonald's omega, inter-rater reliability, temperature sampling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-03.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-03.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review