CoChat
Sign Up
PreprintCited by 11Reliability and Agreement in MeasurementNatural Language Processing TechniquesTopic Modeling

Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale

Open access
Li, Weiyue, Zhao, Minda, Dong, Weixuan, Cai, Jiahui, Wei, Yuze, Pocress, Michael, Li, Yi, Yuan, Wanyan +7 more·2026·arXiv
DOI 10.48550/arxiv.2601.03444arXiv 2601.03444
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.03444

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two scoring granularities (1-10 and 1-5 scales) are considered to analyze the effect of resolution on structured evaluation stability.

    “Two scoring granularities are considered (1-10 and 1-5 scales [9]), allowing us to analyze the effect of resolution on structured evaluation stability.”

    From Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge (2026) · cites this paper as [9]

    The paper’s Abstract says it studies scale choice and consistency, but specifies continuous 0–5, 0–10, and 0–100 ranges—not the cited 1–5 and 1–10 scales. The citing authors’ plan to analyze structured evaluation stability is their own, rather than a finding attributed to this source. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 45

  1. Chiang, C.-H. & Lee, H.-y. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023). · arXiv
  2. Liu, Y. et al. G-Eval: NLG evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023). · arXiv
  3. Fu, J., Ng, S. K., Jiang, Z. & Liu, P. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 6556-6576 (2024).
  4. Zheng, L. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Adv. Neural Inf. Process. Syst. 36, 46595-46623 (2023).
  5. Lee, N., Hong, J., & Thorne, J. Evaluating the consistency of LLM evaluators. In Proceedings of the 31st International Conference on Computational Linguistics, 10650-10659 (2025).
  6. Guerdan, L. et al. Validating LLM-as-a-judge systems under rating indeterminacy https://arxiv.org/abs/2503.05965. 2503.05965 (2025). · arXiv
  7. Chan, C.-M. et al. Chateval: Towards better LLM-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023). · arXiv
  8. Wei, T., Wen, W., Qiao, R., Sun, X. & Ma, J. Rocketeval: Efficient automated LLM evaluation via grading checklist. arXiv preprint arXiv:2503.05142 (2025). · arXiv
  9. Xu, Y., Ruis, L., Rocktäschel, T. & Kirk, R. Investigating non-transitivity in LLM-as-a-judge. arXiv preprint arXiv:2502.14074 (2025). · arXiv
  10. Shrout, P. E. & Fleiss, J. L. Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420 (1979).
  11. Gilardi, F., Alizadeh, M. & Kubli, M. ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. USA. 120, e2305016120 (2023).
  12. Kocmi, T. & Federmann, C. Large language models are state-of-the- art evaluators of translation quality. arXiv preprint arXiv:2302.14520 (2023). · arXiv
  13. Zhuo, T. Y. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, 2232-2242 (2024).
  14. Kim, S. et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations (2023).
  15. Zhu, L., Wang, X., & Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631 (2023). · arXiv
  16. Li, J. et al. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470 (2023). · arXiv
  17. Chan, D., Petryk, S., Gonzalez, J., Darrell, T. & Canny, J. Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 13638-13646 (2023).
  18. Lee, Y., Park, I. & Kang, M. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004 (2024). · arXiv
  19. Chen, D. et al. Mllm-as-a-judge: Assessing multimodal llm-as-a- judge with vision-language benchmark. In Forty-first International Conference on Machine Learning (2024).
  20. Wu, T. et al. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 22227-22238 (2024).
  21. Xu, W. et al. Instructscore: Explainable text generation evaluation with finegrained feedback. arXiv preprint arXiv:2305.14282 (2023). · arXiv
  22. Vu, T. et al. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.10817 (2024). · arXiv
  23. Wang, T. et al. Self-taught evaluators. arXiv preprint arXiv:2408.02666 (2024). · arXiv
  24. Zhuge, M. et al. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934 (2024). · arXiv
  25. Lai, P. et al. Beyond the surface: Enhancing llm-as-a-judge alignment with human via internal representations. arXiv preprint arXiv:2508.03550 (2025). · arXiv
  26. Wang, P. et al. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9440-9450 (2024).
  27. Koo, R. et al. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, 517-545 (2024).
  28. Saito, K., Wachi, A., Wataoka, K. & Akimoto, Y. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076 (2023). · arXiv
  29. Wu, M. & Aji, A. F. Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, 297-312 (2025).
  30. Panickssery, A., Bowman, S. & Feng, S. LLM evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772-68802 (2024).
  31. Li, Q., Dou, S., Shao, K., Chen, C. & Hu, H. Evaluating scoring bias in LLM-as-a-judge. arXiv preprint arXiv:2506.22316 (2025). · arXiv
  32. Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. Length- controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 (2024). · arXiv
  33. Achiam, J. et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  34. Huang, H. et al. On the limitations of fine-tuned judge models for LLM evaluation. arXiv preprint arXiv:2403.02839 (2024). · arXiv
  35. Feng, Y., Wang, S., Cheng, Z., Wan, Y. & Chen, D. Are we on the right way to assessing LLM-as-a-judge? arXiv preprint arXiv:2512.16041 (2025). · arXiv
  36. Novikova, J., Dušek, O. & Rieser, V. Rankme: Reliable human ratings for natural language generation. arXiv preprint arXiv:1803.05928 (2018). · arXiv
  37. Van Der Lee, C., Gatt, A., Van Miltenburg, E., Wubben, S. & Krahmer, E. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th international conference on natural language generation, 355-368 (2019).
  38. Bedi, S. et al. Medhelm: Holistic evaluation of large language models for medical tasks. arXiv preprint arXiv:2505.23802 (2025). · arXiv
  39. Zhao, Z. et al. Evolution without an oracle: Driving effective evolution with llm judges. arXiv preprint arXiv:2511.19489 (2025). · arXiv
  40. Zeng, Z. et al. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641 (2023). · arXiv
  41. Liu, Y., Liu, P. & Cohan, A. On evaluating LLM alignment by evaluating LLMs as judges. arXiv preprint arXiv:2511.20604 (2025). · arXiv
  42. Li, Z. et al. Leveraging large language models for NLG evaluation: Advances and challenges. arXiv preprint arXiv:2401.07103 (2024). · arXiv
  43. Li, D. et al. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2757-2791 (2025).
  44. Gu, J. et al. A survey on LLM-as-a-judge https://arxiv.org/abs/2411. 15594. 2411.15594 (2025).
  45. Chang, Y. et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15, 1-45 (2024).

Cited by

Sorted by citations

Related

Authors
Li, W., Zhao, M., Dong, W., Cai, J., Wei, Y., Pocress, M., Li, Y., Yuan, W., Wang, X., Hou, R., Lou, K., Zeng, W., Yang, Y., Du, Y., & Wang, M.
Affiliations
  • Harvard University (US)
  • Carnegie Mellon University (US)
  • Stanford University (US)
  • University of California San Diego (US)
Corresponding author
Wang, Mengyu
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
3005-1460 (ISSN-L)
DOI
10.48550/arxiv.2601.03444
arXiv
2601.03444
OpenAlex
W7119510601
Cited by
4
References
0
Citations by year
2026: 4
FWCI
19.10
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Artificial Intelligence (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Institutes of Health (1r01ey036222-01)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM-as-a-judge, human-LLM alignment, automated assessment, inter-rater reliability, gender differences in evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review