Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
Open accessAbstract
LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In practice, LLM judges exhibit correlated errors caused by shared latent confounders such as verbosity, stylistic preferences, or training artifacts.
“in practice, LLM judges exhibit correlated errors caused by shared latent confounders-such as verbosity, stylistic preferences, or training artifactscausing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes.”
From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Shi et al., 2024
Section C.1 (page unavailable) finds familial patterns in judges’ choices, but does not establish correlated errors caused by verbosity, stylistic preferences, or training artifacts. More decisively, Section C.2 (page unavailable) says majority voting enhances reliability and could be useful for over 95% of evaluation instances on the two benchmarks—the opposite of the citing sentence’s claim that aggregation provides little gain or amplifies mistakes. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredLLM judges often exhibit correlated errors due to shared training data or modeling choices.
“Moreover, these judges often exhibit correlated errors due to shared training data or modeling choices (Ye et al., 2025;Shi et al., 2024;Wang et al., 2024b;Deutsch et al., 2022;Li et al., 2025).”
From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Shi et al., 2024
Appendix C.1 reports high agreement among some related LLM judges and suggests that similar training data and strategies may influence their positional preferences. It does not establish that judges often make correlated errors, and it presents the proposed cause as a possibility rather than a demonstrated explanation. (LLM, conf=0.85)
Experimental — accuracy not yet measuredLLMs can act as inexpensive and fast proxies for human raters by assessing model generations via scalar quality scores or pairwise preferences.
“LLMs can act as inexpensive and fast proxies for human raters by assessing model generations via (i) scalar quality scores (e.g., 1-10 Likert or percentiles) (Zhu et al., 2025;Wang et al., 2024b;Shi et al., 2024), or (ii) pairwise preferences indicating which of two responses is better, as widely used in RLHF pipelines (Ouyang et al., 2022;Bai et al., 2022).”
From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Shi et al., 2024
The Introduction describes LLM judges as an alternative to costly human evaluation, and Related Work (B.1) says they can score or rank model outputs. Those passages support using LLMs as lower-cost proxies, including score-based evaluation, but do not establish that they are fast or specify 1–10 Likert scores or percentiles. In the citing sentence, Shi et al. is attached to the scalar-score clause, not the separately cited RLHF pairwise-preference clause. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 33
- Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. Preprint, arXiv:2308.07201. · arXiv
- Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun. 2024. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision- language benchmark. Preprint, arXiv:2402.04788. · arXiv
- Cheng-Han Chiang and Hung-yi Lee. 2023a. Can large language models be an alternative to human evalua- tions? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15607-15631, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
- Cheng-Han Chiang and Hung-Yi Lee. 2023b. Can large language models be an alternative to human evalua- tions? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15607-15631.
- James Chua, Edward Rees, Hunar Batra, Samuel R. Bowman, Julian Michael, Ethan Perez, and Miles Turpin. 2024. Bias-augmented consistency train- ing reduces biased reasoning in chain-of-thought. Preprint, arXiv:2403.05518. · arXiv
- Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1-79.
- Google Gemini Team. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805. · arXiv
- Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. 2024. Bias in large language models: Origin, evaluation, and mitigation. arXiv preprint arXiv:2411.10915. · arXiv
- Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364-381. Springer.
- Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using mechanical turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285.
- Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more per- suasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782. · arXiv
- Tom Kocmi and Christian Federmann. 2023. Large lan- guage models are state-of-the-art evaluators of trans- lation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193-203, Tampere, Finland. Euro- pean Association for Machine Translation.
- Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. 2024a. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604. · arXiv
- Ruosen Li, Teerth Patel, and Xinya Du. 2023b. Prd: Peer rank and discussion improve large lan- guage model based evaluations. arXiv preprint arXiv:2307.02762. · arXiv
- Yuran Li, Jama Hussein Mohamud, Chongren Sun, Di Wu, and Benoit Boulet. 2025. Leveraging llms as meta-judges: A multi-agent framework for evaluating llm judgments. Preprint, arXiv:2504.17087. · arXiv
- Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2023c. Split and merge: Aligning position biases in large language model based evaluators. arXiv preprint arXiv:2310.01432. · arXiv
- Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024b. Split and merge: Aligning position biases in llm- based evaluators. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 11084-11108.
- Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. Llm comparative assessment: Zero-shot nlg evalua- tion through pairwise comparisons using large lan- guage models. In Proceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139-151.
- Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yan- ing Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, and Soroush Vosoughi. 2025. Judging with many minds: Do more perspectives mean less preju- dice? arXiv preprint arXiv:2505.19477. · arXiv
- Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. arXiv preprint arXiv:2402.15987. · arXiv
- OpenAI(2023). 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. · arXiv
- Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effec- tive text rankers with pairwise ranking prompting. Preprint, arXiv:2306.17563. · arXiv
- Vyas Raina, Adian Liusie, and Mark Gales. 2024. Is llm-as-a-judge robust? investigating universal adver- sarial attacks on zero-shot llm assessment. Preprint, arXiv:2402.14016. · arXiv
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971. · arXiv
- Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. Preprint, arXiv:2305.17926. · arXiv
- Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. 2025. Eliminating position bias of lan- guage models: A mechanistic approach. In The Thir- teenth International Conference on Learning Repre- sentations.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems, 35:24824-24837.
- Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sain- bayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a- meta-judge. Preprint, arXiv:2407.19594. · arXiv
- Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. Preprint, arXiv:2410.02736. · arXiv
- Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023. Evaluating large lan- guage models at evaluating instruction following. In The Twelfth International Conference on Learning Representations.
- Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024a. Large language models are not robust multiple choice selectors. Preprint, arXiv:2309.03882. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024b. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
- Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631. · arXiv
Cited by
Sorted by citations
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Qian, K., Liu, S., Li, T., Raković, M., Li, X., Guan, R., Molenaar, I., Nawaz, S., Swiecki, Z., Yan, L., et al. (2026) · Computers & Education · doi:10.1016/j.compedu.2025.105448 · Cited by 39 · see citing works
“In this study, we employed the pairwise comparison type of LLM Judge agent that has been widely discussed and tested (Zheng et al., 2024; Shi et al., 2024; Chen et al., 2024).”
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11 · see citing works
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4 · see citing works
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data
Dorner, F. E., Nastl, V. Y., & Hardt, M. (2024) · arXiv · doi:10.48550/arxiv.2410.13341
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2604.22891
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Zhao, J., Shin, C., Huang, T.-H., GNVV, S. S. S. N., & Sala, F. (2026) · arXiv · doi:10.48550/arxiv.2603.00039
- Authors
- Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S.
- Affiliations
- Dartmouth College (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.07791
- OpenAlex
- W4399694266
- Cited by
- 72
- References
- 0
- Citations by year
- 2026: 36 · 2025: 36
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- LLM-as-a-judge, position bias, LLM evaluation, pairwise comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
