CoChat
Sign Up
PreprintCited by 72Topic ModelingNatural Language Processing TechniquesEthics and Social Impacts of AI

Large Language Models are not Fair Evaluators

Open access
Wang, Peiyi, Li, Lei, Chen, Liang, Cai, Zefan, Zhu, Dawei, Lin, Binghuai, Cao, Yunbo, Liu, Qi +2 more·2023·arXiv
DOI 10.48550/arxiv.2305.17926arXiv 2305.17926
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we uncover a systematic bias in the evaluation paradigm of adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. To address this issue, we propose a calibration framework with three simple yet effective strategies: 1) Multiple Evidence Calibration, which requires the evaluator model to generate multiple evaluation evidence before assigning ratings; 2) Balanced Position Calibration, which aggregates results across various orders to determine the final score; 3) Human-in-the-Loop Calibration, which introduces a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when needed. We also manually annotate the "win/tie/lose" outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and extensive experiments demonstrate that our approach successfully mitigates evaluation bias, resulting in closer alignment with human judgments. We release our code and human annotation at \url{https://github.com/i-Eval/FairEval} to facilitate future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.17926

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLM-based evaluations are sensitive to prompt phrasing, label order, response length, and position.

    “LLM-based evaluations are sensitive to prompt phrasing (Zhou et al., 2023), label order (Wang et al., 2023), response length (Stureborg et al., 2024), and position (Shi et al., 2024;Kudugunta et al., 2023).”

    From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Wang et al., 2023

    Section 2.2, “Revealing the Positional Bias,” shows that swapping candidate responses between the Assistant 1 and Assistant 2 positions changes LLM evaluation results. That supports sensitivity to response order, but the paper does not establish sensitivity to the order of evaluation labels—the specific point attributed to Wang et al. here. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Human feedback in RLHF can become costly.

    “However, issues arise with RLHF where human feedback can become costly.”

    From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Wang et al., 2023

    Section 6.1 (page unavailable) says human evaluation is costly and time-consuming at scale, and Section 4.3 reports annotation costs. These passages concern evaluating model responses, not human feedback specifically within RLHF, so the RLHF qualification is unsupported. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In the paper's case, evaluation cost around $0.05 each, which is 1/15th compared to human feedback in Wang et al. (2023).

    “This brings down the cost significantly; in our case, evaluation cost around $0.05 each, which is 1/15th compared to human feedback in the previously cited paper (Wang et al., 2023).”

    From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Wang et al., 2023

    Section 4.1 describes human evaluation of 80 examples, and Table 4 in Section 4.3 lists a $30 cost per human annotator. That is $0.375 per annotation, not the $0.75 implied by calling $0.05 one-fifteenth of the human-feedback cost. The $0.05 figure is the citing authors’ own cost, not a figure attributed to Wang et al.; the paper supports the existence of a human-feedback cost comparison, but not the stated ratio. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Systematic biases of single judges (position, verbosity, self-enhancement, sycophancy, and refusal artefacts) are well documented.

    “The systematic biases its backbone exhibits, e.g., position, verbosity, self-enhancement, sycophancy, and refusal artefacts, are by now well documented (Wang et al., 2023;Panickssery et al., 2024;Saito et al., 2023;Stureborg et al., 2024); they propagate uncorrected to every score, and the cost-quality profile of the resulting evaluation is fixed to that of the single model.”

    From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Wang et al., 2023

    Section 2.2, “Revealing the Positional Bias” (page not provided), documents position-dependent judgments by GPT-4 and ChatGPT, supporting the position-bias example. The passages found do not document verbosity, self-enhancement, sycophancy, or refusal artefacts, so this source supports only part of the cited list. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • LLMs might not be a fair judge if different LLMs are used for agents.

    “we find that LLMs might not be a fair judge if different LLMs are used for agents.”

    From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Wang et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 37

  1. Belinkov, Y.; Poliak, A.; Shieber, S.; Van Durme, B.; and Rush, A. 2019. Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics (ACL).
  2. Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Ka- plan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language Models are Few-Shot Learners. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  3. Cai, Z.; Chang, B.; and Han, W. 2023. Human-in- the-Loop through Chain-of-Thought. arXiv preprint arXiv:2306.07932. · arXiv
  4. Cai, Z.; Tu, L.; and Gimpel, K. 2017. Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (ACL).
  5. Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; Schuh, P.; Shi, K.; Tsvyashchenko, S.; Maynez, J.; Rao, A.; Barnes, P.; Tay, Y.; Shazeer, N. M.; Prabhakaran, V.; Reif, E.; Du, N.; Hutchinson, B. C.; Pope, R.; Bradbury, J.; Austin, J.; Isard, M.; Gur-Ari, G.; Yin, P.; Duke, T.; Levskaya, A.; Ghemawat, S.; Dev, S.; Michalewski, H.; García, X.; Misra, V.; Robinson, K.; Fedus, L.; Zhou, D.; Ippolito, D.; Luan, D.; Lim, H.; Zoph, B.; Spiridonov, A.; Sepassi, R.; Dohan, D.; Agrawal, S.; Omernick, M.; Dai, A. M.; Pillai, T. S.; Pellat, M.; Lewkowycz, A.; Moreira, E.; Child, R.; Polozov, O.; Lee, K.; Zhou, Z.; Wang, X.; Saeta, B.; Díaz, M.; Firat, O.; Catasta, M.; Wei, J.; Meier-Hellstern, K. S.; Eck, D.; Dean, J.; Petrov, S.; and Fiedel, N. 2022. PaLM: Scaling Language Modeling with Pathways. ArXiv, abs/2204.02311. · arXiv
  6. Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; and Sui, Z. 2022. A Survey for In- context Learning. arXiv preprint arXiv:2301.00234. · arXiv
  7. Dubois, Y.; Li, X.; Taori, R.; Zhang, T.; Gulrajani, I.; Ba, J.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387. · arXiv
  8. Gao, P.; Han, J.; Zhang, R.; Lin, Z.; Geng, S.; Zhou, A.; Zhang, W.; Lu, P.; He, C.; Yue, X.; Li, H.; and Qiao, Y. J. 2023. LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. ArXiv, abs/2304.15010. · arXiv
  9. Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  10. Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annota- tion Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics.
  11. He, T.; Zhang, J.; Wang, T.; Kumar, S.; Cho, K.; Glass, J.; and Tsvetkov, Y. 2023. On the Blind Spots of Model-Based Evaluation Metrics for Text Generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12067-12097. Toronto, Canada: Association for Computational Linguistics.
  12. Levy, O.; Remus, S.; Biemann, C.; and Dagan, I. 2015. Do Supervised Distributional Methods Really Learn Lexical Inference Relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics.
  13. Li, L.; Yin, Y.; Li, S.; Chen, L.; Wang, P.; Ren, S.; Li, M.; Yang, Y.; Xu, J.; Sun, X.; et al. 2023. M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. arXiv preprint arXiv:2306.04387. · arXiv
  14. Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74-81. Barcelona, Spain: Association for Computational Linguistics.
  15. Liu, T.; Xin, Z.; Chang, B.; and Sui, Z. 2020a. HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference. In Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association. ISBN 979-10- 95546-34-4.
  16. Liu, T.; Xin, Z.; Ding, X.; Chang, B.; and Sui, Z. 2020b. An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference. In Proceedings of the 24th Conference on Computa- tional Natural Language Learning. Online: Associa- tion for Computational Linguistics.
  17. Lu, Q.; Qiu, B.; Ding, L.; Xie, L.; and Tao, D. 2023. Error analysis prompting enables human-like trans- lation evaluation in large language models: A case study on chatgpt. arXiv preprint arXiv:2303.13809. · arXiv
  18. McCoy, T.; Pavlick, E.; and Linzen, T. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuris- tics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
  19. McHugh, M. L. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3): 276-282.
  20. Min, S.; Wallace, E.; Singh, S.; Gardner, M.; Hajishirzi, H.; and Zettlemoyer, L. 2019. Compositional Ques- tions Do Not Necessitate Multi-hop Reasoning. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL).
  21. OpenAI. 2022. Introducing ChatGPT. OpenAI. 2023. GPT-4 Technical Report. CoRR, abs/2303.08774. · arXiv
  22. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Ma- chine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Lin- guistics, 311-318. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics.
  23. Peng, B.; Li, C.; He, P.; Galley, M.; and Gao, J. 2023. Instruction Tuning with GPT-4. ArXiv, abs/2304.03277. · arXiv
  24. Schwartz, R.; Sap, M.; Konstas, I.; Zilles, L.; Choi, Y.; and Smith, N. A. 2017. The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task. In Proceedings of the 21st Conference on Computational Natural Lan- guage Learning (CoNLL).
  25. Song, Y.; Wang, P.; Zhu, D.; Liu, T.; Sui, Z.; and Li, S. 2023a. RepCL: Exploring Effective Representation for Continual Text Classification. arXiv preprint arXiv:2305.07289. · arXiv
  26. Song, Y.; Xiong, W.; Zhu, D.; Li, C.; Wang, K.; Tian, Y.; and Li, S. 2023b. RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs. arXiv preprint arXiv:2306.06624. · arXiv
  27. Sun, Z.; Shen, Y.; Zhou, Q.; Zhang, H.; Chen, Z.; Cox, D. D.; Yang, Y.; and Gan, C. 2023. Principle- Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. ArXiv, abs/2305.03047. · arXiv
  28. Turpin, M.; Michael, J.; Perez, E.; and Bowman, S. R. 2023. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of- Thought Prompting. CoRR, abs/2305.04388. · arXiv
  29. Wang, P.; Song, Y.; Liu, T.; Lin, B.; Cao, Y.; Li, S.; and Sui, Z. 2022. Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, 6264-6278. Abu Dhabi, United Arab Emirates: Association for Computational Lin- guistics.
  30. Wang, P.; Xun, R.; Liu, T.; Dai, D.; Chang, B.; and Sui, Z. 2021. Behind the Scenes: An Exploration of Trig- ger Biases Problem in Few-Shot Event Classification. In Proceedings of the 30th ACM International Con- ference on Information & Knowledge Management, 1969-1978.
  31. Wang, Y.; Ivison, H.; Dasigi, P.; Hessel, J.; Khot, T.; Chandu, K. R.; Wadden, D.; MacMillan, K.; Smith, N. A.; Beltagy, I.; et al. 2023a. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. arXiv preprint arXiv:2306.04751. · arXiv
  32. Wang, Y.; Yu, Z.; Zeng, Z.; Yang, L.; Wang, C.; Chen, H.; Jiang, C.; Xie, R.; Wang, J.; Xie, X.; et al. 2023b. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimiza- tion. arXiv preprint arXiv:2306.05087. · arXiv
  33. Xia, H.; Wang, P.; Liu, T.; Lin, B.; Cao, Y.; and Sui, Z. 2023. Enhancing Continual Relation Extrac- tion via Classifier Decomposition. In Findings of the Association for Computational Linguistics: ACL 2023, 10053-10062. Toronto, Canada: Association for Computational Linguistics.
  34. Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023. Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data. ArXiv, abs/2304.01196. · arXiv
  35. Yuan, W.; Neubig, G.; and Liu, P. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 27263-27277.
  36. Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020. BERTScore: Evaluating Text Genera- tion with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  37. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685. · arXiv

Cited by

Sorted by citations

Related

Authors
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z.
Affiliations
  • Peking University (CN)
  • Tencent (China) (CN)
  • University of Hong Kong (HK)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.17926
arXiv
2305.17926
OpenAlex
W4378770815
Cited by
34
References
0
Citations by year
2026: 6 · 2025: 3 · 2024: 7 · 2023: 18
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
LLM evaluation, evaluation bias, LLM-as-a-judge, response ranking, human-in-the-loop evaluation, human judgment alignment, ChatGPT

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review