Don't Make Your LLM an Evaluation Benchmark Cheater
Open accessAbstract
Large language models~(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity. To assess the model performance, a typical approach is to construct evaluation benchmarks for measuring the ability level of LLMs in different aspects. Despite that a number of high-quality benchmarks have been released, the concerns about the appropriate use of these benchmarks and the fair comparison of different models are increasingly growing. Considering these concerns, in this paper, we discuss the potential risk and impact of inappropriately using evaluation benchmarks and misleadingly interpreting the evaluation results. Specially, we focus on a special issue that would lead to inappropriate evaluation, \ie \emph{benchmark leakage}, referring that the data related to evaluation sets is occasionally used for model training. This phenomenon now becomes more common since pre-training data is often prepared ahead of model test. We conduct extensive experiments to study the effect of benchmark leverage, and find that it can dramatically boost the evaluation results, which would finally lead to an unreliable assessment of model performance. To improve the use of existing evaluation benchmarks, we finally present several guidelines for both LLM developers and benchmark maintainers. We hope this work can draw attention to appropriate training and evaluation of LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current evaluation paradigms are often compromised by the inadvertent inclusion of benchmark-related data, which inflates performance metrics and obscures models' true capacities.
“As we scale up pre-training data, current evaluation paradigms are often compromised by the inadvertent inclusion of benchmark-related data [341,351], which inflates performance metrics and obscures models' true capacities.”
From A Survey of Large Language Models (2026) · cites this paper as [351]
Benchmark Data Contamination (BDC), the inclusion of benchmark testing samples in the training set, results in falsely inflated performance metrics and undermines the reliability of evaluation conclusions.
“Benchmark Data Contamination (BDC)-the inclusion of benchmark testing samples in the training set-has raised increasing concerns in Large Language Model (LLM) evaluation, leading to falsely inflated performance estimates and undermining evaluation reliability.”
From The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (2025) · cites this paper as Zhou et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong- Yeol Ahn. 2023. Can we trust the evaluation on chatgpt? arXiv preprint arXiv:2303.12767. · arXiv
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. CoRR, abs/2107.03374. · arXiv
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. CoRR, abs/2110.14168. · arXiv
- Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027. · arXiv
- Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in llms: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493. · arXiv
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children's books with explicit memory representa- tions. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. arXiv preprint arXiv:2310.18018. · arXiv
- Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Self-citationDatasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024) · arXiv · doi:10.48550/arxiv.2410.12405 · Cited by 78
“Previous research (Zhu et al., 2023; Pezeshkpour and Hruschka, 2024) compares different prompt templates within datasets (composed of instances) to analyze the prompt sensitivity of LLMs, which is done by examining the shifts in the LLMs’ scores across different prompt templates within the dataset.”
BackgroundMethodLooGLE: Can Long-Context Language Models Understand Long Contexts?
Li, J., Wang, M., Zheng, Z., & Zhang, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.859 · Cited by 55 · see citing works
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · arXiv · doi:10.48550/arxiv.2402.15938 · Cited by 52
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.716 · Cited by 52 · see citing works
Task Contamination: Language Models May Not Be Few-Shot Anymore
Li, C., & Flanigan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29808 · Cited by 47 · see citing works
Liu, M., Hu, W., Ding, J., Xu, J., Li, X., Zhu, L., Bai, Z., Shi, X., Wang, B., Song, H., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020044 · Cited by 41 · see citing works
Zhang, J., Wu, P., London, J., & Tenney, D. (2025) · IEEE Access · doi:10.1109/access.2025.3540075 · Cited by 30 · see citing works
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P. J. F., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.726 · Cited by 19 · see citing works
Zhu, J., Zhou, Y., Ioannidis, V. N., Qian, S., Ai, W., Song, X., & Koutra, D. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635786 · Cited by 18 · see citing works
VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation
Vijayaraghavan, P., Shi, L., Ambrogio, S., Mackin, C., Nitsure, A., Beymer, D., & Degan, E. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691836 · Cited by 18 · see citing works
Benchmark suites instead of leaderboards for evaluating AI fairness
Wang, A., Hertzmann, A., & Russakovsky, O. (2024) · Patterns · doi:10.1016/j.patter.2024.101080 · Cited by 17 · see citing works
Poretsky, E., Blake, V. C., Andorf, C. M., & Sen, T. Z. (2025) · Database · doi:10.1093/database/baaf011 · Cited by 16 · see citing works
A comparison of chain-of-thought reasoning strategies across datasets and models
Hebenstreit, K., Praas, R., Kiesewetter, L. P., & Samwald, M. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1999 · Cited by 15 · see citing works
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.20 · Cited by 37 · see citing works
Yang, Y., Li, J., Guo, J., Pang, P. C., Wang, Y., Yang, X., & Im, S. (2025) · IEEE Access · doi:10.1109/access.2025.3549733 · Cited by 14 · see citing works
Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey
Sakib, M. N., Islam, M. A., Pathak, R., & Arifin, M. M. (2024) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings63359.2024.10863356 · Cited by 13 · see citing works
Moëll, B., Farestam, F., & Beskow, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1557920 · Cited by 12 · see citing works
Bringing AI participation down to scale
Moats, D., & Ganguly, C. (2025) · Patterns · doi:10.1016/j.patter.2025.101241 · Cited by 9 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5 · see citing works
“Unlike previous studies that assess data contamination through the correlation between question entropy and accuracy (Hendrycks et al., 2020; Wei et al., 2023b; Zhou et al., 2023), our method directly controls visibility by considering both the training date and the publication date.”
MethodSelf-citationLiu, P., Chen, Z.-Z., Li, Y.-H., & Wong, W. E. (2025) · 2025 11th International Symposium on System Security, Safety, and Reliability (ISSSR) · doi:10.1109/isssr65654.2025.00047 · Cited by 3 · see citing works
Self-citationLarge Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2 · see citing works
Harnessing Large Language Models for Satellite Ground Tests
Connolly, B. J., & Anderson, K. M. (2024) · AIAA SciTech Forum · doi:10.2514/6.2024-0916 · Cited by 2 · see citing works
Connolly, B. J. (2024) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.3012352 · Cited by 2 · see citing works
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1 · see citing works
In Which Areas of Technical AI Safety Could Geopolitical Rivals Cooperate?
Bucknall, B., Siddiqui, S., Thurnherr, L., McGurk, C., Harack, B., Reuel, A., Paskov, P., Mahoney, C., Mindermann, S., Singer, S., et al. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732201 · Cited by 1 · see citing works
Tong, Z., Sun, F., & Nguyen, L.-M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_14 · Cited by 1 · see citing works
CODEMORPH: Mitigating Data Leakage in Large Language Model Assessment
Rao, H., Zhao, Y., Zhu, W., Xiao, L., Wang, M., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00081 · Cited by 1 · see citing works
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
“This phenomena, known as Benchmark Data Contamination (BDC) , results in falsely inflated performance metrics, thereby undermining the reliability of evaluation conclusions (Zhou et al., 2023; Jiang et al., 2024; Sainz et al., 2023).”
Evaluation of Deep Audio Representations for Hearables
Gröger, F., Baumann, P., Amruthalingam, L., Simon, L., Giurda, R., & Lionetti, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887737
Huang, Z., Long, R., Chen, H., Na, J., Li, Q., Jiang, S., & Sun, Q. (2025) · Research Square · doi:10.21203/rs.3.rs-6350858/v1
Feasibility Study on Reactor Preliminary Design via Fine-Tuning of Large Language Models
Zhang, B., Lai, W., Huang, H., Ren, J., Chen, Q., Zhang, B., & He, C. (2026) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.6c00091
Accountability Challenges in AI-Automated Life Cycle Assessments
Preuss, N., & You, F. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812254
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationEstablishing Best Practices for Building Rigorous Agentic Benchmarks
Zhu, Y., Jin, T., Pruksachatkun, Y., Zhang, A., Liu, S., Cui, S., Kapoor, S., Longpre, S., Meng, K., Weiss, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.02825 · Cited by 2 · see citing works
Self-citation
- Authors
- Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J.
- Affiliations
- Renmin University of China (CN)
- University of Illinois Urbana-Champaign (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.01964
- OpenAlex
- W4388444775
- Cited by
- 25
- References
- 0
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, evaluation benchmark, training data leakage, assessment fairness, model evaluation, pre-training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
