Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Open accessAbstract
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks.Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination.However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges.In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs.CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution.To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution.To facilitate this study, we introduce two benchmarks, i.e., DET-CON and COMIEVAL, for data contamination detection and contamination mitigation evaluation tasks.Extensive experimental results show that CDD achieves the average relative improvements of 21.8%-30.2%over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination.TED substantially mitigates performance improvements up to 66.9% attributed to data contamination across various contamination setups.In realworld applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities.
“The disparity between LLM performance on mARC-QA and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities19,54–57.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · see citing works
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
Self-collaboration Code Generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2023) · arXiv · doi:10.48550/arxiv.2304.07590 · Cited by 256 · see citing works
Self-citationRethinking Benchmark and Contamination for Language Models with Rephrased SamplesPreprint
Yang, S., Chiang, W.-L., Zheng, L.-M., Gonzalez, J. E., & Stoica, I. (2023) · arXiv.org · doi:10.48550/arxiv.2311.04850 · Cited by 242 · see citing works
Li, Y., Guerin, F., & Lin, C.-H. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2312.12343 · Cited by 77 · see citing works
Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model EvaluationPreprint
Li, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2309.10677 · Cited by 59 · see citing works
Competition-Level Problems are Effective LLM EvaluatorsPreprint
Huang, Y.-M., Lin, Z.-W., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2312.02143 · Cited by 38 · see citing works
Detecting Pretraining Data from Large Language Models
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2310.16789 · Cited by 37 · see citing works
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2023) · arXiv · doi:10.48550/arxiv.2308.08493 · Cited by 29 · see citing works
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · see citing works
CLEAN-EVAL: Clean Evaluation on Contaminated Large Language ModelsPreprint
Zhu, W., Hao, H.-P., He, Z., et al. (2023) · NAACL-HLT · doi:10.48550/arxiv.2311.09154 · Cited by 16 · see citing works
Proving Test Set Contamination in Black Box Language Models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2310.17623 · Cited by 11 · see citing works
Self-planning Code Generation with Large Language ModelsPreprint
Jiang, X., Dong, Y., Wang, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.06689 · Cited by 11 · see citing works
Self-citationCodeScore: Evaluating Code Generation by Learning Code ExecutionPreprint
Dong, Y., Ding, J., Jiang, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.09043 · Cited by 10 · see citing works
Self-citationInvestigating Data Contamination in Modern Benchmarks for Large Language Models
Deng, C., Zhao, Y., Tang, X., Gerstein, M., & Cohan, A. (2023) · arXiv · doi:10.48550/arxiv.2311.09783 · Cited by 8 · see citing works
DevEval: Evaluating Code Generation in Practical Software ProjectsPreprint
Li, J., Li, G., Zhao, Y., Li, Y., Jin, Z. G., Zhu, H., Liu, H., Liu, K., Wang, L., Fang, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.06401 · Cited by 5 · see citing works
Self-citationData Contamination Through the Lens of Time
Roberts, M., Thakur, H., Herlihy, C., White, C., & Dooley, S. (2023) · arXiv · doi:10.48550/arxiv.2310.10628 · Cited by 2 · see citing works
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
Research works printed but not linked: 10
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR.
- Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In EMNLP (1), pages 1286- 1305. Association for Computational Linguistics.
- Yihong Dong, Ge Li, and Zhi Jin. 2023c. CODEP: grammatical seq2seq model for general-purpose code generation. In ISSTA. 9 12047
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
- Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contami- nation by evaluation benchmarks. In EMNLP, pages 5075-5084. Association for Computational Linguis- tics.
- Vladimir I Levenshtein et al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710.
- Inbal Magar and Roy Schwartz. 2022. Data contami- nation: From memorization to exploitation. In ACL (2), pages 157-165. Association for Computational Linguistics.
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In ICLR. OpenReview.net. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774. · DOI · Paper page
- Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In SP, pages 1314-1331. IEEE.
- Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, and Ge Li. 2022. Incorporating do- main knowledge through task augmentation for front- end javascript code generation. In ESEC/SIGSOFT FSE, pages 1533-1543. ACM.
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2 · see citing works
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Tao, Y., Wang, T., Dong, Y., Liu, H., Zhang, K., Hu, X., & Li, G. (2025) · arXiv · doi:10.48550/arxiv.2510.09259 · Cited by 1 · see citing works
Self-citation
- Authors
- Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G.
- Affiliations
- China Academy of Space Technology (CN)
- Peking University (CN)
- Beijing Institute of Control Engineering (CN)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 12039-12050
- OpenAlex
- W4402670101
- Cited by
- 52
- References
- 21
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 4
- FWCI
- 15.32
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61832009, 62192730, 62072007, 62192731, 62192733)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- contaminated data, large language models, contamination detection, output distribution, assessment reliability, synthetic training data, benchmark contamination, HumanEval, ChatGPT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
