Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Open accessAbstract
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution. To facilitate this study, we introduce two benchmarks, i.e., DetCon and ComiEval, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8\%-30.2\% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9\% attributed to data contamination across various contamination setups. In real-world applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Detection methods for data contamination have almost exclusively focused on the pre-training and Supervised Fine-Tuning (SFT) stages.
“many researchers have developed detection methods, but they have almost exclusively focused on the pre-training and Supervised Fine-Tuning (SFT) (Dong et al., 2024; Fu et al., 2024; Shi et al., 2024; Zhang et al., 2024b; Xie et al., 2024b; Zhang et al., 2025a;b) stages.”
From Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models (2025) · cites this paper as Dong et al., 2024
The “Data contamination detection” section discusses several methods aimed at contamination of pre-training data, and §4.1 describes the authors’ detection experiments using fine-tuned models. The paper does not establish that detection methods have focused almost exclusively on pre-training and supervised fine-tuning stages. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2× · see citing works
“7B (Nijkamp et al., 2023); for logical reasoning tasks, we select Llama2-7B (Touvron et al., 2023) and Bloom-7B (Scao et al., 2022).”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 10× · see citing works
“2) HumanEval_R is reconstructed on HumanEval, which replaces its function signature, translates its requirements into German, French, and Chinese, selects different public test cases from the work (Dong et al., 2023a) to prompt, and remains the private test cases for testing.”
InfluentialTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 2× · see citing works
“We choose the HumanEval (Chen et al., 2021) dataset for code generation and the GSM8K (Cobbe et al., 2021) dataset for logical reasoning.”
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 2× · see citing works
“For code generation tasks, we use CodeLlama-7B (Rozière et al., 2023) and CodeGen-6.”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.08242 · Cited by 325 · cited 1×
“Data contamination, also known as data leakage, refers to the scenario where the test data has been included in the model’s training data (Magar et al., 2022; Dickson, 2023), leading to the model performing exceptionally well on these leaked test data.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 1× · see citing works
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · cited 1× · see citing works
“2) HumanEval_R is reconstructed on HumanEval, which replaces its function signature, translates its requirements into German, French, and Chinese, selects different public test cases from the work (Dong et al., 2023a) to prompt, and remains the private test cases for testing.”
BackgroundPrivacy Risks of General-Purpose Language Models
Pan, X., Zhang, M., Ji, S., & Yang, M. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00095 · Cited by 188 · cited 1× · see citing works
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
Self-citationLi, Y., Guerin, F., & Lin, C.-H. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2312.12343 · Cited by 77 · cited 1× · see citing works
“The work (Li et al., 2023b) leverages temporal information to construct a benchmark beginning from January 2023.”
CodeScore: Evaluating Code Generation by Learning Code Execution
Dong, Y., Ding, J., Jiang, X., Li, G., Li, Z., & Jin, Z. G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695991 · Cited by 76 · see citing works
Self-citationEstimating Contamination via Perplexity: Quantifying Memorisation in Language Model EvaluationPreprint
Li, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2309.10677 · Cited by 59 · cited 1× · see citing works
“The work (Li, 2023) assumed that data leaked into the training set tends to exhibit lower perplexity and utilizes perplexity analysis for detection.”
Competition-Level Problems are Effective LLM EvaluatorsPreprint
Huang, Y.-M., Lin, Z.-W., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2312.02143 · Cited by 38 · cited 2× · see citing works
“Although acknowledged the significance, data contamination detection and trustworthy evaluation for LLMs still persist as open and challenging issues (Yang et al., 2023; Huang et al., 2023).”
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · cited 2× · see citing works
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
Incorporating domain knowledge through task augmentation for front-end JavaScript code generation
Shen, S., Zhu, X. P., Dong, Y., Guo, Q., Zhen, Y., & Li, G. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558965 · Cited by 28 · see citing works
Self-citationDon't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · cited 3× · see citing works
“Some work (Pan et al., 2020; Zhou et al., 2023; Jacovi et al., 2023; Dodge et al., 2021) exposed the serious consequences of data contamination and urged attention to this problem.”
CODEP: Grammatical Seq2Seq Model for General-Purpose Code Generation
Dong, Y., Li, G., & Jin, Z. G. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598048 · Cited by 22 · see citing works
Self-citationProving Test Set Contamination in Black Box Language Models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2310.17623 · Cited by 11 · cited 1× · see citing works
“Recent work tried to detect contamination without access to the pre-training corpus (Oren et al., 2023; Deng et al., 2023; Golchin et al., 2023).”
Rethinking Benchmark and Contamination for Language Models with Rephrased SamplesPreprint
Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., & Stoica, I. G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04850 · Cited by 11 · cited 5× · see citing works
“To this end, LLM Decontaminator (Yang et al., 2023) detected the similarity of test data and training data based on other advanced LLMs.”
InfluentialDevEval: Evaluating Code Generation in Practical Software ProjectsPreprint
Li, J., Li, G., Zhao, Y., Li, Y., Jin, Z. G., Zhu, H., Liu, H., Liu, K., Wang, L., Fang, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.06401 · Cited by 5 · cited 1× · see citing works
“Following the previous works (Jiang et al., 2023; Dong et al., 2023b,c; Li et al., 2024), in generating these samples, we also record the outputs of greedy search with a temperature parameter of 0 (1 sample) and 50 samples obtained by sampling with a temperature of 0.8.”
InfluentialSelf-citationData Contamination Through the Lens of Time
Roberts, M., Thakur, H., Herlihy, C., White, C., & Dooley, S. (2023) · arXiv · doi:10.48550/arxiv.2310.10628 · Cited by 2 · cited 1× · see citing works
“Consequently, it becomes formidable to prevent data contamination for LLMs. Data contamination exerts a profound and deleterious impact on LLMs (Zhou et al., 2023; Jacovi et al., 2023; Roberts et al., 2023).”
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
5 of the 29 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Ger- stein, and Arman Cohan. 2023. Investigating data contamination in modern benchmarks for large lan- guage models. CoRR, abs/2311.09783. · arXiv
- Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023b. Self-collaboration code generation via chat- gpt. CoRR, abs/2304.07590. · arXiv
- Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in llms: Tracing data contamination in large language models. CoRR, abs/2308.08493. · arXiv
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
- Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code generation with large language model. CoRR, abs/2303.06689. · arXiv
- Vladimir I Levenshtein et al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710.
- Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. CoRR, abs/2310.16789. · arXiv
- Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. 2023. CLEAN-EVAL: clean evaluation on contaminated large language models. CoRR, abs/2311.09154. 11 12049 · arXiv
Cited by
Sorted by citations
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · see citing works
“In particular, large language models (LLMs) have achieved excellent performance and demonstrate promising potential on code generation tasks [15, 18, 38, 64].”
Self-citationLimitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
“The disparity between LLM performance on M-ARC and conventional medical QA aligns with studies suggesting that benchmark successes may stem from overfitting to statistical patterns in training data rather than reflecting emergent reasoning abilities [51, 52, 53, 54, 19].”
CodeScore: Evaluating Code Generation by Learning Code Execution
Dong, Y., Ding, J., Jiang, X., Li, G., Li, Z., & Jin, Z. G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695991 · Cited by 76 · see citing works
Self-citationKim, J., Wang, K., Weng, C., & Liu, C. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.08.010 · Cited by 53 · see citing works
Benchmarking the diagnostic performance of open source LLMs in 1933 Eurorad case reports
Kim, S. H., Schramm, S., Adams, L. C., Braren, R., Bressem, K. K., Keicher, M., Platzek, P., Paprottka, K. J., Zimmer, C., Hedderich, D. M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01488-3 · Cited by 41 · see citing works
“Another framework evaluates data contamination based on LLM output distribution characteristics 30 .”
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 47 · see citing works
Automating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30 · see citing works
Arun, S., Tedla, M., & Vaidhyanathan, K. (2025) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa65012.2025.00013 · Cited by 21 · see citing works
“An interesting threat to external validity can be contamination of LLMs [54], which is when LLMs are trained or fine-tuned on data that is later used to evaluate their performance, leading to unrealistically high scores.”
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 22 · see citing works
“Recent advances in Large Language Models (LLMs) have instigated revolutionary changes in the fields of artificial intelligence, natural language processing, and software engineering [1].”
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.20 · Cited by 37 · see citing works
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y.-A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00028 · Cited by 20 · see citing works
“Further, to fairly evaluate newly released models, it is crucial to understand whether they are trained on (and memorize) evaluation benchmarks [22], [25], [33], [53], [55], [64], [77].”
Demystifying Memorization in LLM-Based Program Repair via a General Hypothesis Testing Framework
Kong, J., Xie, X., & Liu, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729390 · Cited by 8 · see citing works
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 7 · see citing works
“A critical concern when evaluating AI models is the issue of benchmark leakage [Dong et al., 2024, Matton et al., 2024].”
Jiang, X., Dong, Y., Tao, Y., Liu, H., Jin, Z. G., & Li, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00133 · Cited by 7 · see citing works
Self-citationLiu, R., Li, J., Zivkovic, M., & Abbasi, A. (2025) · MIS Quarterly · doi:10.25300/misq/2024/18573 · Cited by 6 · see citing works
VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
Wang, Z., Shao, M., Bhandari, J., Mankali, L., Karri, R., Sinanoglu, O., Shafique, M., & Knechtel, J. (2025) · IEEE International Conference on LLM-Aided Design (ICLAD) · doi:10.1109/iclad65226.2025.00017 · Cited by 6 · see citing works
“Using the setup in [44] [45], we ran 50 sample inferences per model for each problem in VerilogEval and RTLLM for CDD and Min-K% Prob (K=20) evaluations.”
InfluentialStolwijk, S. B., Boukes, M., Yeung, W. N., Liao, Y., Münker, S., Kroon, A. C., & Trilling, D. (2025) · Communication Methods and Measures · doi:10.1080/19312458.2025.2553300 · Cited by 6 · see citing works
“However, recent work suggests that the benchmark performance of GLLMs might be inflated, since these benchmark tasks might be part of the training data (Dong et al., 2024; Mirzadeh et al., 2025).”
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 4 · see citing works
“AI peer-review systems learn from large corpora of scientific literature, drawing on academic repositories, web-crawled content, and scholarly databases to internalize argumentative structure, evaluation norms, and domain knowledge 93 .”
Quantification of Large Language Model Distillation
Lee, S., Zhou, J., Ao, C., Li, K., Du, X., He, S., Wu, H., Liu, T., Liu, J., Alinejad-Rokny, H., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.248 · Cited by 4 · see citing works
Self-citationKartal, A., Manalil, N. F., Cheng, C. D., Chung, L. K., Gebhard, H., Greenberg, M. D., Härtl, R., & Elsayed, G. A. (2025) · Global Spine Journal · doi:10.1177/21925682251411225 · Cited by 3 · see citing works
Less Is More: Failing Test Generation with Large Language Models
Li, T.-O., Yan, X., Zong, W., Wang, Y., Tian, H., Xu, C., Wang, Y., Cheung, S.-C., & Kramer, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3793675 · Cited by 3 · see citing works
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
“Dong et al. [11] proposed the CDD method to identify peaks in the output distribution to detect data contamination.”
Self-citationCodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2 · see citing works
“Recently, Dong et al. [35] detected data contamination by identifying the peakedness of LLM’s output distribution and showed potential.”
Wen, A., Lu, Q., Chuang, Y.-N., Wang, G., Yuan, J., Zhang, J., Wang, L. H., Fu, S., Miller, K., Jia, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02253-2 · Cited by 2 · see citing works
“Opacity on the data used for model training purposes raises signi fi cant concerns about answermemorization 39,40 .”
Self-citationDetecting Training Data For Large Language Models: A Survey
Yang, C., Li, J., Shulin, L. A. N., Wang, Y., Du, H., Gong, C., Yao, X., Niyato, D. T., & Zhu, L. (2026) · ACM Computing Surveys · doi:10.1145/3779430 · Cited by 2 · see citing works
DARE : Diverse Visual Question Answering with Robustness Evaluation
Sterz, H., Pfeiffer, J., & Vulić, I. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.29 · Cited by 3 · see citing works
The mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 11 · see citing works
“We query LM ref for synthetic sequences repeatedly and apply rejection sampling until we have a sufficient number of reference canaries satisfying | T ( X”
Salido, E. S., Gonzalo, J. A., & Marco, G. (2026) · IEEE Access · doi:10.1109/access.2026.3651579 · Cited by 7 · see citing works
“Recent analyses demonstrate that models may appear to generalize while actually relying on leaked or previously seen data [35], and contamination can lead to misleading performance estimates in widely used benchmarks [12], [36], [37].”
Tong, Z., Sun, F., & Nguyen, L.-M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_14 · Cited by 1 · see citing works
“Besides the papers mentioned in Table 1, benchmark contamination scenario was also investigated in studies: [27,8,12,67,25].”
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute
Ma, Y., Li, Y., Dong, Y., Jiang, X., Li, Y., Liu, Y., Cao, R., Chen, J., Huang, F., & Li, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00309 · Cited by 4 · see citing works
“To enhance efficiency in multi-round inference, we adopt a history pruning mechanism inspired by DeepSeek R1 [9].”
Self-citationSWE-Bench+: Enhanced LLM Coding Benchmark
Xue, H., Aleithan, R., Enan, N., Uddin, G., & Wang, S. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814924 · Cited by 1 · see citing works
“Prior work has studied dataset quality issues in vulnerability detection [6, 9, 32], defect prediction [3, 26], code summarization [27], and code generation [10, 16, 20, 23, 31].”
Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents
Rafiee, M., Soudani, H., Abbasiantaeb, Z., Aliannejadi, M., Hasibi, F., & Zamani, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808629 · Cited by 1 · see citing works
“A n s w e r : M E D I AN ( 3 , 3 , 3 , 4 ) = 3”
IDE: A Multi-Agent-Driven Iterative Framework for Dynamic Evaluation of LLMs
Tong, X., Jin, B., Wang, J., Wu, X., Xia, T., & Han, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890123 · Cited by 1 · see citing works
“However, static benchmarks have certain limitations, particularly regarding data contamination [16]–[18].”
Memorization in LLM-Based Program Repair
Kong, J., Cheng, M., & Xie, X. (2025) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr66717.2025.00011 · Cited by 1 · see citing works
“Recent studies [17]–[20] have proposed state-of-the-art metrics to quantify the degree of memorization for any given input.”
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Tao, Y., Wang, T., Dong, Y., Liu, H., Zhang, K., Hu, X., & Li, G. (2025) · arXiv · doi:10.48550/arxiv.2510.09259 · Cited by 1 · see citing works
“Prior work on data contamination detection in LLMs has mainly focused on the pre-training and Supervised Fine-Tuning stages (Mireshghallah et al., 2022b; Fu et al., 2023; Mattern et al., 2023; Shi et al., 2024; Xie et al., 2024b; Gonen et al., 2023; Dong et al., 2024; Zhang et al., 2025b).”
InfluentialSelf-citationSun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
“This phenomena, known as Benchmark Data Contamination (BDC) , results in falsely inflated performance metrics, thereby undermining the reliability of evaluation conclusions (Zhou et al., 2023; Jiang et al., 2024; Sainz et al., 2023).”
Data Optimization for LLMs: A Survey
Wu, O. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.36227/techrxiv.174776562.20873028/v1
Learning or Cheating? Assessing Data Contamination in Large Vision-Language Models
Masry, A., Ahmed, M. F., & Mahbub, R. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204323
Data Leakage Detection in Large Vision-Language Models via Multimodal Perturbation
Wang, X., Liu, Z., Zhan, Y., Zhao, K., Wang, K., & Lian, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3398-5_15
MPCLA: Multi-perspective Cross-Lingual Alignment for Efficient Preference Leakage Mitigating in LLMs
Wu, Q., Peng, S., Lu, Z., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5761-5_33
AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation
Zhang, T., Mao, X.-J., Wang, S., Zhao, Y.-X., Lu, Y., Zhang, J., Zhang, Z., Yang, K., & Yu, Y. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00103 · Cited by 2 · see citing works
“…adaptation data. frame = sys._getframe(size) frame = frame.f_back if not frame: ) frame = frame.f_back if not frame: In our study, we adopt CDD [81] and a mutation-based evaluation [82], [83] to detect data leakage. return size except ValueError: return size except ValueError: Specifically,…”
Specializing LLMs for Code: A Framework for Efficient Adaptation and Robust Evaluation
Singh, A., & Chen, J. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391170
“While benchmarks such as HumanEval (which contains 164 hand-crafted programming problems) have been widely used, studies have shown that portions of HumanEval are already present in training corpora, thereby contaminating evaluation results [13].”
Suntwal, S., & Brown, S. A. (2026) · Information Systems Research · doi:10.1287/isre.2023.0457
When Reasoning Leaks Membership: Membership Inference Attack on Black-box Large Reasoning Models
Hu, R., Shang, Y.-M., Luo, W., Tao, Y., & Zhang, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792588
A Pedagogical Image Generation Framework for Math Word Problems Based on Situation Decomposition
Du, Y., Li, Q., He, B., et al. (2025) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir67391.2025.11491050
“This helps mitigate potential data contamination issues [44] and verifies whether LLMs can construct accurate problem-solving models.”
Regan, C., & Xie, Y. (2026) · Journal of risk and financial management · doi:10.3390/jrfm19060382
RADS-PDD: Reproducibility-aware dynamic similarity-based pre-training data detection against LLMs
Fan, X., Miyamoto, R., Mo, F., Chen, C., Matsumoto, T., Kido, F., & Yamana, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109393
Revisiting TuRTLe: A Comprehensive Evaluation of LLMs for RTL Generation
Albertí, M., Gutierrez-Gomez, C., García-Gasulla, D., Parisi, E., Ghorab, R. M., Montenegro, O., Homs, B., Moretó, M., & Kestor, G. (2026) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3831369
Lin, J., Zhang, B., Wang, S., Liu, Z., Gutfreund, D., Shun, J., & Zhu, Y. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817481
“Recent studies have indicated that pretraining contamination , the overlap between benchmark data and model pretraining corpora, can inflate performance and undermine fair comparison across model generations [10, 23, 25, 32, 33, 40].”
A survey of reward hacking in agentic large language model systems
Morampudi, A., Irrink, U., Grandhi, R., Pagadala, V., & Maddula, M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01980-z
“• Strong empirical evidence : Reward-model overoptimization [9, 16, 17], sycophancy [10], LLM-judge biases [11, 18], benchmark contamination [42, 49, 50], and specification gaming in RL and digital evolution [2, 79].”
CONSPIR ED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
Bates, L., Glockner, M., Nakov, P., & Gurevych, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.794 · Cited by 1 · see citing works
Wilde, V. K. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03184-w
“…of a contaminated sample – one in which an unknown fraction of observations is drawn from a different generating process (Tukey, 1960), here model-assisted rather than unaided responding – not in the machine-learning sense of benchmark data leaking into model training corpora (Dong et al., 2024).”
Byun, Y., Lee, D., Kwak, C., Yoon, C. Y., & Seo, Y. J. (2026) · Digital Health · doi:10.1177/20552076261490211
Babacan, S. (2026) · Clinical Anatomy · doi:10.1002/ca.70240 · Cited by 1 · see citing works
Search-Time Data Contamination
Han, Z., Mankikar, M., Michael, J., & Wang, Z. (2025) · arXiv · doi:10.48550/arxiv.2508.13180
“This phenomenon is exemplified in several recent works [3, 4, 20].”
- Authors
- Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G.
- Affiliations
- China Academy of Space Technology (CN)
- Peking University (CN)
- Beijing Institute of Control Engineering (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 12039-12050
- arXiv
- 2402.15938
- OpenAlex
- W4402670101
- Cited by
- 52
- References
- 29
- Citations by year
- 2026: 22 · 2025: 26 · 2024: 4
- FWCI
- 15.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61832009, 62192730, 62072007, 62192731, 62192733)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- contaminated data, large language models, contamination detection, output distribution, assessment reliability, synthetic training data, benchmark contamination, HumanEval, ChatGPT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
