Proving Test Set Contamination in Black Box Language Models
Open accessAbstract
Large language models are trained on vast amounts of internet data, prompting concerns and speculation that they have memorized public benchmarks. Going from speculation to proof of contamination is challenging, as the pretraining data used by proprietary models are often not publicly accessible. We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples. We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Using our test, we audit five popular publicly accessible language models for test set contamination and find little evidence for pervasive contamination.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study applies three BDC detection methods from distinct categories-Min-K% Prob, Sharded Rank Comparison Test, and TS-Guessing-to 14 candidate models, and retains only models deemed uncontaminated by all three detection methods on all benchmarks.
“To achieve this, we apply three BDC detection methods from distinct categories-Min-K% Prob (Shi et al., 2023), Sharded Rank Comparison Test (Oren et al., 2023), and TS-Guessing (Deng et al., 2023)-to 14 candidate models. We adopt a rigorous criterion: only models deemed uncontaminated by all three detection methods on all benchmarks are retained”
From The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (2025) · cites this paper as Oren et al., 2023
The p-value from the Sharded Rank Comparison Test serves as an indicator of contamination.
“The resulting p-value serves as an indicator of contamination (see Tab. 9).”
From The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (2025) · cites this paper as Oren et al., 2023
OLMo-7B is identified as contaminated on TruthfulQA.
“OLMo-7B is identified as contaminated on TruthfulQA.”
From The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination (2025) · cites this paper as Oren et al., 2023
Cites
Still being checked.
Cited by
Sorted by citations
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · arXiv · doi:10.48550/arxiv.2402.15938 · Cited by 52
“Recent work tried to detect contamination without access to the pre-training corpus (Oren et al., 2023; Deng et al., 2023; Golchin et al., 2023).”
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.716 · Cited by 52 · see citing works
Task Contamination: Language Models May Not Be Few-Shot Anymore
Li, C., & Flanigan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29808 · Cited by 47 · see citing works
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · see citing works
Lookahead Bias in Pretrained Language Models
Sarkar, S., & Vafa, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4754678 · Cited by 23 · see citing works
An archival perspective on pretraining data
Desai, M. A., Pasquetto, I. V., Jacobs, A. Z., & Card, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100966 · Cited by 21 · see citing works
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.20 · Cited by 37 · see citing works
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y.-A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00028 · Cited by 20 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
Blind Baselines Beat Membership Inference Attacks for Foundation Models
Das, D., Zhang, J., & Tramèr, F. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00016 · Cited by 5 · see citing works
Anindya, A. L., Olsson, T. N., Jensen, M., García-Bonete, M.-J., Wheatley, S. P., Bokarewa, M. I., Mezzasalma, S. A., & Katona, G. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad5784 · Cited by 4 · see citing works
Quantification of Large Language Model Distillation
Lee, S., Zhou, J., Ao, C., Li, K., Du, X., He, S., Wu, H., Liu, T., Liu, J., Alinejad-Rokny, H., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.248 · Cited by 4 · see citing works
Cross-domain multi-step thinking: Zero-shot fine-grained traffic sign recognition in the wild
Gan, Y., Li, G., Togo, R., Maeda, K., Ogawa, T., & Haseyama, M. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114172 · Cited by 3 · see citing works
Zhao, Z., Bangash, A. A., Cogo, F. R., Adams, B., & Hassan, A. E. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3533972 · Cited by 4 · see citing works
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1 · see citing works
Tong, Z., Sun, F., & Nguyen, L.-M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_14 · Cited by 1 · see citing works
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
“Sharded Rank Comparison Test (Oren et al., 2023): This method partitions the test examples into shards, computes the log-likelihoods for both the original and shuffled orders within each shard, and calculates a shard-specific score based on their difference.”
Influential
- Authors
- Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.17623
- OpenAlex
- W4387994765
- Cited by
- 11
- References
- 0
- Citations by year
- 2025: 6 · 2024: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2146755)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, pre-training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
