The DOI matches its Crossref record; 18 of the 22 references it cites were found in a registry or an index.
Leakage in data mining: formulation, detection, and avoidance
Abstract
Deemed "one of the top ten data mining mistakes", leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. In addition to our own industry experience with real-life projects, controversies around several major public data mining competitions held recently such as the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge are evidence that this issue is as relevant today as it has ever been. While acknowledging the importance and prevalence of leakage in both synthetic competitions and real-life data mining projects, existing literature has largely left this idea unexplored. What little has been said turns out not to be broad enough to cover more complex cases of leakage, such as those where the classical i.i.d. assumption is violated, that have been recently documented. In our new approach, these cases and others are explained by explicitly defining modeling goals and analyzing the broader framework of the data mining problem. The resulting definition enables us to derive general methodology for dealing with the issue. We show that it is possible to avoid leakage with a simple specific approach to data management followed by what we call a learn-predict separation, and present several ways of detecting leakage when the modeler has no control over how the data have been collected.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Common errors such as data leakage (the model having access to information at training/building time that it would not have at test/deployment/prediction time; Kaufman et al., 2011), and spurious correlations/biases (that exist in training data but do not reflect the deployment context of the model) are often found by observing model explanations and finding predictors that should not show up as”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
In the citing paper
“This nonstationary nature of the data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage (Kaufman et al., 2011; Samala et al., 2020).”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857
Data preparation for data mining
Pyle, D. (1999) · Cited by 1,368
Data-Snooping Biases in Tests of Financial Asset Pricing Models
Lo, A. W., & MacKinlay, A. C. (1990) · Review of Financial Studies · doi:10.1093/rfs/3.3.431 · Cited by 1,134 · cited 1×
“In some domains such as time-series prediction, where typically only a single history measuring the phenomenon of interest is available for analysis, this form of leakage is endemic and commonly known as data snooping/dredging [Lo and MacKinlay 1990] (a situation where researchers do not form a hypothesis in advance o…”
BackgroundHandbook of Statistical Analysis and Data Mining Applications
(2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374765-5.x0001-0 · Cited by 700 · cited 4×
“Two medical data mining contests held the following year and which also exhibited leakage are discussed in [7, 13].”
BackgroundInfluentialNordhausen, K. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_18.x · Cited by 637 · cited 1×
“For a complete reference see [1].”
KDD-Cup 2000 organizers' report: peeling the onion
Kohavi, R., Brodley, C. E., Frasca, B., Mason, L., & Zheng, Z. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/380995.381033 · Cited by 256
Link prediction by de-anonymization: How We Won the Kaggle Social Network Challenge
Narayanan, A., Shi, E., & Rubinstein, B. I. P. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033446 · Cited by 176 · cited 4×
“As another example, the pneumonia detection database in the INFORMS 2008 challenge discussed in Perlich et al. [2008] and Rosset et al. [2010] implies that a certain combination of missing diagnosis code and some other features is highly informative of the target.”
BackgroundMethodInfluentialLessons and Challenges from Mining Retail E-Commerce Data
Kohavi, R., Mason, L., Parekh, R., & Zheng, Z. (2004) · Machine Learning · doi:10.1023/b:mach.0000035473.11134.83 · Cited by 142
Research works printed but not linked: 3
- Kohavi R., Proceedings of the Fifth WEBKDD Workshop.
- Narayanan A., Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN). Preprint.
- 10.1002/sam.v3:4 · DOI
Cited by
Items 101–140 of 140
Prediction of Imminent Battery Depletion in Implantable Cardioverter-Defibrillator
Neupane, S., & Goswami, T. (2026) · Sci · doi:10.3390/sci8040072 · Cited by 1
“To prevent information leakage [33], GMMs were trained solely on the 32 real devices.”
Yusupov, R. R., Markov, E., & Васильевна, В. В. (2026) · International Russian Smart Industry Conference (SmartIndustryCon) · doi:10.1109/smartindustrycon68821.2026.11493135 · Cited by 1
An Estimate Method of Event Influence Scope Based on Special Field
Xue, Y., Xu, L., Wang, L., Zhang, G., & Luo, X. (2015) · International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI) · doi:10.1109/iiki.2015.15 · Cited by 1
GAPS: Generality and Precision with Shapley Attribution
Daley, B., Ratul, Q. E. A., Serra, E., & Cuzzocrea, A. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021127 · Cited by 1
“It is a combination of six different interpretation models: LIME [28], Shapley sampling values [30], DeepLIFT [31], QII [32], Layer-wise relevance propagation [33], and Shapley regression values [34].”
MethodModel-driven data curation pipeline for LC–MS-based untargeted metabolomics
Riquelme, G., Bortolotto, E. E., Dombald, M., & Monge, M. E. (2023) · Metabolomics · doi:10.1007/s11306-023-01976-1 · Cited by 1
“In addition, the possibility of data leakage, which is often overlooked when evaluating the DR on study samples, should be considered in model training during data analysis, since it will lead to an over-estimation of the predictive performance (Kaufman et al., 2011).”
BackgroundZhang, C., Nigo, M., Patel, S. A., Yu, D., Septimus, E. J., & Wu, H. (2024) · Statistics in Biosciences · doi:10.1007/s12561-023-09411-8 · Cited by 1
Deep Learning-Aided Research and the Aim-of-Science Controversy
Onishi, Y. (2024) · Journal for General Philosophy of Science · doi:10.1007/s10838-023-09667-0 · Cited by 1
“The authors mention two particular ways that models can be trapped by such ungeneralizable features, data leakage and dataset shift (Kaufman et al. 2011 ; Quiñonero-Candela et al. 2009).”
Evaluation and Analysis of the NLP Model Zoo for Ukrainian Text Classification
Panchenko, D., Maksymenko, D., Turuta, O., Yerokhin, A., Daniiel, Y., & Turuta, O. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-20834-8_6 · Cited by 5
Self Organizing Supply Chains for Micro-Prediction: Present and Future uses of the ROAR ProtocolPreprint
Cotton, P. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1907.07514 · Cited by 1
“Data leakage plagues the contest literature as noted in (Kaufman & Rosset, 2011) and (Narayanan et al., 2011).8 The CTF is primarily a research not a production paradigm, and cannot be expected to help practitioners materially with model deployment, ongoing model relevance or model oversight.”
BackgroundTime-resolved compound repositioning predictions on a text-mined knowledge networkPreprint
Mayers, M., Li, T. S., Queralt-Rosinach, N., & Su, A. I. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/625459
Team voyTECH: User Activity Modeling with Boosting TreesPreprint
Bayer, I., & Zouzias, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01620
“By definition, target encodings introduce target leakage and could lead to poor generalization performance, hence, target encoding regularization must always be used [10].”
BackgroundDeep metric learning improves lab of origin prediction of genetically engineered plasmidsPreprint
Soares, I. M., Camargo, F. H. F., Marques, A., & Crook, O. M. (2021) · Research Square · doi:10.21203/rs.3.rs-1072312/v1
Automatic feature selection for performing Unit 2 of vault in wheel gymnastics
Kitajima, E., Sato, T., Kurata, K., & Miyata, R. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287095
Riezler, S., & Hagmann, M. (2024) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-57065-0_1
Riezler, S., & Hagmann, M. (2024) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-57065-0_2
Poli, A., Thiriet, J., Altakroury, H., Ecosse, S., Mahdar-Recorbet, L., Stortini, N., Duman, C., Koïdé, N., & Trognon, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4786094/v1
Zhang, S., & Song, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-7832524/v1
Intelligent Network Intrusion Detection Using ML With Hybrid XGBoost and Anomaly Fusion
Magapu, S., Chindiri, P., Reddy, C. P. B., Kalyan, V. V., Reddy, M. S. S. L., & Mohamed, S. I. P. (2025) · International Conference on Intelligent Systems for Cybersecurity (ISCS) · doi:10.1109/iscs69371.2025.11386234
“From a system engineering perspective, ensuring reproducibility and auditability demands robust ETL pipelines, leakage mitigation, and OLAP-style analytics that aggregate detections by relevant dimensions like hour and service [3], [8].”
InfluentialAn Encoder-Only Transformer with Features-as-Tokens for Robust Network Intrusion Detection
Hider, M. M. N., Rahma, F., & Kurniawardhani, A. (2025) · International Seminar on Research of Information Technology and Intelligent Systems (ISRITI) · doi:10.1109/isriti68345.2025.11393121
Dănciulescu, R. C., Ivănescu, R. C., STĂNESCU, D.-R., Băiaşu, A. F., Alexandru, D. O., & Şerbănescu, M. S. (2026) · Romanian Journal of Morphology and Embryology · doi:10.47162/rjme.66.4.09
Prediction of Student Admission in Higher Education Using Deep Learning and Oversampling Method
Navastara, D. A., Ramadhany, T. P., Fatichah, C., Millaturrosyidah, Y., & Suciati, N. (2025) · International Conference of Science and Information Technology in Smart Administration (ICSINTESA) · doi:10.1109/icsintesa68165.2025.11413656
From sequences to knowledge, improving and learning from sequence alignments
Blassel, L. (2022) · theses.fr (ABES) · doi:10.70675/edb2db56z9b8ez4f44zb76az28681bf8bc51
Martino, T. D. (2024) · theses.fr (ABES) · doi:10.70675/ffcd9df2zcdeaz4de0z9715z936a2c9dc9ca
Predicting Customer Participation in Affiliate Marketing Using Boosting Models in E-Commerce
Gündoğmuş, Y. E., Raştak, O. A., Tomakyan, S., Keçeci, S., & Rençberoğlu, E. (2025) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3789982.3790061
Piewfad, S., Aimtongkham, P., & Musikawan, P. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498159
Reconstructing the Hydroclimatic History of the Mississippi River Basin Over the Past Millennium
Mcdaniel, B., Fosu, B., Nguyen, H. T. T., Lin, Y., He, J., Mercer, A., & Dyer, J. L. (2026) · Water Resources Research · doi:10.1029/2025wr042421
“Because CCA rotates both predictors and predictand to maximize their correlations, it is more susceptible to data leakage, in which the model inadvertently incorporates attributes of the predictand during calibration (Kapoor & Narayanan, 2023; Kaufman et al., 2011).”
Decentralizing Reputation Scoring Using Blockchain Behavioral Data and Machine Learning
Ahmed, M. S., Joy, S. I., Halder, N., Islam, A., & Alam, S. A. (2026) · International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) · doi:10.1109/icaiset66439.2026.11542120
“As formally defined by Kaufman et al. [9], leakage occurs when a model is trained with features that would not be practically available at the time of prediction.”
Calderón, L. S. (2026) · Array · doi:10.1016/j.array.2026.101039
Universal graph-based identifiers of chemical structures for linking large material databases
Muraoka, K., Tanimoto, T., Munekata, T., & Nakayama, A. (2026) · Nature Communications · doi:10.1038/s41467-026-74536-5 · Cited by 1
“This phenomenon, known as leakage, can lead to overestimated model performance and suboptimal solutions since the trained model is already familiar with a portion of the test data 53,54 .”
Nur, M. Z. A. T., & Rachmatullah, M. I. C. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620708
de Capretz, P. O., Nyström, A., Björkelund, A., Björk, J., Ohlsson, M., & Ekelund, U. E. G. (2026) · Cardiology · doi:10.1159/crd/absag001
Hierarchically structured data can undermine machine learning results in geoscience
Scharf, T. W., Daggitt, M. L., & Kirkland, C. L. (2026) · Earth and Planetary Science Letters · doi:10.1016/j.epsl.2026.120250
Jeon, S., Choi, W., Kwon, S., Lee, J., & Bae, J. (2026) · Sensors · doi:10.3390/s26165095
“This can lead to data leakage and overly optimistic performance estimates [14,15].”
Decentralizing Reputation Scoring Using Blockchain Behavioral Data and Machine Learning
Provat, N. H., Abyad, S. M., Antu, S. M. R. R., Pramanik, S., Dhrubo, A. F. H., Qayum, M. A., & Sajjad, M. (2026) · International Conference on Methods and Models in Automation and Robotics · doi:10.1109/mmar70562.2026.11667963
Amini, A., & Avazpour, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10527737/v1
Pattern Stability Analysis in Post-Crisis SME Lending: A Time-Split Classification Framework
Wibowo, A. B., Engolodoe, C. R., Hakim, F. A., & Puspita, R. (2026) · International Conference on Smart Data, Intelligence, and Analytics (ICoSDIA) · doi:10.1109/icosdia70683.2026.11690439
Benchmarking hyperparameter optimization strategies for crop genomic prediction
Xu, H., Wang, Y., Ma, Y., Wang, H., Cao, Y., Yan, S., Li, J., Zhang, Y., & Chen, Y. (2026) · Frontiers in Genetics · doi:10.3389/fgene.2026.1934071
Ahmad, A., Waqar, M. M., Ali, H. O., & Kim, S. C. (2026) · Applied Sciences · doi:10.3390/app16199560
ALDEMİR, A. A., Korkmaz, A., & Akin, E. (2026) · Mathematics · doi:10.3390/math14203708
Metadata
- Authors
- Kaufman, S., Rosset, S., & Perlich, C.
- Affiliations
- Tel Aviv University (IL)
- Media6Degrees, New York, NY, USA
- ORCID
- Rosset, Saharon
- Venue
- Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2011-08-21
- Pages
- 556-563
- OpenAlex
- W2044879407
- Cited by
- 137
- References
- 22
- Citations by year
- 2026: 22 · 2025: 10 · 2024: 15 · 2023: 13 · 2022: 15 · 2021: 23 · 2020: 13 · 2019: 5 · 2018: 2 · 2017: 4 · 2016: 4 · 2015: 4 · 2014: 2 · 2013: 1 · 2012: 3
- FWCI
- 1.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- leakage, data mining, data mining competition, data management, leak detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.