The DOI matches its Crossref record; 18 of the 22 references it cites were found in a registry or an index.
Leakage in data mining: formulation, detection, and avoidance
Abstract
Deemed "one of the top ten data mining mistakes", leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. In addition to our own industry experience with real-life projects, controversies around several major public data mining competitions held recently such as the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge are evidence that this issue is as relevant today as it has ever been. While acknowledging the importance and prevalence of leakage in both synthetic competitions and real-life data mining projects, existing literature has largely left this idea unexplored. What little has been said turns out not to be broad enough to cover more complex cases of leakage, such as those where the classical i.i.d. assumption is violated, that have been recently documented. In our new approach, these cases and others are explained by explicitly defining modeling goals and analyzing the broader framework of the data mining problem. The resulting definition enables us to derive general methodology for dealing with the issue. We show that it is possible to avoid leakage with a simple specific approach to data management followed by what we call a learn-predict separation, and present several ways of detecting leakage when the modeler has no control over how the data have been collected.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Common errors such as data leakage (the model having access to information at training/building time that it would not have at test/deployment/prediction time; Kaufman et al., 2011), and spurious correlations/biases (that exist in training data but do not reflect the deployment context of the model) are often found by observing model explanations and finding predictors that should not show up as”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
In the citing paper
“This nonstationary nature of the data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage (Kaufman et al., 2011; Samala et al., 2020).”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857
Data preparation for data mining
Pyle, D. (1999) · Cited by 1,368
Data-Snooping Biases in Tests of Financial Asset Pricing Models
Lo, A. W., & MacKinlay, A. C. (1990) · Review of Financial Studies · doi:10.1093/rfs/3.3.431 · Cited by 1,134 · cited 1×
“In some domains such as time-series prediction, where typically only a single history measuring the phenomenon of interest is available for analysis, this form of leakage is endemic and commonly known as data snooping/dredging [Lo and MacKinlay 1990] (a situation where researchers do not form a hypothesis in advance o…”
BackgroundHandbook of Statistical Analysis and Data Mining Applications
(2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374765-5.x0001-0 · Cited by 700 · cited 4×
“Two medical data mining contests held the following year and which also exhibited leakage are discussed in [7, 13].”
BackgroundInfluentialNordhausen, K. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_18.x · Cited by 637 · cited 1×
“For a complete reference see [1].”
KDD-Cup 2000 organizers' report: peeling the onion
Kohavi, R., Brodley, C. E., Frasca, B., Mason, L., & Zheng, Z. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/380995.381033 · Cited by 256
Link prediction by de-anonymization: How We Won the Kaggle Social Network Challenge
Narayanan, A., Shi, E., & Rubinstein, B. I. P. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033446 · Cited by 176 · cited 4×
“As another example, the pneumonia detection database in the INFORMS 2008 challenge discussed in Perlich et al. [2008] and Rosset et al. [2010] implies that a certain combination of missing diagnosis code and some other features is highly informative of the target.”
BackgroundMethodInfluentialLessons and Challenges from Mining Retail E-Commerce Data
Kohavi, R., Mason, L., Parekh, R., & Zheng, Z. (2004) · Machine Learning · doi:10.1023/b:mach.0000035473.11134.83 · Cited by 142
Research works printed but not linked: 3
- Kohavi R., Proceedings of the Fifth WEBKDD Workshop.
- Narayanan A., Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN). Preprint.
- 10.1002/sam.v3:4 · DOI
Cited by
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
“Relying only on validation accuracy has many well studied problems, as practitioners consistently overestimate their model’s accuracy (Patel et al. 2008), propagate feedback loops (Sculley et al. 2015), or fail to notice data leaks (Kaufman, Rosset, and Perlich 2011).”
BackgroundFederated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842
“Recent works on privacy attacks within the FL setting have raised concerns on data leakage during model training 59 .”
BackgroundFairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
“Fairness in data mining can be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [19].”
Background
Metadata
- Authors
- Kaufman, S., Rosset, S., & Perlich, C.
- Affiliations
- Tel Aviv University (IL)
- Media6Degrees, New York, NY, USA
- ORCID
- Rosset, Saharon
- Venue
- Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2011-08-21
- Pages
- 556-563
- OpenAlex
- W2044879407
- Cited by
- 137
- References
- 22
- Citations by year
- 2026: 22 · 2025: 10 · 2024: 15 · 2023: 13 · 2022: 15 · 2021: 23 · 2020: 13 · 2019: 5 · 2018: 2 · 2017: 4 · 2016: 4 · 2015: 4 · 2014: 2 · 2013: 1 · 2012: 3
- FWCI
- 1.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- leakage, data mining, data mining competition, data management, leak detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Leakage in data mining: formulation, detection, and avoidance. https://app.cochat.ai/papers/10.1145/2020408.2020496
- The paper
- Kaufman, S., Rosset, S., & Perlich, C. (2011). Leakage in data mining: formulation, detection, and avoidance. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. https://doi.org/10.1145/2020408.2020496
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.