Leakage in data mining: formulation, detection, and avoidance
The DOI matches its Crossref record; 18 of the 22 references it cites were found in a registry or an index.
Abstract
Deemed "one of the top ten data mining mistakes", leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. In addition to our own industry experience with real-life projects, controversies around several major public data mining competitions held recently such as the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge are evidence that this issue is as relevant today as it has ever been. While acknowledging the importance and prevalence of leakage in both synthetic competitions and real-life data mining projects, existing literature has largely left this idea unexplored. What little has been said turns out not to be broad enough to cover more complex cases of leakage, such as those where the classical i.i.d. assumption is violated, that have been recently documented. In our new approach, these cases and others are explained by explicitly defining modeling goals and analyzing the broader framework of the data mining problem. The resulting definition enables us to derive general methodology for dealing with the issue. We show that it is possible to avoid leakage with a simple specific approach to data management followed by what we call a learn-predict separation, and present several ways of detecting leakage when the modeler has no control over how the data have been collected.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Common errors such as data leakage (the model having access to information at training/building time that it would not have at test/deployment/prediction time; Kaufman et al., 2011), and spurious correlations/biases (that exist in training data but do not reflect the deployment context of the model) are often found by observing model explanations and finding predictors that should not show up as”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
In the citing paper
“This nonstationary nature of the data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage (Kaufman et al., 2011; Samala et al., 2020).”
From Explainable machine learning for public policy: Use cases, gaps, and research directions (2023) · cites this paper as Kaufman et al., 2011
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857
Data preparation for data mining
Pyle, D. (1999) · Cited by 1,368
Data-Snooping Biases in Tests of Financial Asset Pricing Models
Lo, A. W., & MacKinlay, A. C. (1990) · Review of Financial Studies · doi:10.1093/rfs/3.3.431 · Cited by 1,134 · cited 1×
“In some domains such as time-series prediction, where typically only a single history measuring the phenomenon of interest is available for analysis, this form of leakage is endemic and commonly known as data snooping/dredging [Lo and MacKinlay 1990] (a situation where researchers do not form a hypothesis in advance o…”
BackgroundHandbook of Statistical Analysis and Data Mining Applications
(2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374765-5.x0001-0 · Cited by 700 · cited 4×
“Two medical data mining contests held the following year and which also exhibited leakage are discussed in [7, 13].”
BackgroundInfluentialNordhausen, K. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_18.x · Cited by 637 · cited 1×
“For a complete reference see [1].”
KDD-Cup 2000 organizers' report: peeling the onion
Kohavi, R., Brodley, C. E., Frasca, B., Mason, L., & Zheng, Z. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/380995.381033 · Cited by 256
Link prediction by de-anonymization: How We Won the Kaggle Social Network Challenge
Narayanan, A., Shi, E., & Rubinstein, B. I. P. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033446 · Cited by 176 · cited 4×
“As another example, the pneumonia detection database in the INFORMS 2008 challenge discussed in Perlich et al. [2008] and Rosset et al. [2010] implies that a certain combination of missing diagnosis code and some other features is highly informative of the target.”
BackgroundMethodInfluentialLessons and Challenges from Mining Retail E-Commerce Data
Kohavi, R., Mason, L., Parekh, R., & Zheng, Z. (2004) · Machine Learning · doi:10.1023/b:mach.0000035473.11134.83 · Cited by 142
Research works printed but not linked: 3
- Kohavi R., Proceedings of the Fifth WEBKDD Workshop.
- Narayanan A., Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN). Preprint.
- 10.1002/sam.v3:4 · DOI
Cited by
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
“Relying only on validation accuracy has many well studied problems, as practitioners consistently overestimate their model’s accuracy (Patel et al. 2008), propagate feedback loops (Sculley et al. 2015), or fail to notice data leaks (Kaufman, Rosset, and Perlich 2011).”
BackgroundFederated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842
“Recent works on privacy attacks within the FL setting have raised concerns on data leakage during model training 59 .”
BackgroundFairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
“Fairness in data mining can be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [19].”
BackgroundMachine Learning in Environmental Research: Common Pitfalls and Best Practices
Zhu, J.-J., Yang, M., & Ren, Z. J. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c00026 · Cited by 672
Machine Learning for Catalysis Informatics: Recent Applications and Prospects
Toyao, T., Maeno, Z., Takakusagi, S., Kamachi, T., Takigawa, I., & Shimizu, K.-I. (2020) · ACS Catalysis · doi:10.1021/acscatal.9b04186 · Cited by 617
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv
Routes for breaching and protecting genetic privacy
Erlich, Y., & Narayanan, A. (2014) · Nature Reviews Genetics · doi:10.1038/nrg3723 · Cited by 458
Fairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.83 · Cited by 371
“Fairness in machine learning can be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [13].”
BackgroundKubota, K. J., Chen, J. A., & Little, M. A. (2016) · Movement Disorders · doi:10.1002/mds.26693 · Cited by 210
“23 It can be extremely hard to identify sources of data leakage because data can be confounded by all manner of unknown effects.”
BackgroundLess is More: Lighter and Faster Deep Neural Architecture for Tomato Leaf Disease Classification
Ahmed, S., Hasan, M. B., Ahmed, T., Sony, M. R. K., & Kabir, M. H. (2022) · IEEE Access · doi:10.1109/access.2022.3187203 · Cited by 197
“This problem is known as data leakage [52].”
BackgroundBeyond crowd judgments: Data-driven estimation of market value in association football
Müller, O., Simons, A., & Weinmann, M. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.05.005 · Cited by 184
Effect of data leakage in brain MRI classification using 2D convolutional neural networks
Yagis, E., Atnafu, S. W., García Seco de Herrera, A., Marzi, C., Scheda, R., Giannelli, M., Tessa, C., Citi, L., & Diciotti, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-01681-w · Cited by 161
“Several studies employed deep learning methods for image improvement and transformation 5–10 .”
BackgroundUsing and understanding cross-validation strategies. Perspectives on Saeb et al.
Little, M. A., Varoquaux, G., Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix020 · Cited by 158
“They have, in particular, been the subject of substantial investigation in the data mining community, where they are known as “leakages,”(see [8] for many”
BackgroundMaking the Case for Safety of Machine Learning in Highly Automated Driving
Burton, S., Gauerhof, L., & Heinzemann, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66284-8_1 · Cited by 174
Conserved and divergent gene regulatory programs of the mammalian neocortex
Zemke, N. R., Armand, E. J., Wang, W., Lee, S., Zhou, J., Li, Y. E., Liu, H., Tian, W., Nery, J. R., Castanon, R. G., et al. (2023) · Nature · doi:10.1038/s41586-023-06819-6 · Cited by 114
“We selected test and validation datasets by identifying chromosomes with high degrees of sequence similarity across species to minimize data leakage 56 .”
BackgroundMethodLee, E., Balasubramanian, H., Tsolias, A., Anakwe, S. U., Medalla, M., Shenoy, K. V., & Chandrasekaran, C. (2021) · eLife · doi:10.7554/elife.67490 · Cited by 103
Spatiotemporal estimation of satellite-borne and ground-level NO2 using full residual deep networks
Li, L., & Wu, J. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2020.112257 · Cited by 89
Construction of a quality model for machine learning systems
Siebert, J., Jöckel, L., Heidrich, J., Trendowicz, A., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09557-y · Cited by 113
Explainable machine learning for public policy: Use cases, gaps, and research directions
Amarasinghe, K., Rodolfa, K. T., Lamba, H., & Ghani, R. (2023) · Data & Policy · doi:10.1017/dap.2023.2 · Cited by 73
“s.Commonerrors suchas data leakage (themodelhaving accesstoinformationattraining/building timethatitwouldnothaveattest/deployment/prediction time that is accidentally being used in the training data) [19], and spurious correlations/biases (that exist in training data but do not reflect the deployment context of the mod…”
BackgroundTowards resource-frugal deep convolutional neural networks for hyperspectral image segmentation
Nalepa, J., Antoniak, M., Myller, M., Lorenzo, P. R., & Marcinkiewicz, M. (2020) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.102994 · Cited by 67
An Empirical Study of the Impact of Data Splitting Decisions on the Performance of AIOps Solutions
Lyu, Y., Li, H., Sayagh, M., Jiang, Z. M. (., & Hassan, A. E. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3447876 · Cited by 60
Towards Guidelines for Assessing Qualities of Machine Learning Systems
Siebert, J., Joeckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-58793-2_2 · Cited by 58
Cautionary Guidelines for Machine Learning Studies with Combinatorial Datasets
Zahrt, A. F., Henle, J. J., & Denmark, S. E. (2020) · ACS Combinatorial Science · doi:10.1021/acscombsci.0c00118 · Cited by 63
Deep Learning-Based Damage Mapping With InSAR Coherence Time Series
Stephenson, O. L., Köhne, T., Zhan, E., Cahill, B. E., Yun, S., Ross, Z. E., & Simons, M. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3084209 · Cited by 67
“When the training set Dt is split into its training and validation components, any correlation between time series from the same geographic area could lead to data leakage [55], whereby information from the validation set can also be found in the training set.”
BackgroundChen, H., Yim, A. Y. F. L., Griffith, G. R., de Jonge, W. J., Mannens, M. M. A. M., Ferrero, E. A., Henneman, P., & de Winther, M. P. J. (2019) · Frontiers in Immunology · doi:10.3389/fimmu.2019.02887 · Cited by 54
“As the test set should remain hidden from the training set, raw microarray and RNA-seq data was used instead of normalized data to prevent data leakage (56).”
MethodMachine Learning (In) Security: A Stream of Problems
Ceschin, F., Botacin, M. F., Bifet, A., Pfahringer, B., Oliveira, L. S., Gomes, H. M., & Gregio, A. R. A. (2023) · Digital Threats Research and Practice · doi:10.1145/3617897 · Cited by 64
“atches. However, when considering cybersecurity data, which are obtained from a stream, it is not real to consider data from different or mixed epochs to train and test a model (known as data leakage [87] or temporal inconsistency in this case), given that it could increase the detection accuracy because the model kno…”
BackgroundMaintenance intervention predictions using entity-embedding neural networks
Bukhsh, Z. A., Stipanović, I., Saeed, A., & Dorée, A. G. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103202 · Cited by 56
“However, for the sake of robust classifier and to avoid data leakage problem [42], all prediction tasks are trained on the same set of features data (see Table 2), where the output of one model is not used as a feature for learning the other task.”
MethodDomain knowledge integration into deep learning for typhoon intensity classification
Higa, M., Tanahara, S., Adachi, Y., Ishiki, N., Nakama, S., Yamada, H., Ito, K., Kitamoto, A., & Miyata, R. (2021) · Scientific Reports · doi:10.1038/s41598-021-92286-w · Cited by 47
“In contrast, to prevent data leakage, time-series data such as typhoon images should normally be split sequentially so that the timestamps of the training data are earlier than those of the test data 30 .”
Tunkiel, A. T., Sui, D., & Wiktorski, T. (2021) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2021.109760 · Cited by 44
Filho, A. D. P. C., Batista, A. F. D. M., & dos Santos, H. G. (2021) · Journal of Medical Internet Research · doi:10.2196/10969 · Cited by 44
“Data leakage occurs when one or more features used to train the algorithm has hidden within itself the result of the outcome, and is considered one of the most frequent mistakes in machine learning [3].”
BackgroundPrediction of Genetic Interactions Using Machine Learning and Network Properties
Madhukar, N. S., Elemento, O., & Pandey, G. (2015) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2015.00172 · Cited by 39
Soranzo, E., Guardiani, C., Chen, Y., Wang, Y., & Wu, W. (2022) · Acta Geotechnica · doi:10.1007/s11440-022-01783-3 · Cited by 41
“soil classification, design of deep foundations and tunnels and soil dynamics [35, 34,58, 64, 70, 72, 76, 74].”
BackgroundReeve, K. A., Ön, B. I., Havla, J. B., Burns, J., Gosteli, M., Alabsawi, A., Alayash, Z., Götschi, A., Seibold, H., Mansmann, U., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013606.pub2 · Cited by 59
Deep neural network-based heat radiation modelling between particles and between walls and particles
Tausendschön, J., & Radl, S. (2021) · International Journal of Heat and Mass Transfer · doi:10.1016/j.ijheatmasstransfer.2021.121557 · Cited by 40
Dataset artefacts in anti-spoofing systems: A case study on the ASVspoof 2017 benchmark
Chettri, B., Benetos, E., & Sturm, B. L. T. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3036777 · Cited by 35
“Data leakage can often lead to overestimation of model performance, producing too-good-to-be-true results [41].”
BackgroundWhy do probabilistic clinical models fail to transport between sites
Lasko, T. A., Strobl, E. V., & Stead, W. W. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01037-4 · Cited by 64
“Anover fi tmodelperformswellontrainingdatabutpoorlyontestdata,even though the two datasets are drawn from the same underlying distribution 25 .”
BackgroundDatta, A., Matlock, M. K., Dang, N. L., Moulin, T. A., Woeltje, K. F., Yanik, E. L., & Swamidass, S. J. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3033405 · Cited by 33
Empirical Study: Visual Analytics for Comparing Stacking to Blending Ensemble Learning
Chatzimparmpas, A., Martins, R. M., Kucher, K., & Kerren, A. (2021) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs52396.2021.00008 · Cited by 36
Neto, E. J. D. M., & Clementino, M. B. M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272287 · Cited by 56
Exploring Interpretability for Predictive Process Analytics
Sindhgatta, R., Ouyang, C., & Moreira, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65310-1_31 · Cited by 25
“All these observations reveal that the predictive model of single agg with XGBoost exhibits a problem of data leakage [12], “where information about the label of prediction that should not legitimately be available is present in the input”.”
BackgroundArnaud, V., Pellegrino, F., Keenan, S., St-Gelais, X., Mathevon, N., Levréro, F., & Coupé, C. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010325 · Cited by 25
“…can prevent from falling in the most obvious traps, but other sources of leakage can still prove problematic (for more information, please refer to [13] for a general discussion, [47] for a methodological approach in animal species acoustic monitoring, and [91] for an evaluation in music…”
Sha, H., Xu, P., Lin, M., Chen, P., & Dou, Q. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.116652 · Cited by 21
Benčević, M., Galić, I., Habijan, M., & Pižurica, A. (2022) · Applied Sciences · doi:10.3390/app12105217 · Cited by 25
Evaluating Attribution Methods in Machine Learning Interpretability
Ratul, Q. E. A., Serra, E., & Cuzzocrea, A. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671501 · Cited by 30
“It unifies six explanation techniques LIME [35], Shapley sampling values [37], DeepLIFT [38], QII [39], Layer-wise relevance propagation [40], Shapley regression values [41] by defining the class of additive feature attribution method.”
MethodPredicting severely imbalanced data disk drive failures with machine learning models
Ahmed, J., & Green, R. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100361 · Cited by 27
On the efficiency of user identification: a system-based approach
Malatras, A., Geneiatakis, D., & Vakalis, I. (2016) · International Journal of Information Security · doi:10.1007/s10207-016-0340-2 · Cited by 20
“While it would have been possible to monitor a greater variety of system properties and after careful examination of data characteristics, we opted to discard certain properties since they do not exhibit any type of variation and can lead to data leakages [44,45] and therefore influence the results.”
BackgroundForecasting strong aftershocks in earthquake clusters from northeastern Italy and western Slovenia
Gentili, S., & Giovambattista, R. D. (2020) · Physics of The Earth and Planetary Interiors · doi:10.1016/j.pepi.2020.106483 · Cited by 18
Improving credit risk assessment in P2P lending with explainable machine learning survival analysis
Bone-Winkel, G. F., & Reichenbach, F. (2024) · Digital Finance · doi:10.1007/s42521-024-00114-3 · Cited by 30
Janßen, R., Beck, A. J., Werner, J., Dellwig, O., Alneberg, J., Kreikemeyer, B., Maser, E., Böttcher, C., Achterberg, E. P., Andersson, A. F., et al. (2021) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.626048 · Cited by 17
The solar influencer next door: Predicting low income solar referrals and leads
Sigrin, B., Sekar, A., & Tome, E. (2021) · Energy Research & Social Science · doi:10.1016/j.erss.2021.102417 · Cited by 16
A Recommender System for Increasing Energy Efficiency of Solar-Powered Smart Homes
Meteier, Q., Kamali, M. E., Angelini, L., & Khaled, O. A. (2023) · Sensors · doi:10.3390/s23187974 · Cited by 15
Ulivi, M., Meroni, V., Orlandini, L. C., Prandoni, L., Rossi, N., Peretti, G. M., Dui, L. G., Mangiavini, L., & Ferrante, S. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103775 · Cited by 15
Considerations on Fairness-Aware Data Mining
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · 2012 IEEE 12th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2012.101 · Cited by 15
“Fairness in data mining can also be interpreted as a sub-notion of legitimacy, which means that models can be deployed in the real world [22].”
Folguera-Blasco, N., Boshier, F. A. T., Uatay, A., Pichardo-Almarza, C., Lai, M., Biasetti, J., Dearden, R., Gibbs, M., & Kimko, H. (2024) · Frontiers in Systems Biology · doi:10.3389/fsysb.2024.1380685 · Cited by 15
Interpretability of Machine Learning Models: How Can One Explain Machine Learning Models?
Czerwińska, U. (2022) · Tourism on the verge · doi:10.1007/978-3-030-88389-8_14 · Cited by 23
TabGNN: Multiplex Graph Neural Network for Tabular Data PredictionPreprint
Guo, X., Quan, Y., Zhao, H., Yao, Q., Li, Y., & Tu, W.-W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09127 · Cited by 54
“Considering that samples tend to be attached with timestamps, e.g., Apply time in Figure 1, it can lead to data leakage problem [18] if the temporal constraint is not processed properly.”
BackgroundMeta-Scaler: A Meta-Learning Framework for the Selection of Scaling Techniques
de Amorim, L. D., Cavalcanti, G., & Cruz, R. M. O. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3366615 · Cited by 10
“To avoid the data leakage problem [80], the scaling was applied as follows: for each iteration of the fivefold CV, the STs were fitted only for the training data and then used to transform both the training and test sets.”
Assessment of milk metabolites as biomarkers for predicting feed efficiency in dairy sheep
Marina, H., Arranz, J. J., Suárez‐Vega, A., Pelayo, R., Gutiérrez‐Gil, B., Toral, P. G., Hervás, G., Frutos, P., & Fonseca, P. A. D. S. (2024) · Journal of Dairy Science · doi:10.3168/jds.2023-23984 · Cited by 13
“This variability could be explained by the reduced sample size or other concepts such as data leakage (use of information in the training set, which is not expected to be available in the testing step) (Kaufman et al., 2011).”
BackgroundReproducing complex simulations of economic impacts of climate change with lower-cost emulators
Takakura, J., Fujimori, S., Takahashi, K., Hanasaki, N., Hasegawa, T., Hirabayashi, Y., Honda, Y., Iizumi, T., Park, C., Tamura, M., et al. (2021) · Geoscientific model development · doi:10.5194/gmd-14-3121-2021 · Cited by 16
The Real World Significance of Performance Prediction.
Pardos, Z. A., Wang, Q. Y., & Trivedi, S. (2012) · Cited by 21
Stein, S., Leng, C., Thornton, S. F., & Randrianandrasana, M. F. (2020) · Computational Materials Science · doi:10.1016/j.commatsci.2020.110053 · Cited by 12
Ukrainian News Corpus as Text Classification Benchmark
Panchenko, D., Maksymenko, D., Turuta, O., Luzan, M., Tytarenko, S., & Turuta, O. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-14841-5_37 · Cited by 15
Building a Reproducible Machine Learning PipelinePreprint
Sugimura, P., & Hartl, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.04570 · Cited by 10
“Updated data leads to two related problems: (1) leakage due to the use of data that was not available at the time of scoring [5] and (2) concept drift due to the use of training data which is not representative of data at the time of scoring [12].”
BackgroundMishra, L. N., & Senapati, B. (2026) · IEEE Access · doi:10.1109/access.2026.3655398 · Cited by 12
Disinformation techniques for entity resolution
Whang, S. E., & García-Molina, H. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505636 · Cited by 8
“Reference [7] defines the leakage of information in a general data mining context and provides detection and prevention techniques for leakage.”
BackgroundImproving lab-of-origin prediction of genetically engineered plasmids via deep metric learning
Soares, I. M., Camargo, F. H. F., Marques, A., & Crook, O. M. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00234-z · Cited by 9
“However, when training a machine learning algorithm, this may be perceived as data leakage between these sets 46 , as the model does not need to learn to extract different features to identify such sequences.”
BackgroundA Discussion on Generalization in Next-Activity Prediction
Abb, L., Pfeiffer, P., Fettke, P., & Rehse, J.-R. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-50974-2_2 · Cited by 8
Statically detecting data leakages in data science code
Subotić, P., Bojanić, U., & Stojić, M. (2022) · State Of the Art in Java Program Analysis · doi:10.1145/3520313.3534657 · Cited by 7
“A notable data science specific bug is a data leakage [6].”
Mapstone, C., & Płusa, B. (2025) · Development · doi:10.1242/dev.202066 · Cited by 7
A Multi-Learning Training Approach for Distinguishing Low and High Risk Cancer Patients
Povoa, L. V., Calvi, U. C. B., Lorena, A. C., Ribeiro, C. H. C., & da Silva, I. T. (2021) · IEEE Access · doi:10.1109/access.2021.3104820 · Cited by 6
AI Extension of SQuaRE Data Quality Model
Nakajima, S., & Nakatani, T. (2021) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c55045.2021.00053 · Cited by 7
Bogdanovski, O. P., Svenningsson, C., Månsson, S., Oxenstierna, A., & Sopasakis, A. (2023) · Agriculture · doi:10.3390/agriculture13040813 · Cited by 6
Time-resolved evaluation of compound repositioning predictions on a text-mined knowledge network
Mayers, M., Li, T. S., Queralt-Rosiñach, N., & Su, A. I. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3297-0 · Cited by 6
A Learning Based Approach for Real-Time Emotion Classification of Tweets
Janssens, O., Rik Van De Walle, B., & Hoecke, S. V. (2015) · Lecture notes in social networks · doi:10.1007/978-3-319-19003-7_7 · Cited by 7
“If the hashtags are kept in the dataset, data leakage [20] would be created which results in an unrealistically good classifier.”
BackgroundFonseca, P. A. D. S., Suárez‐Vega, A., Esteban‐Blanco, C., Marina, H., Pelayo, R., Gutiérrez‐Gil, B., & Arranz, J.-J. (2025) · BMC Genomics · doi:10.1186/s12864-025-11520-1 · Cited by 7
“An important issue here is data leakage [108], which occurs when training data contain information unavailable during testing.”
Static analysis by abstract interpretation against data leakage in machine learning
Urban, C., Subotić, P., & Drobnjaković, F. (2025) · Science of Computer Programming · doi:10.1016/j.scico.2025.103338 · Cited by 5
“Data leakage has been identified as a pervasive problem by the data science community (Kapoor and Narayanan, 2023; Kaufman et al., 2012; Nis, 2018).”
Deep Learning for Indonesia Standard Industrial Classification
Dwicahyo, M., & Yuniarto, B. (2020) · doi:10.1109/iceltics50595.2020.9315361 · Cited by 5
“But since we can not fully prevent information leakage [14] from the cartesian product of generation process, we apply the steps below to split train and test data: 1) Split data from guidebook to train and test with ratio 1:9, use smaller ratio for train data 2) Fine-tune the model with LSR…”
MethodA survey of computational methods for fossil data analysis
Žliobaitė, I., Puolamäki, K., Eronen, J. T., & Fortelius, M. (2017) · Evolutionary ecology research · Cited by 6
The impact of concept drift and data leakage on log level prediction models
Ouatiti, Y. E., Sayagh, M., Kerzazi, N., Adams, B., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10518-9 · Cited by 7
“Data leakage refers to situations where the usage of random train/test splits biases the model by giving it access to future information (Kaufman et al. 2011).”
Network Report: A Structured Description for Network Datasets
Zheng, X., Rossi, R. A., Ahmed, N. K., & Moritz, D. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557115 · Cited by 6
“Leakage refers to the use of information in the model training process that would not be expected to be available at prediction time, causing the predictive scores (metrics) to overestimate the model’s utility when run in a production environment [27].”
BackgroundKoh, R. G. L., Khan, M. A., Rashidiani, S., Hassan, S. I., Tucci, V., Liu, T., Nesovic, K., Kumbhare, D. A., & Doyle, T. E. (2023) · IEEE Access · doi:10.1109/access.2023.3316019 · Cited by 6
“Proper reporting is 232 crucial, as an incorrectly trained or improperly used algorithm 233 can lead to misrepresented results (e.g., higher accuracy due 234 to train-test contamination leading to data leakage) [65], [66], 235 [67], [68], [69].”
BackgroundZemke, N. R., Armand, E. J., Wang, W., Lee, S. Y., Zhou, J., Li, Y. E., Liu, H., Tian, W., Nery, J. R., Castanon, R., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.08.536119 · Cited by 4
Multivariate Time Series for Customs Revenue Forecasting Using LSTM Neural Networks
Cahyono, N. D., Sumpeno, S., & Setiiadi, E. (2023) · International Conference on Information Technology and Computing (ICITCOM) · doi:10.1109/icitcom60176.2023.10442562 · Cited by 3
Cluster Membership of Galaxies Using Multi-Layer Perceptron Neural Network
Hashimoto, Y., & Liu, C.-H. M. (2022) · Universe · doi:10.3390/universe8070339 · Cited by 3
“…use the trained model for the entire sample, e.g., when we naively apply the model to the entire dataset of galaxies, it may create at least two distinct data subsets with different prediction accuracy, by a mechanism similar to ‘data leakage’ during the validation process of the ML training [51].”
Influence of Weak Labels for Emotion Recognition of Tweets
Janssens, O., Verstockt, S., Mannens, E., Hoecke, S. V., & Rik Van De Walle, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-13817-6_12 · Cited by 3
“However, if the hashtags are kept in the dataset, data leakage [9] occurs which results in an unrealistically good classifier.”
BackgroundLee, E., Balasubramanian, H., Tsolias, A., Anakwe, S. U., Medalla, M., Shenoy, K. V., & Chandrasekaran, C. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.02.07.430135 · Cited by 4
“This results in data leakage (Moscovich and Rosset, 2019; Kaufman et al., 2011) which potentially hurts out-ofdataset performance.”
Estimated limits of organism-specific training for epitope predictionPreprint
Ashford, J., & Campelo, F. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.02.466801 · Cited by 3
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 134
“1 [23] and Qwen-2.”
Jacobucci, R., Shao, W., Kobrinsky, V., & Ammerman, B. A. (2026) · JMIR Mental Health · doi:10.2196/90581 · Cited by 4
“Performance estimates can be inflated by 100% ‐ 300% when the evaluation strategy is misaligned with the intended use case [35,36].”
Ebrahimi, M. H., & Niknam, S. A. (2026) · Machines · doi:10.3390/machines14070756 · Cited by 2
“The calculated parameters were then applied to transform the validation and test datasets [48].”
Convolutional and hybrid neural network for cluster membership
Hashimoto, Y., & Liu, C.-H. M. (2024) · Chinese Journal of Physics · doi:10.1016/j.cjph.2024.05.036 · Cited by 3
Simpler predictive models provide higher accuracy for ovarian cancer detection
Wood, D. E., Roy, J., & Ballew, B. J. (2025) · PeerJ · doi:10.7717/peerj.20525 · Cited by 2
Two New Philosophical Problems for Robo-Ethics
Buechner, J. (2018) · Information · doi:10.3390/info9100256 · Cited by 3
“A robo-agent trained by a data-set to recognize one kind of object might actually recognize another kind of object, because of, say, leakage in the data mining set (see [10] for a discussion of data leakage and [11,12] for examples of data leakage).”
BackgroundRemaining Useful Life Estimation of Bearings
Ferreira, H., & Sousa, A. C. D. (2021) · International Journal of Prognostics and Health Management · doi:10.36001/ijphm.2020.v11i2.2922 · Cited by 2
“We next describe how information may be indirectly leaked in the various phases of model fitting and evaluation (Kaufman et al., 2011).”
MethodDatastack: Unification of Heterogeneous Machine Learning Dataset Interfaces
Lübbering, M., Pielka, M., Henk, I., & Sifa, R. (2022) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew55742.2022.00014 · Cited by 4
Deep learning model for early prediction of material fracture in tensile testing
Jubair, F., Alhamayel, A., Aljaiose, R., & Darabkh, K. A. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09641-x · Cited by 2
Robust relationship extraction in the biomedical domain
Thomas, P. E. (2015) · edoc Publication server (Humboldt University of Berlin) · doi:10.18452/17372 · Cited by 2
A Comparative Analysis of Artificial Neural Network and Convolutional Neural Network Models
Bhattacharya, N., & Kumar, P. (2025) · Advances in psychology, mental health, and behavioral studies (APMHBS) book series · doi:10.4018/979-8-3693-7545-7.ch007 · Cited by 1
Metadata
- Authors
- Kaufman, S., Rosset, S., & Perlich, C.
- Affiliations
- Tel Aviv University (IL)
- Media6Degrees, New York, NY, USA
- ORCID
- Rosset, Saharon
- Venue
- Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2011-08-21
- Pages
- 556-563
- OpenAlex
- W2044879407
- Cited by
- 137
- References
- 22
- Citations by year
- 2026: 22 · 2025: 10 · 2024: 15 · 2023: 13 · 2022: 15 · 2021: 23 · 2020: 13 · 2019: 5 · 2018: 2 · 2017: 4 · 2016: 4 · 2015: 4 · 2014: 2 · 2013: 1 · 2012: 3
- FWCI
- 1.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- leakage, data mining, data mining competition, data management, leak detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.