Cross-validation pitfalls when selecting and assessing regression and classification models
Open accessAbstract
BACKGROUND: We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches. METHODS: We describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case. RESULTS: We show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models. CONCLUSIONS: We demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
All ML modeling was performed using stratified and nested cross-validation.
“All the ML modeling was performed using stratified and nested cross-validation [71,72].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
A Simplex Method for Function Minimization
Nelder, J. A., & Mead, R. (1965) · The Computer Journal · doi:10.1093/comjnl/7.4.308 · Cited by 28,905 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909 · see citing works
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · see citing works
(2012) · Cited by 8,009 · see citing works
A survey of cross-validation procedures for model selectionPreprint
Arlot, S., & Célisse, A. (2009) · Cited by 3,627 · see citing works
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · see citing works
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · see citing works
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 7× · see citing works
“It is important to note that both Stone [2] and Varma and Simon [9] use leave-one-out cross-validation, while we use V-fold cross-validation.”
BackgroundMethodResultInfluentialSelection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,487 · see citing works
The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
Allen, D. M. (1974) · Technometrics · doi:10.1080/00401706.1974.10489157 · Cited by 1,469 · see citing works
Submodel Selection and Evaluation in Regression. The X-Random Case
Breiman, L., & Spector, P. C. (1992) · International Statistical Review · doi:10.2307/1403680 · Cited by 689 · see citing works
Derivation and Validation of Toxicophores for Mutagenicity Prediction
Kazius, J., McGuire, R., & Bursi, R. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm040835a · Cited by 668 · see citing works
BURMAN, P. (1989) · Biometrika · doi:10.1093/biomet/76.3.503 · Cited by 634 · see citing works
Ridge regression iterative estimation of the biasing parameter
Hoerl, A. E., & Kennard, R. W. (1976) · Communication in Statistics- Theory and Methods · doi:10.1080/03610927608827333 · Cited by 218 · see citing works
Asymptotics of cross-validated risk estimation in estimator selection and performance assessment
Dudoit, S., & van der Laan, M. J. (2005) · Statistical Methodology · doi:10.1016/j.stamet.2005.02.003 · Cited by 164 · see citing works
General Melting Point Prediction Based on a Diverse Compound Data Set and Artificial Neural Networks
Karthikeyan, M., Glen, R. C., & Bender, A. (2005) · Journal of Chemical Information and Modeling · doi:10.1021/ci0500132 · Cited by 145 · see citing works
Assessing the reliability of a QSAR model's predictions
He, L., & Jurs, P. C. (2005) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2005.03.003 · Cited by 120 · see citing works
Perspectives in Probability and Statistics. Papers in Honour of M. S. Bartlett.
Hajnal, J., & Gani, J. (1978) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344458 · Cited by 100 · see citing works
A bias correction for the minimum error rate in cross-validation
Tibshirani, R. J., & Tibshirani, R. J. (2009) · The Annals of Applied Statistics · doi:10.1214/08-aoas224 · Cited by 99 · see citing works
The Use of Rule‐Based and QSPR Approaches in ADME Profiling: A Case Study on Caco‐2 Permeability
Pham‐The, H., González‐Álvarez, I., Bermejo, M., Garrigues, T., Le‐Thi‐Thu, H., & Cabrera‐Pérez, M. Á. (2013) · Molecular Informatics · doi:10.1002/minf.201200166 · Cited by 62 · see citing works
Modeling Phospholipidosis Induction: Reliability and Warnings
Goracci, L., Ceccarelli, M., Bonelli, D., et al. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400113t · Cited by 44 · see citing works
V-fold cross-validation improved: V-fold penalizationPreprint
Arlot, S. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0802.0566 · Cited by 37 · see citing works
Bernau, C., Augustin, T., & Boulesteix, A. (2013) · Biometrics · doi:10.1111/biom.12041 · Cited by 21 · see citing works
Selection bias in working with the top genes in supervised classification of tissue samples
Zhu, X., Ambroise, C., & McLachlan, G. J. (2005) · Statistical Methodology · doi:10.1016/j.stamet.2005.09.011 · Cited by 16 · see citing works
A Mathematical Model for Prediction of Drug Molecule Diffusion Across the Blood-Brain Barrier
Burns, J., Weaver, D. F., Burns, J., et al. (2004) · Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques · doi:10.1017/s0317167100003759 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901 · see citing works
On-line application of near infrared (NIR) spectroscopy in food production
Porep, J. U., Kammerer, D. R., & Carle, R. (2015) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2015.10.002 · Cited by 611 · see citing works
El-Sappagh, S., Alonso, J. M., Islam, S. M. R., Sultan, A. M., & Kwak, K. S. (2021) · Scientific Reports · doi:10.1038/s41598-021-82098-3 · Cited by 363 · see citing works
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185 · see citing works
Machine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174 · see citing works
Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling
Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Camara, A., Mac Kain, A., Saillard, C., & Schiratti, J.-B. (2023) · doi:10.1101/2023.07.21.23292757 · Cited by 148 · see citing works
Opoku Asare, K., Terhorst, Y., Vega, J., Peltonen, E., Lagerspetz, E., & Ferreira, D. (2021) · JMIR mHealth and uHealth · doi:10.2196/26540 · Cited by 145 · see citing works
Tseng, Y.-J., Wang, H.-Y., Lin, T.-W., Lu, J.-J., Hsieh, C.-H., & Liao, C.-T. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.11768 · Cited by 89 · see citing works
Analyzing Raman spectroscopic data
Ryabchykov, O., Guo, S., & Bocklitz, T. (2019) · Physical Sciences Reviews · doi:10.1515/psr-2017-0043 · Cited by 74 · see citing works
Ghasemzadeh, H., Hillman, R. E., & Mehta, D. D. (2024) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2023_jslhr-23-00273 · Cited by 72 · see citing works
Chandrashekar, P. B., Alatkar, S., Wang, J., Hoffman, G. E., He, C., Jin, T., Khullar, S., Bendl, J., Fullard, J. F., Roussos, P., et al. (2023) · Genome Medicine · doi:10.1186/s13073-023-01248-6 · Cited by 37 · see citing works
Machine learning for predicting levetiracetam treatment response in temporal lobe epilepsy
Croce, P., Ricci, L., Pulitano, P., Boscarino, M., Zappasodi, F., Lanzone, J., Narducci, F., Mecarelli, O., Di Lazzaro, V., Tombini, M., et al. (2021) · Clinical Neurophysiology · doi:10.1016/j.clinph.2021.08.024 · Cited by 38 · see citing works
Dallora, A. L., Kvist, O., Berglund, J. S., Ruiz, S. D., Boldt, M., Flodmark, C.-E., & Anderberg, P. (2020) · JMIR Medical Informatics · doi:10.2196/18846 · Cited by 13 · see citing works
Pairing Machine Learning and Clinical Psychology: How You Evaluate Predictive Performance Matters
Jacobucci, R., Littlefield, A. K., Millner, A. J., Kleiman, E., & Steinley, D. (2020) · doi:10.31234/osf.io/2yber · Cited by 12 · see citing works
Teghipco, A., Newman-Norlund, R., Fridriksson, J., Rorden, C., & Bonilha, L. (2024) · Communications Medicine · doi:10.1038/s43856-024-00541-8 · Cited by 11 · see citing works
- Authors
- Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S.
- Venue
- Journal of Cheminformatics
- Published
- 2014-03-29
- Volume
- 6
- Issue
- 1
- Pages
- 10-10
- ISSN
- 1758-2946 (ISSN-L)
- PMC
- PMC3994246
- OpenAlex
- W2154290668
- Cited by
- 1,090
- References
- 31
- Citations by year
- 2026: 146 · 2025: 127 · 2024: 128 · 2023: 132 · 2022: 143 · 2021: 124 · 2020: 101 · 2019: 55 · 2018: 59 · 2017: 29 · 2016: 20 · 2015: 22 · 2014: 4
- FWCI
- 25.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Cheminformatics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- cross-validation, model selection, model evaluation, repeated cross-validation, nested cross-validation, feature selection, double cross-validation, QSAR, prediction error estimation, cloud computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
