The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Prediction error estimation: a comparison of resampling methods
Open accessAbstract
Motivation: In genomic studies, thousands of features are collected on relatively few samples. One of the goals of these studies is to build classifiers to predict the outcome of future observations. There are three inherent steps to this process: feature selection, model selection and prediction assessment. With a focus on prediction assessment, we compare several methods for estimating the ‘true’ prediction error of a prediction model in the presence of feature selection. Results: For small studies where features are selected from thousands of candidates, the resubstitution and simple split-sample estimates are seriously biased. In these small samples, leave-one-out cross-validation (LOOCV), 10-fold cross-validation (CV) and the .632+ bootstrap have the smallest bias for diagonal discriminant analysis, nearest neighbor and classification trees. LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis. Additionally, LOOCV, 5- and 10-fold CV, and the .632+ bootstrap have the lowest mean square error. The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios. Differences in performance among resampling methods are reduced as the number of specimens available increase. Contact: annette.molinaro@yale.edu Supplementary Information: A complete compilation of results and R code for simulations and analyses are available in Molinaro et al. (2005) (http://linus.nci.nih.gov/brb/TechReport.htm).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
R: A Language for Data Analysis and Graphics
Ihaka, R., & Gentleman, R. (1996) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1996.10474713 · Cited by 11,220
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,459
Bootstrap Methods and Their Application
Loughin, T. M., Davison, A. C., & Hinkley, D. V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669707 · Cited by 5,418
Rosenwald, A., Wright, G., Chan, W. C., Connors, J. M., Campo, E., Fisher, R. I., Gascoyne, R. D., Muller-Hermelink, H. K., Smeland, E. B., Giltnane, J. M., et al. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa012914 · Cited by 3,829
Self-citation
Cited by
A review of feature selection techniques in bioinformatics
Saeys, Y., Inza, I., & Larrañaga, P. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm344 · Cited by 5,244
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
“On realistic synthetic and real biological data, Molinaro et al. (2005) compared the bias of LOO, VFCV and .632+ bootstrap: The bias decreases with n t , and is generally minimal for LOO.”
InfluentialBias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485
“However, low variance methods can also have a large inherent bias [12] (either positive or negative).”
BackgroundSelf-citationMeinshausen, N., & Bühlmann, P. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00740.x · Cited by 2,215
Adaptive sparse polynomial chaos expansion based on least angle regression
Blatman, G., & Sudret, B. (2011) · Journal of Computational Physics · doi:10.1016/j.jcp.2010.12.021 · Cited by 1,502
Merghadi, A., Yunus, A. P., Dou, J., Whiteley, J., ThaiPham, B., Bui, D. T., Avtar, R., & Abderrahmane, B. (2020) · Earth-Science Reviews · doi:10.1016/j.earscirev.2020.103225 · Cited by 1,228
Berrar, D. (2019) · Encyclopedia of Bioinformatics and Computational Biology · doi:10.1016/b978-0-12-809633-8.20349-x · Cited by 958
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Molecular Diagnosis of Burkitt's Lymphoma
Dave, S. S., Fu, K., Wright, G. W., Lam, L. T., Kluin, P., Boerma, E.-J., Greiner, T. C., Weisenburger, D. D., Rosenwald, A., Ott, G., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa055759 · Cited by 897
Self-citationBlatman, G., & Sudret, B. (2010) · Probabilistic Engineering Mechanics · doi:10.1016/j.probengmech.2009.10.003 · Cited by 832
Lee, L. C., Liong, C.-Y., & Jemain, A. A. (2018) · The Analyst · doi:10.1039/c8an00599k · Cited by 795
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792
Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
Raschka, S. (2018) · arXiv · doi:10.48550/arxiv.1811.12808 · Cited by 735
Motwani, M., Dey, D., Berman, D. S., Germano, G., Achenbach, S., Al-Mallah, M. H., Andreini, D., Budoff, M. J., Cademartiri, F., Callister, T. Q., et al. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw188 · Cited by 692
Artificial Intelligence in Cardiovascular Imaging
Dey, D., Slomka, P. J., Leeson, P., Comaniciu, D., Shrestha, S., Sengupta, P. P., & Marwick, T. H. (2019) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.12.054 · Cited by 680
Dupuy, A., & Simon, R. M. (2007) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djk018 · Cited by 632
Self-citationAn Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Permutation Tests for Studying Classifier Performance
Ojala, M., & Garriga, G. C. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.108 · Cited by 555
Opportunities and Challenges for Machine Learning in Materials Science
Morgan, D., & Jacobs, R. (2020) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070218-010015 · Cited by 549
Statistical Evaluation of a Biomarker
Ray, P., Manach, Y. L., Riou, B., Houle, T. T., & Warner, D. S. (2010) · Anesthesiology · doi:10.1097/aln.0b013e3181d47604 · Cited by 503
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification
Baltruschat, I. M., Nickisch, H., Grass, M., Knopp, T., & Saalbach, A. (2019) · Scientific Reports · doi:10.1038/s41598-019-42294-8 · Cited by 455
Evaluating Random Forests for Survival Analysis Using Prediction Error Curves
Mogensen, U. B., Ishwaran, H., & Gerds, T. A. (2012) · Journal of Statistical Software · doi:10.18637/jss.v050.i11 · Cited by 443
A Gene Signature Predicting for Survival in Suboptimally Debulked Patients with Ovarian Cancer
Bonome, T., Levine, D. A., Shih, J., Randonovich, M., Pise-Masison, C. A., Bogomolniy, F., Ozbun, L., Brady, J., Barrett, J. C., Boyd, J., et al. (2008) · Cancer Research · doi:10.1158/0008-5472.can-07-6595 · Cited by 439
Optimally splitting cases for training and testing high dimensional classifiers
Dobbin, K. K., & Simon, R. M. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-31 · Cited by 392
Self-citationRoadmap for Developing and Validating Therapeutically Relevant Genomic Classifiers
Simon, R. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.02.8712 · Cited by 382
Self-citationUra, S., Honda, M., Yamashita, T., Ueda, T., Takatori, H., Nishino, R., Sunakozaka, H., Sakai, Y., Horimoto, K., & Kaneko, S. (2009) · Hepatology · doi:10.1002/hep.22749 · Cited by 376
Deep Learning for Prediction of Obstructive Disease From Fast Myocardial Perfusion SPECT
Betancur, J., Commandeur, F., Motlagh, M., Sharir, T., Einstein, A. J., Bokhari, S., Fish, M. B., Ruddy, T. D., Kaufmann, P., Sinusas, A. J., et al. (2018) · JACC: Cardiovascular Imaging · doi:10.1016/j.jcmg.2018.01.020 · Cited by 358
Permutation Tests for Studying Classifier Performance
Ojala, M., & Garriga, G. C. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859913 · Cited by 322
Machine learning and landslide studies: recent advances and applications
Tehrani, F. S., Calvello, M., Liu, Z., Zhang, L., & Lacasse, S. (2022) · Natural Hazards · doi:10.1007/s11069-022-05423-7 · Cited by 294
Kinetic models in industrial biotechnology – Improving cell factory performance
Almquist, J., Cvijovic, M., Hatzimanikatis, V., Nielsen, J., & Jirstrand, M. (2014) · Metabolic Engineering · doi:10.1016/j.ymben.2014.03.007 · Cited by 293
On the overestimation of random forest’s out-of-bag error
Janitza, S., & Hornung, R. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201904 · Cited by 278
A wavelet-based technique to predict treatment outcome for Major Depressive Disorder
Mumtaz, W., Xia, L., Mohd Yasin, M. A., Azhar Ali, S. S., & Malik, A. S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0171409 · Cited by 268
Yoon, J. (2021) · Computational Economics · doi:10.1007/s10614-020-10054-w · Cited by 256
Sartorius, D., Le Manach, Y., David, J.-S., Rancurel, E., Smail, N., Thicoïpé, M., Wiel, E., Ricard-Hibon, A., Berthier, F., Gueugniaud, P.-Y., et al. (2010) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181cc4a67 · Cited by 255
Simon, R. (2010) · Personalized Medicine · doi:10.2217/pme.09.49 · Cited by 206
Self-citationSong, Y., & Liò, P. (2010) · Journal of Biomedical Science and Engineering · doi:10.4236/jbise.2010.36078 · Cited by 191
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
Predictive accuracy and sources of variability in calculated free testosterone estimates
Sartorius, G., Ly, L. P., Sikaris, K., McLachlan, R., & Handelsman, D. J. (2009) · Annals of Clinical Biochemistry: International Journal of Laboratory Medicine · doi:10.1258/acb.2008.008171 · Cited by 175
Likelihood based observability analysis and confidence intervals for predictions of dynamic models
Kreutz, C., Raue, A., & Timmer, J. (2012) · BMC Systems Biology · doi:10.1186/1752-0509-6-120 · Cited by 134
“the Markov Chain Monte Carlo (MCMC) methods [11,12], and bootstrap based approaches [4,13] are numerically demanding and provide only rough approximations for ODE models.”
MethodTinker, L. F., Sarto, G. E., Howard, B. V., Huang, Y., Neuhouser, M. L., Mossavar-Rahmani, Y., Beasley, J. M., Margolis, K. L., Eaton, C. B., Phillips, L. S., et al. (2011) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.111.018648 · Cited by 122
Phosphosignature predicts dasatinib response in non-small cell lung cancer
Klammer, M., Kaminski, M., Zedler, A., Oppermann, F., Blencke, S., Marx, S., Müller, S., Tebbe, A., Godl, K., & Schaab, C. (2012) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m111.016410 · Cited by 118
Jamal, W., Das, S., Oprescu, I.-A., Maharatna, K., Apicella, F., & Sicca, F. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/4/046019 · Cited by 93
“[43] A.”
BackgroundSelecting Representative Data Sets
Borovicka, T., Jirina, M., Kordik, P., & Jiri, M. (2012) · Advances in Data Mining Knowledge Discovery and Applications · doi:10.5772/50787 · Cited by 94
Garami, J., Valikhani, A., Parkes, D., Haber, P., Mahlberg, J., Misiak, B., Frydecka, D., & Moustafa, A. A. (2019) · Psychological Reports · doi:10.1177/0033294118764918 · Cited by 87
Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation
Soneson, C., Gerster, S., & Delorenzi, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100335 · Cited by 84
Artificial Intelligence in Cardiovascular Medicine
Seetharam, K., Shrestha, S., & Sengupta, P. P. (2019) · Current Treatment Options in Cardiovascular Medicine · doi:10.1007/s11936-019-0728-1 · Cited by 76
A microRNA Signature Associated with Early Recurrence in Breast Cancer
Pérez-Rivas, L. G., Jerez, J. M., Carmona, R., de Luque, V., Vicioso, L., Claros, M. G., Viguera, E., Pajares, B., Sánchez, A., Ribelles, N., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0091884 · Cited by 67
Measuring the stability of results from supervised statistical learning
Philipp, M., Rusch, T., Hornik, K., & Strobl, C. (2018) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2018.1473779 · Cited by 56
Leveraging Cheminformatics Strategies for Inorganic Discovery: Application to Redox Potential Design
Janet, J. P., Gani, T. Z. H., Steeves, A. H., Ioannidis, E. I., & Kulik, H. J. (2017) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.7b00808 · Cited by 56
Ghazal, M. M., & Hammad, A. (2022) · International Journal of Construction Management · doi:10.1080/15623599.2020.1738205 · Cited by 52
Lee, J. T., Park, E., Hwang, J.-M., Jung, T.-D., & Park, D. (2020) · Scientific Reports · doi:10.1038/s41598-020-71713-4 · Cited by 49
Development of a Brief Version of the Current Opioid Misuse Measure (COMM): The COMM-9
McCaffrey, S. A., Black, R. A., Villapiano, A. J., Jamison, R. N., & Butler, S. F. (2019) · Pain Medicine · doi:10.1093/pm/pnx311 · Cited by 35
Probabilistic classifiers with high-dimensional data
Kim, K. I., & Simon, R. (2011) · Biostatistics · doi:10.1093/biostatistics/kxq069 · Cited by 23
Self-citationMetabolomic changes associated with frontotemporal lobar degeneration syndromes
Murley, A. G., Jones, P. S., Coyle Gilchrist, I., Bowns, L., Wiggins, J., Tsvetanov, K. A., & Rowe, J. B. (2020) · Journal of Neurology · doi:10.1007/s00415-020-09824-1 · Cited by 18
Kim, H. B., Nguyen, H. T., Jin, Q., Tamby, S., Gelaf Romer, T., Sung, E., Liu, R., Greenstein, J. L., Suarez, J. I., Storm, C., et al. (2022) · Anaesthesia Critical Care & Pain Medicine · doi:10.1016/j.accpm.2021.101015 · Cited by 18
Holistic AI analysis of hybrid cardiac perfusion images for mortality prediction
Marcinkiewicz, A. M., Zhang, W., Shanbhag, A., Miller, R. J. H., Lemley, M., Ramirez, G., Buchwald, M., Killekar, A., Kavanagh, P. B., Feher, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01526-0 · Cited by 12
“…more accurate evaluation 40 ; 2) maximizing data utilization while minimizing over fi tting and cross-contamination of information among data splits 41 ; 3) ensuring each data point contributes to the test set exactly once, providing independent and non-overlapping predictions for robust…”
Sun, Z., Du, X., Song, F., Wang, S., Ni, M., Li, L., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688831 · Cited by 8
Zhou, J., Shanbhag, A. D., Han, D., Marcinkiewicz, A. M., Buchwald, M., Miller, R. J. H., Killekar, A., Manral, N., Grodecki, K., Geers, J., et al. (2025) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jeaf007 · Cited by 6
Thermodynamics shape the in vivo enzyme burden of glycolytic pathways
Khana, D. B., Jen, A., Shishkova, E., Thusoo, E., Williams, J., Henkel, A., Stevenson, D. M., Coon, J. J., & Amador-Noguez, D. (2025) · doi:10.1101/2025.01.31.635972 · Cited by 5
“S1), with low normalized root mean square error (NRMSE) (Table S4), as determined via leave-one-out cross-validation (Materials and Methods) (49–51).”
Metadata
- Authors
- Molinaro, A. M., Simon, R., & Pfeiffer, R. M.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2005-05-19
- Volume
- 21
- Issue
- 15
- Pages
- 3301-3307
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W2105981176
- Cited by
- 1,339
- References
- 29
- FWCI
- 13.25
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.