The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation
Open accessAbstract
Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation (Varma and Simon in BMC Bioinform 7(1):91, 2006) and a method by Tibshirani and Tibshirani (Ann Appl Stat 822-829, 2009), BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based statistical criterion we stop training of models on new folds of inferior (with high probability) configurations. We name the method Bootstrap Bias Corrected with Dropping CV (BBCD-CV) that is both efficient and provides accurate performance estimates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 3×
“2005); around the same time (Varma and Simon 2006), report a bias in error estimation when using K-fold crossvalidation, and suggest the use of the NCV protocol as an almost unbiased estimate of the true performance.”
BackgroundMethodTHE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES
COCHRAN, W. G. (1950) · Biometrika · doi:10.1093/biomet/37.3-4.256 · Cited by 1,465
Cross-validation pitfalls when selecting and assessing regression and classification models
Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-10 · Cited by 1,090
Mining Free-Text Medical Notes for Suicide Risk Assessment
Adamou, M., Antoniou, G., Greasidou, E., Lagani, V., Charonyktakis, P., & Tsamardinos, I. (2018) · Proceedings of the 10th Hellenic Conference on Artificial Intelligence · doi:10.1145/3200947.3201020 · Cited by 22
Self-citation
Research works printed but not linked: 36
- Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. · DOI · Paper page
- Bernau, C., Augustin, T., & Boulesteix, A. L. (2013). Correcting the optimal resampling-based error rate by estimating the error rate of wrapper algorithms. Biometrics, 69(3), 693–702. · DOI
- Borboudakis, G., Stergiannakos, T., Frysali, M., Klontzas, E., Tsamardinos, I., & Froudakis, G. E. (2017). Chemically intuited, large-scale screening of MOFs by machine learning techniques. npj Computational Materials, 3(1), 40. · DOI · Paper page
- Boulesteix, A. L., & Strobl, C. (2009). Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction. BMC Medical Research Methodology, 9(1), 85. · DOI
- Braga-Neto, U. M., & Dougherty, E. R. (2004). Is cross-validation valid for small-sample microarray classification? Bioinformatics, 20(3), 374–380. · DOI · Paper page
- Ding, Y., Tang, S., Liao, S. G., Jia, J., Oesterreich, S., Lin, Y., et al. (2014). Bias correction for selecting the minimal-error classifier from many machine learning models. Bioinformatics, 30(22), 3152–3158. · DOI
- Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Boca Raton: CRC Press. · DOI · Paper page
- Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. · DOI · Paper page
- Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22. · DOI · Paper page
- Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675–701. · DOI · Paper page
- Garnett, R., Osborne, M. A., & Roberts, S. J. (2010). Bayesian optimization for sensor set selection. In: Proceedings of the 9th ACM/IEEE international conference on information processing in sensor networks, (pp. 209–219).
- Greasidou, E. (2017). Bias correction of the cross-validation performance estimate and speed up of its execution time. Master’s thesis, University of Crete, School of Sciences and Engineering, Computer Science Department.
- Guyon, I., Alamdari, A.R.S.A., Dror, G., & Buhmann, J.M. (2006). Performance prediction challenge. In: The 2006 IEEE International Joint Conference on Neural Network Proceedings, (pp. 1649–1656). IEEE.
- Guyon, I., Bennett, K., Cawley, G., Escalante, H. J., Escalera, S., Ho, T. K., Macià, N., Ray, B., Saeed, M., Statnikov, A., & Viegas, E. (2015). Design of the 2015 ChaLearn AutoML Challenge. In: Proceedings of IJCNN · DOI
- Guyon, I., Gunn, S., Ben-Hur, A., & Dror, G. (2004). Result analysis of the NIPS 2003 feature selection challenge. In: Advances in neural information processing systems, (pp. 545–552).
- Harrell, F. E., Califf, R. M., Pryor, D. B., Lee, K. L., & Rosati, R. A. (1982). Evaluating the yield of medical tests. Journal of the American medical association, 247(18), 2543–2546. · DOI · Paper page
- Iizuka, N., Oka, M., Yamada-Okabe, H., Nishida, M., Maeda, Y., Mori, N., et al. (2003). Oligonucleotide microarray for prediction of early intrahepatic recurrence of hepatocellular carcinoma after curative resection. Lancet, 361(9361), 923–929. · DOI · Paper page
- Jensen, D. D., & Cohen, P. R. (2000). Multiple comparisons in induction algorithms. Machine Learning, 38(3), 309–338. · DOI · Paper page
- Jensen, J. L. W. V. (1906). Sur les fonctions convexes et les Inégalités entre les valeurs moyennes. Acta mathematica, 30(1), 175–193. · DOI · Paper page
- Krueger, T., Panknin, D., & Braun, M. (2015). Fast cross-validation via sequential testing. Journal of Machine Learning Research, 16, 1103–1155.
- Lagani, V., Athineou, G., Farcomeni, A., Tsagris, M., & Tsamardinos, I., et al. (2017). Feature Selection with the R Package MXM: Discovering Statistically Equivalent Feature Subsets. Journal of Statistical Software 80(i07). · DOI
- Maron, O., & Moore, A. W. (1994). Hoeffding races: Accelerating model selection search for classification and function approximation. In: Advances in neural information processing systems, (pp. 59–59).
- Nankervis, J. C. (2005). Computational algorithms for double bootstrap confidence intervals. Computational Statistics & Ddata Analysis, 49(2), 461–475. · doi:10.1016/j.csda.2004.05.023
- Orfanoudaki, G., Markaki, M., Chatzi, K., Tsamardinos, I., & Economou, A. (2017). MatureP: Prediction of secreted proteins with exclusive information from their mature regions. Scientific Reports, 7(1), 3263. · DOI
- Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1(3), 317–328. · DOI · Paper page
- Schwarz, G., et al. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. · DOI · Paper page
- Seibold, H., Bernau, C., Boulesteix, A.L., & De Bin, R. (2017). On the choice and influence of the number of boosting steps for high-dimensional linear cox-models. Computational Statistics. https://doi.org/10.1007/s00180-017-0773-8 . · DOI
- Simantiraki, O., Charonyktakis, P., Pampouchidou, A., Tsiknakis, M., & Cooke, M. (2017). Glottal source features for automatic speech-based depression assessment. Proceedings of Interspeech, 2017, 2700–2704. · DOI
- Statnikov, A., Aliferis, C. F., Tsamardinos, I., Hardin, D., & Levy, S. (2005). A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. Bioinformatics, 21(5), 631–643. · DOI · Paper page
- Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267–288. · DOI · Paper page
- Tibshirani, R. J., & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. The Annals of Applied Statistics, 3(2), 822–829. · DOI
- Tsamardinos, I., Rakhshani, A., & Lagani, V. (2015). Performance-estimation properties of cross-validation-based protocols with simultaneous hyper-parameter optimization. International Journal on Artificial Intelligence Tools, 24(05), 1540,023. · DOI
- Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann.
- Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2016). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann.
- Yousefi, M. R., Hua, J., & Dougherty, E. R. (2011). Multiple-rule bias in the comparison of classification rules. Bioinformatics, 27(12), 1675–1683. · doi:10.1093/bioinformatics/btr262
- Zheng, A. X., & Bilenko, M. (2013). Lazy paired hyper-parameter tuning. In: IJCAI
Cited by
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660
Marcos-Zambrano, L. J., Karaduzovic-Hadziabdic, K., Loncar Turukalo, T., Przymus, P., Trajkovik, V., Aasmets, O., Berland, M., Gruca, A., Hasic, J., Hron, K., et al. (2021) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.634511 · Cited by 416
Self-citationHo, S. Y., Phua, K., Wong, L., & Bin Goh, W. W. (2020) · Patterns · doi:10.1016/j.patter.2020.100129 · Cited by 247
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
Mining Free-Text Medical Notes for Suicide Risk Assessment
Adamou, M., Antoniou, G., Greasidou, E., Lagani, V., Charonyktakis, P., & Tsamardinos, I. (2018) · Proceedings of the 10th Hellenic Conference on Artificial Intelligence · doi:10.1145/3200947.3201020 · Cited by 22
Self-citation
Metadata
- Authors
- Tsamardinos, I., Greasidou, E., & Borboudakis, G.
- Venue
- Machine Learning
- Published
- 2018-05-09
- Volume
- 107
- Issue
- 12
- Pages
- 1895-1922
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2746117291
- Cited by
- 211
- References
- 9
- FWCI
- 9.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- Funders
- European Research Council (617393)
- Toshiba of Europe
- Gnosis Data Analysis PC
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation. https://app.cochat.ai/papers/10.1007/s10994-018-5714-4
- The paper
- Tsamardinos, I., Greasidou, E., & Borboudakis, G. (2018). Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation. Machine Learning. https://doi.org/10.1007/s10994-018-5714-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.