Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 211

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation

Open access
Tsamardinos, Ioannis, Greasidou, Elissavet, Borboudakis, Giorgos · 2018 · Machine Learning

Abstract

Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation (Varma and Simon in BMC Bioinform 7(1):91, 2006) and a method by Tibshirani and Tibshirani (Ann Appl Stat 822-829, 2009), BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based statistical criterion we stop training of models on new folds of inferior (with high probability) configurations. We name the method Bootstrap Bias Corrected with Dropping CV (BBCD-CV) that is both efficient and provides accurate performance estimates.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10994-018-5714-4

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 9 references

Research works printed but not linked: 36

  1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. · DOI · Paper page
  2. Bernau, C., Augustin, T., & Boulesteix, A. L. (2013). Correcting the optimal resampling-based error rate by estimating the error rate of wrapper algorithms. Biometrics, 69(3), 693–702. · DOI
  3. Borboudakis, G., Stergiannakos, T., Frysali, M., Klontzas, E., Tsamardinos, I., & Froudakis, G. E. (2017). Chemically intuited, large-scale screening of MOFs by machine learning techniques. npj Computational Materials, 3(1), 40. · DOI · Paper page
  4. Boulesteix, A. L., & Strobl, C. (2009). Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction. BMC Medical Research Methodology, 9(1), 85. · DOI
  5. Braga-Neto, U. M., & Dougherty, E. R. (2004). Is cross-validation valid for small-sample microarray classification? Bioinformatics, 20(3), 374–380. · DOI · Paper page
  6. Ding, Y., Tang, S., Liao, S. G., Jia, J., Oesterreich, S., Lin, Y., et al. (2014). Bias correction for selecting the minimal-error classifier from many machine learning models. Bioinformatics, 30(22), 3152–3158. · DOI
  7. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Boca Raton: CRC Press. · DOI · Paper page
  8. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. · DOI · Paper page
  9. Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22. · DOI · Paper page
  10. Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675–701. · DOI · Paper page
  11. Garnett, R., Osborne, M. A., & Roberts, S. J. (2010). Bayesian optimization for sensor set selection. In: Proceedings of the 9th ACM/IEEE international conference on information processing in sensor networks, (pp. 209–219).
  12. Greasidou, E. (2017). Bias correction of the cross-validation performance estimate and speed up of its execution time. Master’s thesis, University of Crete, School of Sciences and Engineering, Computer Science Department.
  13. Guyon, I., Alamdari, A.R.S.A., Dror, G., & Buhmann, J.M. (2006). Performance prediction challenge. In: The 2006 IEEE International Joint Conference on Neural Network Proceedings, (pp. 1649–1656). IEEE.
  14. Guyon, I., Bennett, K., Cawley, G., Escalante, H. J., Escalera, S., Ho, T. K., Macià, N., Ray, B., Saeed, M., Statnikov, A., & Viegas, E. (2015). Design of the 2015 ChaLearn AutoML Challenge. In: Proceedings of IJCNN · DOI
  15. Guyon, I., Gunn, S., Ben-Hur, A., & Dror, G. (2004). Result analysis of the NIPS 2003 feature selection challenge. In: Advances in neural information processing systems, (pp. 545–552).
  16. Harrell, F. E., Califf, R. M., Pryor, D. B., Lee, K. L., & Rosati, R. A. (1982). Evaluating the yield of medical tests. Journal of the American medical association, 247(18), 2543–2546. · DOI · Paper page
  17. Iizuka, N., Oka, M., Yamada-Okabe, H., Nishida, M., Maeda, Y., Mori, N., et al. (2003). Oligonucleotide microarray for prediction of early intrahepatic recurrence of hepatocellular carcinoma after curative resection. Lancet, 361(9361), 923–929. · DOI · Paper page
  18. Jensen, D. D., & Cohen, P. R. (2000). Multiple comparisons in induction algorithms. Machine Learning, 38(3), 309–338. · DOI · Paper page
  19. Jensen, J. L. W. V. (1906). Sur les fonctions convexes et les Inégalités entre les valeurs moyennes. Acta mathematica, 30(1), 175–193. · DOI · Paper page
  20. Krueger, T., Panknin, D., & Braun, M. (2015). Fast cross-validation via sequential testing. Journal of Machine Learning Research, 16, 1103–1155.
  21. Lagani, V., Athineou, G., Farcomeni, A., Tsagris, M., & Tsamardinos, I., et al. (2017). Feature Selection with the R Package MXM: Discovering Statistically Equivalent Feature Subsets. Journal of Statistical Software 80(i07). · DOI
  22. Maron, O., & Moore, A. W. (1994). Hoeffding races: Accelerating model selection search for classification and function approximation. In: Advances in neural information processing systems, (pp. 59–59).
  23. Nankervis, J. C. (2005). Computational algorithms for double bootstrap confidence intervals. Computational Statistics & Ddata Analysis, 49(2), 461–475. · doi:10.1016/j.csda.2004.05.023
  24. Orfanoudaki, G., Markaki, M., Chatzi, K., Tsamardinos, I., & Economou, A. (2017). MatureP: Prediction of secreted proteins with exclusive information from their mature regions. Scientific Reports, 7(1), 3263. · DOI
  25. Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1(3), 317–328. · DOI · Paper page
  26. Schwarz, G., et al. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. · DOI · Paper page
  27. Seibold, H., Bernau, C., Boulesteix, A.L., & De Bin, R. (2017). On the choice and influence of the number of boosting steps for high-dimensional linear cox-models. Computational Statistics. https://doi.org/10.1007/s00180-017-0773-8 . · DOI
  28. Simantiraki, O., Charonyktakis, P., Pampouchidou, A., Tsiknakis, M., & Cooke, M. (2017). Glottal source features for automatic speech-based depression assessment. Proceedings of Interspeech, 2017, 2700–2704. · DOI
  29. Statnikov, A., Aliferis, C. F., Tsamardinos, I., Hardin, D., & Levy, S. (2005). A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis. Bioinformatics, 21(5), 631–643. · DOI · Paper page
  30. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267–288. · DOI · Paper page
  31. Tibshirani, R. J., & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. The Annals of Applied Statistics, 3(2), 822–829. · DOI
  32. Tsamardinos, I., Rakhshani, A., & Lagani, V. (2015). Performance-estimation properties of cross-validation-based protocols with simultaneous hyper-parameter optimization. International Journal on Artificial Intelligence Tools, 24(05), 1540,023. · DOI
  33. Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann.
  34. Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2016). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann.
  35. Yousefi, M. R., Hua, J., & Dougherty, E. R. (2011). Multiple-rule bias in the comparison of classification rules. Bioinformatics, 27(12), 1675–1683. · doi:10.1093/bioinformatics/btr262
  36. Zheng, A. X., & Bilenko, M. (2013). Lazy paired hyper-parameter tuning. In: IJCAI

Cited by

See all 211

Metadata

Authors
Tsamardinos, I., Greasidou, E., & Borboudakis, G.
Venue
Machine Learning
Published
2018-05-09
Volume
107
Issue
12
Pages
1895-1922
ISSN
  • 0885-6125 (ISSN-L)
  • 1573-0565
OpenAlex
W2746117291
Cited by
211
References
9
FWCI
9.19
Type
journal article
Language
en
Open access
hybrid
Licence
creativecommons.org/licenses/by/4.0 (tdm)
Funders
  • European Research Council (617393)
  • Toshiba of Europe
  • Gnosis Data Analysis PC

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation. https://app.cochat.ai/papers/10.1007/s10994-018-5714-4
The paper
Tsamardinos, I., Greasidou, E., & Borboudakis, G. (2018). Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation. Machine Learning. https://doi.org/10.1007/s10994-018-5714-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.