Estimating the error rate of a prediction rule: improvement on cross-validation
Abstract
We construct a prediction rule on the basis of some data, and then wish to estimate the error rate of this rule in classifying future observations. Cross-validation provides a nearly unbiased estimate, using only the original data. Cross-validation turns out to be related closely to the bootstrap estimate of the error rate. This article has two purposes: to understand better the theoretical basis of the prediction problem, and to investigate some related estimators, which seem to offer considerably improved estimation in small samples.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The paper's statistical analysis supports model αβγδζ, in which there are two distinct populations of superarchaics, one contributing to archaics and another contributing to the ancestors of modern humans.
“Our statistical analysis supports model αβγδζ , in which there are two distinct populations of superarchaics, one contributing to archaics and another contributing to the ancestors of modern humans.”
From The Genetic Legacy of Introgression from Two Deeply Divergent Lineages in the Genus <i>Homo</i> (2026) · cites this paper as [6]
The review excluded studies that used cross-validation approaches such as leave-one-out cross-validation.
“used cross-validation approaches such as 'leave-one-out' cross-validation (Efron 1983);”
From Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma (2018) · cites this paper as Efron 1983
Cites
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602 · see citing works
Self-citationCross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · see citing works
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
Self-citationJournal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229 · see citing works
Self-citationThe Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · see citing works
The Jackknife: The Bootstrap and Other Resampling Plans.
Low, L. Y., & Efron, B. (1983) · Journal of the American Statistical Association · doi:10.2307/2288218 · Cited by 2,728 · see citing works
Self-citationThe Influence Curve and its Role in Robust Estimation
Hampel, F. R. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10482962 · Cited by 2,554 · see citing works
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,508 · see citing works
The Jackknife Estimate of Variance
Efron, B., & Stein, C. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345462 · Cited by 1,424 · see citing works
Self-citationA Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation
Efron, B., & Gong, G. D. (1983) · The American Statistician · doi:10.2307/2685844 · Cited by 1,160 · see citing works
Self-citationThe Efficiency of Logistic Regression Compared to Normal Discriminant Analysis
Efron, B. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10480319 · Cited by 546 · see citing works
Self-citationEstimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.2307/1266219 · Cited by 463 · see citing works
Asymptotics for and against cross-validation
Stone, M. (1977) · Biometrika · doi:10.1093/biomet/64.1.29 · Cited by 257 · see citing works
Using Subsample Values as Typical Values
Hartigan, J. A. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501057 · Cited by 130 · see citing works
A Note on the Modified Likelihood for Density Estimation
Wong, W. H. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477995 · Cited by 21 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Efron, B., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013815 · Cited by 6,033 · see citing works
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Radiomics: the process and the challenges
Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S. A., Schabath, M. B., Forster, K., Aerts, H. J. W. L., Dekker, A., Fenstermacher, D., et al. (2012) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2012.06.010 · Cited by 2,258 · see citing works
“have pioneered the work, studied various cross-validation methods and proposed unbiased error estimation called the bootstrap [93,94].”
MethodLinear Model Selection by Cross-validation
Shao, J. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476299 · Cited by 1,790 · see citing works
Shen, W., Punyanitya, M., Wang, Z., Gallagher, D., St.-Onge, M.-P., Albu, J., Heymsfield, S. B., & Heshka, S. (2004) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00744.2004 · Cited by 1,692 · see citing works
Benza, R. L., Miller, D. P., Gomberg-Maitland, M., Frantz, R. P., Foreman, A. J., Coffey, C. S., Frost, A., Barst, R. J., Badesch, D. B., Elliott, C. G., et al. (2010) · Circulation · doi:10.1161/circulationaha.109.898122 · Cited by 1,547 · see citing works
Machine learning in bioinformatics
Larrañaga, P., Calvo, B., Santana, R., Bielza, C., Galdiano, J., Inza, I., Lozano, J. A., Armañanzas, R., Santafé, G., Pérez, A., et al. (2006) · Briefings in Bioinformatics · doi:10.1093/bib/bbk007 · Cited by 867 · see citing works
Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
Raschka, S. (2018) · arXiv · doi:10.48550/arxiv.1811.12808 · Cited by 735 · see citing works
External validation is necessary in prediction research:
Bleeker, S. E., Moll, H. A., Steyerberg, E. W., Donders, A. R. T., Derksen-Lubsen, G., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00207-5 · Cited by 710 · see citing works
Machine learning and its applications to biology
Tarca, A. L., Carey, V. J., Chen, X.-W., Romero, R., & Drăghici, S. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030116 · Cited by 656 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574 · see citing works
An Empirical Evaluation of Explanations for State Repression
HILL, D. W., & JONES, Z. M. (2014) · American Political Science Review · doi:10.1017/s0003055414000306 · Cited by 450 · see citing works
Disease state prediction from resting state functional connectivity
Craddock, R. C., Holtzheimer, P. E., Hu, X. P., & Mayberg, H. S. (2009) · Magnetic Resonance in Medicine · doi:10.1002/mrm.22159 · Cited by 411 · see citing works
Evaluating Prediction Rules for <i>t</i> -Year Survivors With Censored Regression Models
Uno, H., Cai, T., Tian, L., & Wei, L. J. (2007) · Journal of the American Statistical Association · doi:10.1198/016214507000000149 · Cited by 413 · see citing works
Tutorial in biostatistics: data‐driven subgroup identification and analysis in clinical trials
Lipkovich, I., Dmitrienko, A., & B., R. (2017) · Statistics in Medicine · doi:10.1002/sim.7064 · Cited by 291 · see citing works
Muchlinski, D., Siroky, D., He, J., & Kocher, M. (2016) · Political Analysis · doi:10.1093/pan/mpv024 · Cited by 234 · see citing works
Valpione, S., Pasquali, S., Campana, L. G., Piccin, L., Mocellin, S., Pigozzo, J., & Chiarion-Sileni, V. (2018) · Journal of Translational Medicine · doi:10.1186/s12967-018-1467-x · Cited by 201 · see citing works
Kim, J., Heshka, S., Gallagher, D., Kotler, D. P., Mayer, L., Albu, J., Shen, W., Freda, P. U., & Heymsfield, S. B. (2004) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00260.2004 · Cited by 205 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185 · see citing works
Thiruganasambandamoorthy, V., Kwong, K., Wells, G. A., Sivilotti, M. L. A., Mukarram, M., Rowe, B. H., Lang, E., Perry, J. J., Sheldon, R., Stiell, I. G., et al. (2016) · Canadian Medical Association Journal · doi:10.1503/cmaj.151469 · Cited by 149 · see citing works
Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma
Chuchu, N., Takwoingi, Y., Dinnes, J., Matin, R. N., Bassett, O., Moreau, J. F., Bayliss, S. E., Davenport, C., Godfrey, K., O'Connell, S., et al. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013192 · Cited by 144 · see citing works
Ansart, M., Epelbaum, S., Bassignana, G., Bône, A., Bottani, S., Cattai, T., Couronné, R., Faouzi, J., Koval, I., Louis, M., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101848 · Cited by 116 · see citing works
Spatial+: A new cross-validation method to evaluate geospatial machine learning models
Wang, Y., Khodadadzadeh, M., & Zurita-Milla, R. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103364 · Cited by 81 · see citing works
Neanderthal-Denisovan ancestors interbred with a distantly related hominin
Rogers, A. R., Harris, N. S., & Achenbach, A. A. (2020) · Science Advances · doi:10.1126/sciadv.aay5483 · Cited by 77 · see citing works
Reduction of selection bias in genomewide studies by resampling
Sun, L., & Bull, S. B. (2005) · Genetic Epidemiology · doi:10.1002/gepi.20068 · Cited by 73 · see citing works
Computerized adaptive measurement of depression: A simulation study
Gardner, W., Shear, K., Kelleher, K. J., Pajer, K. A., Mammen, O., Buysse, D., & Frank, E. (2004) · BMC Psychiatry · doi:10.1186/1471-244x-4-13 · Cited by 65 · see citing works
Machine Learning and Imaging Informatics in Oncology
Tseng, H.-H., Wei, L., Cui, S., Luo, Y., Ten Haken, R. K., & El Naqa, I. (2020) · Oncology · doi:10.1159/000493575 · Cited by 64 · see citing works
Leveraging Cheminformatics Strategies for Inorganic Discovery: Application to Redox Potential Design
Janet, J. P., Gani, T. Z. H., Steeves, A. H., Ioannidis, E. I., & Kulik, H. J. (2017) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.7b00808 · Cited by 56 · see citing works
Albéri, L., Lintas, A., Kretz, R., Schwaller, B., & Villa, A. E. P. (2013) · Journal of Neurophysiology · doi:10.1152/jn.00375.2012 · Cited by 49 · see citing works
Lessons and tips for designing a machine learning study using EHR data
Arbet, J., Brokamp, C., Meinzen-Derr, J., Trinkley, K. E., & Spratt, H. M. (2021) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.513 · Cited by 46 · see citing works
Zhao, X., Wang, Z., Zhang, J., Hua, J., He, W., & Zhu, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0053561 · Cited by 39 · see citing works
Challagulla, V., Walsh, K. B., & Subedi, P. (2014) · BioEnergy Research · doi:10.1007/s12155-013-9373-9 · Cited by 16 · see citing works
Pairing Machine Learning and Clinical Psychology: How You Evaluate Predictive Performance Matters
Jacobucci, R., Littlefield, A. K., Millner, A. J., Kleiman, E., & Steinley, D. (2020) · doi:10.31234/osf.io/2yber · Cited by 12 · see citing works
Linear and Non-Linear Regression Methods in Epidemiology and Biostatistics
Vittinghoff, E., McCulloch, C. E., Glidden, D. V., & Shiboski, S. C. (2011) · Essential Statistical Methods for Medical Statistics · doi:10.1016/b978-0-444-53737-9.50006-2 · Cited by 6 · see citing works
The Genetic Legacy of Introgression from Two Deeply Divergent Lineages in the Genus <i>Homo</i>
Rogers, A. R., Islam, M. T., Brand, C. M., & Webster, T. H. (2026) · bioRxiv · doi:10.64898/2026.03.22.713509
“This conclusion rests on the “bootstrap estimate of predictive error” (bepe), a statistical method for choosing among models [6, 7].”
- Authors
- Efron, B.
- Affiliations
- Stanford University (US)
- Corresponding author
- Efron, Bradley
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1983-06
- Volume
- 78
- Issue
- 382
- Pages
- 316-331
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2040615655
- Cited by
- 2,300
- References
- 16
- Citations by year
- 2026: 40 · 2025: 45 · 2024: 44 · 2023: 57 · 2022: 87 · 2021: 77 · 2020: 92 · 2019: 100 · 2018: 99 · 2017: 81 · 2016: 77 · 2015: 103 · 2014: 94 · 2013: 100 · 2012: 104
- FWCI
- 19.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- error rate estimation, prediction rules, cross-validation, bootstrapping, classification, small sample estimation, estimation bias, generalization error
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review