The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
On the overestimation of random forest’s out-of-bag error
Open accessAbstract
The ensemble method random forests has become a popular classification tool in bioinformatics and related fields. The out-of-bag error is an error estimation technique often used to evaluate the accuracy of a random forest and to select appropriate values for tuning parameters, such as the number of candidate predictors that are randomly drawn for a split, referred to as mtry. However, for binary classification problems with metric predictors it has been shown that the out-of-bag error can overestimate the true prediction error depending on the choices of random forests parameters. Based on simulated and real data this paper aims to identify settings for which this overestimation is likely. It is, moreover, questionable whether the out-of-bag error can be used in classification tasks for selecting tuning parameters like mtry, because the overestimation is seen to depend on the parameter mtry. The simulation-based and real-data based studies with metric predictor variables performed in this paper show that the overestimation is largest in balanced settings and in settings with few observations, a large number of predictor variables, small correlations between predictors and weak effects. There was hardly any impact of the overestimation on tuning parameter selection. However, although the prediction performance of random forests was not substantially affected when using the out-of-bag error for tuning parameter selection in the present studies, one cannot be sure that this applies to all future data. For settings with metric predictor variables it is therefore strongly recommended to use stratified subsampling with sampling fractions that are proportional to the class sizes for both tuning parameter selection and error estimation in random forests. This yielded less biased estimates of the true prediction error. In unbalanced settings, in which there is a strong interest in predicting observations from the smaller classes well, sampling the same number of observations from each class is a promising alternative.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., et al. (1999) · Science · doi:10.1126/science.286.5439.531 · Cited by 11,516
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Gene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536
Alon, U., Barkai, N., Notterman, D. A., Gish, K., Ybarra, S., Mack, D., & Levine, A. J. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.12.6745 · Cited by 4,167
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
Bias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,688
Gene selection and classification of microarray data using random forest
Díaz-Uriarte, R., & Alvarez de Andrés, S. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-3 · Cited by 2,929
2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,800
Gene expression correlates of clinical prostate cancer behavior
Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo, P., Renshaw, A. A., D'Amico, A. V., Richie, J. P., et al. (2002) · Cancer Cell · doi:10.1016/s1535-6108(02)00030-2 · Cited by 2,512
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485
Prediction of central nervous system embryonal tumour outcome based on gene expression
Pomeroy, S. L., Tamayo, P., Gaasenbeek, M., Sturla, L. M., Angelo, M., McLaughlin, M. E., Kim, J. Y. H., Goumnerova, L. C., Black, P. M., Lau, C., et al. (2002) · Nature · doi:10.1038/415436a · Cited by 2,371
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339
Boulesteix, A., Janitza, S., Kruppa, J., & König, I. R. (2012) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1072 · Cited by 924
Self-citationOliveira, S., Oehler, F., San-Miguel-Ayanz, J., Camia, A., & Pereira, J. M. C. (2012) · Forest Ecology and Management · doi:10.1016/j.foreco.2012.03.003 · Cited by 747
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Probst, P., Bischl, B., & Boulesteix, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09596 · Cited by 565
BagBoosting for tumor classification with gene expression data
Dettling, M. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth447 · Cited by 542
Ensemble machine learning on gene expression data for cancer classification.
Tan, A. C., & Gilbert, D. (2003) · PubMed · Cited by 404
Classification Trees With Unbiased Multiway Splits
Kim, H., & Loh, W.-Y. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753168271 · Cited by 391
Breiman, L. (1996) · Cited by 322
Boosting for tumor classification with gene expression data
Dettling, M., & Bühlmann, P. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btf867 · Cited by 318
Random Forests for Genetic Association Studies
Goldstein, B. A., Polley, E. C., & Briggs, F. B. S. (2011) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1691 · Cited by 272
Random Forests: some methodological insightsPreprint
Genuer, R., Poggi, J., & Tuleau-Malot, C. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0811.3619 · Cited by 236
Goldstein, B. A., Hubbard, A. E., Cutler, A., & Barcellos, L. F. (2010) · BMC Genetics · doi:10.1186/1471-2156-11-49 · Cited by 225
Nicodemus, K. K. (2011) · Briefings in Bioinformatics · doi:10.1093/bib/bbr016 · Cited by 204
Bias of the Random Forest Out-of-Bag (OOB) Error for Certain Input Parameters
Mitchell, M. W. (2011) · Open Journal of Statistics · doi:10.4236/ojs.2011.13024 · Cited by 186
Hassane, D. C., Guzman, M. L., Corbett, C., Li, X., Abboud, R., Young, F., Liesveld, J. L., Carroll, M., & Jordan, C. T. (2008) · Blood · doi:10.1182/blood-2007-11-126003 · Cited by 185
Estimating Generalization Error on Two-Class Datasets Using Out-of-Bag Estimates
Bylander, T. (2002) · Machine Learning · doi:10.1023/a:1013964023376 · Cited by 184
Predictor correlation impacts machine learning algorithms: implications for genomic studies
Nicodemus, K. K., & Malley, J. D. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp331 · Cited by 170
Boulesteix, A., Bender, A., Bermejo, J. L., & Strobl, C. (2011) · Briefings in Bioinformatics · doi:10.1093/bib/bbr053 · Cited by 148
Nicodemus, K. K., Callicott, J. H., Higier, R. G., Luna, A., Nixon, D. C., Lipska, B. K., Vakkalanka, R., Giegling, I., Rujescu, D., Clair, D. S., et al. (2010) · Human Genetics · doi:10.1007/s00439-009-0782-y · Cited by 127
Tunability: Importance of Hyperparameters of Machine Learning AlgorithmsPreprint
Probst, P., Boulesteix, A., & Bischl, B. (2019) · arXiv (Cornell University) · Cited by 123
Characterizing Forest Change Using Community-Based Monitoring Data and Landsat Time Series
DeVries, B., Pratihast, A. K., Verbesselt, J., Kooistra, L., & Herold, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147121 · Cited by 120
Ten Simple Rules for Reducing Overoptimistic Reporting in Methodological Computational Research
Boulesteix, A. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004191 · Cited by 78
Building Lightweight Intrusion Detection System Based on Random Forest
Kim, D. S., Lee, S. M., & Park, J. S. (2006) · Lecture notes in computer science · doi:10.1007/11760191_33 · Cited by 29
Hornung, R., Bernau, C., Truntzer, C., Wilson, R., Stadler, T., & Boulesteix, A. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0088-9 · Cited by 26
Self-citationWhen is a genomic classifier ready for prime time?
Simon, R. M. (2004) · Nature Clinical Practice Oncology · doi:10.1038/ncponc0006 · Cited by 24
Bernau, C., Augustin, T., & Boulesteix, A. (2013) · Biometrics · doi:10.1111/biom.12041 · Cited by 21
Combined genomic expressions as a diagnostic factor for oral squamous cell carcinoma
Kim, K., Zhang, X., & Cha, I. (2013) · Genomics · doi:10.1016/j.ygeno.2013.11.007 · Cited by 16
Cited by
Hyperparameters and tuning strategies for random forest
Probst, P., Wright, M. N., & Boulesteix, A. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1301 · Cited by 1,640
A Survey of Ensemble Learning: Concepts, Algorithms, Applications, and Prospects
Mienye, I. D., & Sun, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3207287 · Cited by 1,298
Metadata
- Authors
- Janitza, S., & Hornung, R.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-08-06
- Volume
- 13
- Issue
- 8
- Pages
- e0201904
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2608146202
- Cited by
- 278
- References
- 45
- FWCI
- 11.17
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.