Consistency of random forests
Open accessAbstract
Random forests are a learning algorithm proposed by Breiman [ Mach. Learn. 45 (2001) 5–32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data-dependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman’s [ Mach. Learn. 45 (2001) 5–32] original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Scornet et al. (2015) show that Breiman's (2001) forests are consistent in an additive regression framework.
“Scornet et al. (2015), who show that Breiman's (2001) forests are consistent in an additive regression framework.”
From A random forest guided tour (2016) · cites this paper as Scornet et al. (2015)
Scornet et al. (2015) show that random forests asymptotically perform, with high probability, splits along the S informative variables.
“It is shown in Scornet et al. (2015) that random forests asymptotically perform, with high probability, splits along the S informative variables (in the sense of Section 3.1).”
From A random forest guided tour (2016) · cites this paper as Scornet et al. (2015)
Assuming regularity conditions and considering a modification of Breiman's forests in which all directions are preselected for splitting, Scornet et al. (2015) prove that, with probability 1-ξ, for all n large enough and all 1 ≤ q ≤ k, the first k cut directions are among the S informative variables.
“Assuming some regularity conditions on the regression model, and considering a modification of Breiman's forests in which all directions are preselected for splitting, Scornet et al. (2015) prove that, with probability 1 -ξ , for all n large enough and all 1 ≤ q ≤ k ,”
From A random forest guided tour (2016) · cites this paper as Scornet et al. (2015)
Scornet et al. (2015) proved a consistency result in the context of additive regression models for the pruned version of Breiman's forest.
“Scornet et al. (2015) proved a consistency result in the context of additive regression models for the pruned version of Breiman's forest.”
From A random forest guided tour (2016) · cites this paper as Scornet et al. (2015)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Hastie, T., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013604 · Cited by 12,575 · see citing works
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300 · see citing works
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
Random forests for classification in ecology
Cutler, D. R., Edwards, T. C., Beard, K. H., Cutler, A., Hess, K. T., Gibson, J., & Lawler, J. J. (2007) · Ecology · doi:10.1890/07-0539.1 · Cited by 4,950 · see citing works
Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling
Svetnik, V., Liaw, A. I., Tong, C., Culberson, J. C., Sheridan, R. P., & Feuston, B. P. (2003) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci034160g · Cited by 3,613 · see citing works
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A. W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995316 · Cited by 3,545 · see citing works
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · see citing works
Gene selection and classification of microarray data using random forest
Díaz‐Uriarte, R., & de Andrés, S. Á. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-3 · Cited by 2,924 · see citing works
The Jackknife: The Bootstrap and Other Resampling Plans.
Low, L. Y., & Efron, B. (1983) · Journal of the American Statistical Association · doi:10.2307/2288218 · Cited by 2,728 · see citing works
Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas169 · Cited by 2,714 · see citing works
Prasad, A., Iverson, L. R., & Liaw, A. I. (2006) · Ecosystems · doi:10.1007/s10021-005-0054-1 · Cited by 2,297 · see citing works
Real-time human pose recognition in parts from single depth images
Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A. W., Finocchio, M., Blake, A., Cook, M., & Moore, R. (2013) · Communications of the ACM · doi:10.1145/2398356.2398381 · Cited by 2,050 · see citing works
Consistent Nonparametric Regression
Stone, C. (1977) · The Annals of Statistics · doi:10.1214/aos/1176343886 · Cited by 1,971 · see citing works
A Distribution-Free Theory of Nonparametric Regression
Györfi, L., Köhler, M., Krzyżak, A., & Walk, H. (2002) · Springer series in statistics · doi:10.1007/b97848 · Cited by 1,738 · see citing works
Meinshausen, N. (2006) · Cited by 1,522 · see citing works
Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008) · Cited by 1,310 · see citing works
Additive Regression and Other Nonparametric Models
Stone, C. J. (1985) · The Annals of Statistics · doi:10.1214/aos/1176349548 · Cited by 1,265 · see citing works
Adaptive estimation of a quadratic functional by model selection
Laurent, B., & Massart, P. (2000) · The Annals of Statistics · doi:10.1214/aos/1015957395 · Cited by 1,080 · see citing works
Analysis of a Random Forests ModelPreprint
Gérard Biau, A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1005.0208 · Cited by 672 · see citing works
Nordhausen, K. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_18.x · Cited by 646 · see citing works
Bühlmann, P., & Yu, B. (2002) · The Annals of Statistics · doi:10.1214/aos/1031689014 · Cited by 609 · see citing works
Random Forests and Adaptive Nearest Neighbors
Lin, Y., & Jeon, Y. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001230 · Cited by 507 · see citing works
Consistency of Random Forests and Other Averaging Classifiers
Gérard Biau, A., Devroye, L., & Lugosi, G. (2007) · Cited by 457 · see citing works
Real-Time Human Pose Recognition in Parts from Single Depth Images
Shotton, J., Fitzgibbon, A. W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2012) · Studies in computational intelligence · doi:10.1007/978-3-642-28661-2_5 · Cited by 434 · see citing works
A Scalable Bootstrap for Massive Data
Kleiner, A., Talwalkar, A., Sarkar, P., & Jordan, M. I. (2014) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12050 · Cited by 391 · see citing works
Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife
Wager, S., Hastie, T. J., & Efron, B. (2013) · PubMed · doi:10.48550/arxiv.1311.4555 · Cited by 372 · see citing works
Consistency of random survival forests
Ishwaran, H., & Kogalur, U. B. (2010) · Statistics & Probability Letters · doi:10.1016/j.spl.2010.02.020 · Cited by 269 · see citing works
Amaratunga, D., Cabrera, J., & Lee, Y.-S. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn356 · Cited by 229 · see citing works
Zhu, R., Zeng, D., & Kosorok, M. R. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1036994 · Cited by 167 · see citing works
CONSISTENCY FOR A SIMPLE MODEL OF RANDOM FORESTS
Breiman, L. (2004) · Cited by 165 · see citing works
Gérard Biau, A., & Devroye, L. (2010) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2010.06.019 · Cited by 115 · see citing works
Histogram regression estimation using data-dependent partitions
Nobel, A. B. (1996) · The Annals of Statistics · doi:10.1214/aos/1032526958 · Cited by 108 · see citing works
On the Distribution of the Number of Admissible Points in a Vector Random Sample
Barndorff-Nielsen, O. E., & Sobel, M. (1966) · Theory of Probability and Its Applications · doi:10.1137/1111020 · Cited by 98 · see citing works
Variance reduction in purely random forests
Genuer, R. (2012) · Journal of nonparametric statistics · doi:10.1080/10485252.2012.677843 · Cited by 89 · see citing works
Consistency of Online Random ForestsPreprint
Denil, M., Matheson, D., & de Freitas, N. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4853 · Cited by 59 · see citing works
Asymptotic Theory for Random ForestsPreprint
Wager, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.0352 · Cited by 47 · see citing works
Bai, Z., Devroye, L., Hwang, H., et al. (2005) · Random Structures and Algorithms · doi:10.1002/rsa.20053 · Cited by 45 · see citing works
Clémençon, S., Depecker, M., & Vayatis, N. (2013) · Cited by 30 · see citing works
Ensemble Trees and CLTs: Statistical Inference for Supervised LearningPreprint
Mentch, L., & Hooker, G. (2014) · arXiv (Cornell University) · Cited by 20 · see citing works
Gene Selection and Classification
(2008) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789812790743_0007 · Cited by 7 · see citing works
On the asymptotics of random forestsPreprint
Scornet, E. (2014) · doi:10.48550/arxiv.1409.2090 · Cited by 7 · see citing works
Self-citationStandard Errors for Bagged Predictors and Random ForestsPreprint
Wager, S., Hastie, T., & Efron, B. (2013) · arXiv (Cornell University) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- BOUCHERON, S., LUGOSI, G. and MASSART, P. (2013). Concentration Inequalities: A Nonasymp- totic Theory of Independence. Oxford Univ. Press, Oxford.
- POLITIS, D. N., ROMANO, J. P. and WOLF, M. (1999). Subsampling. Springer, New York.
- SCORNET, E., BIAU, G. and VERT, J. (2015). Supplement to "Consistency of random forests." DOI:10.1214/15-AOS1321SUPP. · DOI
Cited by
Sorted by citations
Biau, G., & Scornet, E. (2016) · Test · doi:10.1007/s11749-016-0481-7 · Cited by 4,149 · see citing works
Self-citationEstimation and Inference of Heterogeneous Treatment Effects using Random Forests
Wager, S., & Athey, S. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1319839 · Cited by 2,917 · see citing works
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence
Watson, D. (2019) · Minds and Machines · doi:10.1007/s11023-019-09506-6 · Cited by 148 · see citing works
Empirical Asset Pricing via Machine Learning
Gu, S., Kelly, B. T., & Xiu, D. (2018) · The Review of Financial Studies · doi:10.2139/ssrn.3159577 · Cited by 52 · see citing works
- Authors
- Scornet, E., Biau, G., & Vert, J.-P.
- Affiliations
- Sorbonne Université (FR)
- Université Paris Sciences et Lettres (FR)
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2015-08-01
- Volume
- 43
- Issue
- 4
- Pages
- 1716-1741
- ISSN
- 0090-5364 (ISSN-L)
- 2168-8966
- OpenAlex
- W2162387923
- Cited by
- 612
- References
- 47
- Citations by year
- 2026: 59 · 2025: 76 · 2024: 72 · 2023: 70 · 2022: 67 · 2021: 67 · 2020: 64 · 2019: 49 · 2018: 31 · 2017: 20 · 2016: 19 · 2015: 9 · 2014: 6 · 2013: 3
- FWCI
- 23.44
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Annals of Statistics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- random forest, consistency, additive regression, randomized decision trees, tree structure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
