Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning
Abstract
This review presents a unified, efficient model of random decision forests which can be applied to a number of machine learning, computer vision, and medical image analysis tasks. Our model extends existing forest-based techniques as it unifies classification, regression, density estimation, manifold learning, semisupervised learning, and active learning under the same decision forest framework. This gives us the opportunity to write and optimize the core implementation only once, with application to many diverse tasks. The proposed model may be used both in a discriminative or generative way and may be applied to discrete or continuous, labeled or unlabeled data. The main contributions of this review are: (1) Proposing a unified, probabilistic and efficient model for a variety of learning tasks; (2) Demonstrating margin-maximizing properties of classification forests; (3) Discussing probabilistic regression forests in comparison with other nonlinear regression algorithms; (4) Introducing density forests for estimating probability density functions; (5) Proposing an efficient algorithm for sampling from a density forest; (6) Introducing manifold forests for nonlinear dimensionality reduction; (7) Proposing new algorithms for transductive learning and active learning. Finally, we discuss how alternatives such as random ferns and extremely randomized trees stem from our more general forest model. This document is directed at both students who wish to learn the basics of decision forests, as well as researchers interested in the new contributions. It presents both fundamental and novel concepts in a structured way, with many illustrative examples and real-world applications. Thorough comparisons with state-of-the-art algorithms such as support vector machines, boosting and Gaussian processes are presented and relative advantages and disadvantages discussed. The many synthetic examples and existing commercial applications demonstrate the validity of the proposed model and its flexibility.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
The Nature of Statistical Learning Theory
Vapnik, V. (2000) · Statistics for Engineering and Information Science · doi:10.1007/978-1-4757-3264-1 · Cited by 46,387 · see citing works
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585 · see citing works
On Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,715 · see citing works
Jolliffe, I. T. (1986) · Springer series in statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,367 · see citing works
Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
Belkin, M., & Niyogi, P. (2003) · Neural Computation · doi:10.1162/089976603321780317 · Cited by 7,562 · see citing works
The Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600 · see citing works
Chapelle, O., Schölkopf, B., & Zien, A. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262033589.001.0001 · Cited by 4,560 · see citing works
Non-Uniform Random Variate Generation
Devroye, L. (1986) · Springer eBooks · doi:10.1007/978-1-4613-8643-8 · Cited by 4,110 · see citing works
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A. W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995316 · Cited by 3,545 · see citing works
Self-citationNumerical Methods for Least Squares Problems
Björck, Å. (1996) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611971484 · Cited by 3,276 · see citing works
Active Learning with Statistical Models
Cohn, D. A., Zoubin Ghahramani, A., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 2,054 · see citing works
GTM: The Generative Topographic Mapping
Bishop, C., Svensén, M., & Williams, C. K. I. (1998) · Neural Computation · doi:10.1162/089976698300017953 · Cited by 1,389 · see citing works
Shape Quantization and Recognition with Randomized Trees
Amit, Y., & Geman, D. (1997) · Neural Computation · doi:10.1162/neco.1997.9.7.1545 · Cited by 1,277 · see citing works
Image Classification using Random Forests and Ferns
Bosch, A., Zisserman, A., & Muñoz, X. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409066 · Cited by 1,254 · see citing works
Neal, R. M. (2001) · Statistics and Computing · doi:10.1023/a:1008923215028 · Cited by 1,208 · see citing works
Semantic texton forests for image categorization and segmentation
Shotton, J., Johnson, M., & Cipolla, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587503 · Cited by 1,027 · see citing works
Self-citationDetecting pedestrians using patterns of motion and appearance
Viola, Jones, & Snow (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238422 · Cited by 1,002 · see citing works
Sharing Visual Features for Multiclass and Multiview Object Detection
Torralba, A. B., Murphy, K. P., & Freeman, W. T. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1055 · Cited by 695 · see citing works
Statnikov, A., Wang, L., & Aliferis, C. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-319 · Cited by 684 · see citing works
Keypoint recognition using randomized trees
Lepetit, V., & Fua, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.188 · Cited by 681 · see citing works
An empirical evaluation of supervised learning in high dimensions
Caruana, R., Karampatziakis, N., & Yessenalina, A. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390169 · Cited by 531 · see citing works
Saffari, A., Leistner, C., Santner, J., Godec, M., & Bischof, H. (2009) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2009.5457447 · Cited by 502 · see citing works
Real time head pose estimation with random regression forests
Fanelli, G., Gall, J., & Gool, L. V. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995458 · Cited by 441 · see citing works
Efficient regression of general-activity human poses from depth images
Girshick, R., Shotton, J., Kohli, P., Criminisi, A., & Fitzgibbon, A. W. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126270 · Cited by 347 · see citing works
Self-citationMarkov Random Fields for Vision and Image Processing
(2011) · The MIT Press eBooks · doi:10.7551/mitpress/8579.001.0001 · Cited by 296 · see citing works
GeoS: Geodesic Image Segmentation
Criminisi, A., Sharp, T., & Blake, A. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_9 · Cited by 277 · see citing works
Self-citationMulticlass multiple kernel learning
Zien, A., & Ong, C. S. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273646 · Cited by 269 · see citing works
A reduction formula for normal multivariate integrals
Plackett, R. L. (1954) · Biometrika · doi:10.2307/2332716 · Cited by 262 · see citing works
A REDUCTION FORMULA FOR NORMAL MULTIVARIATE INTEGRALS
Plackett, R. L. (1954) · Biometrika · doi:10.1093/biomet/41.3-4.351 · Cited by 260 · see citing works
Random Subwindows for Robust Image Classification
Marée, R., Geurts, P., Piater, J., & Wehenkel, L. A. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.287 · Cited by 232 · see citing works
Menze, B. H., Kelm, B. M., Splitthoff, D. N., Köthe, U., & Hamprecht, F. A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23783-6_29 · Cited by 226 · see citing works
Randomized trees for human pose detection
Rogez, G., Rihan, J., Ramalingam, S., et al. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587617 · Cited by 177 · see citing works
Implementing Decision Trees and Forests on a GPU
Sharp, T. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88693-8_44 · Cited by 176 · see citing works
Accurate Visual Metrology from Single and Multiple Uncalibrated Images
Criminisi, A. (2001) · Springer eBooks · doi:10.1007/978-0-85729-327-5 · Cited by 162 · see citing works
Self-citationSemi-Supervised Random Forests
Leistner, C., Saffari, A., Santner, J., & Bischof, H. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459198 · Cited by 136 · see citing works
Entangled Decision Forests and Their Application for Semantic Segmentation of CT Images
Montillo, A., Shotton, J., Winn, J. M., Iglesias, J. E., Metaxas, D., & Criminisi, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22092-0_16 · Cited by 126 · see citing works
Self-citationLempitsky, V., Verhoek, M., Noble, J. A., & Blake, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01932-6_48 · Cited by 124 · see citing works
Tree-based Classifiers for Bilayer Video Segmentation
Yin, P., Criminisi, A., Winn, J. M., & Essa, I. A. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383008 · Cited by 124 · see citing works
Self-citationKernel Methods in Computer Vision
Lampert, C. H. (2009) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000027 · Cited by 111 · see citing works
Meinshausen, N. (2010) · The Annals of Applied Statistics · doi:10.1214/10-aoas367 · Cited by 107 · see citing works
GRAM: A framework for geodesic registration on anatomical manifolds
Hamm, J., Ye, D. H., Verma, R., & Davatzikos, C. A. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.06.001 · Cited by 106 · see citing works
Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., & Kohli, P. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126429 · Cited by 102 · see citing works
The Joint Manifold Model for Semi-supervised Multi-valued Regression
Navaratnam, R., Fitzgibbon, A. W., & Cipolla, R. (2007) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408976 · Cited by 101 · see citing works
Age regression from faces using random forests
Montillo, A., & Ling, H. (2009) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2009.5414103 · Cited by 79 · see citing works
Detection, Grading and Classification of Coronary Stenoses in Computed Tomography Angiography
Kelm, B. M., Mittal, S., Zheng, Y., Tsymbal, A., Bernhardt, D., Vega-Higuera, F., Zhou, S. K., Meer, P., & Comaniciu, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6_4 · Cited by 75 · see citing works
On the Manifold Structure of the Space of Brain Images
Gerber, S., Taşdizen, T., Joshi, S. C., & Whitaker, R. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04268-3_38 · Cited by 44 · see citing works
Semi-supervised video segmentation using tree structured graphical models
Budvytis, I., Badrinarayanan, V., & Cipolla, R. (2011) · doi:10.1109/cvpr.2011.5995600 · Cited by 38 · see citing works
The Relevance Voxel Machine (RVoxM): A Bayesian Method for Image-Based Prediction
Sabuncu, M. R., & Leemput, K. V. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6_13 · Cited by 24 · see citing works
Laplacian Eigenmaps Manifold Learning for Landmark Localization in Brain MR Images
Guerrero, R., Wolz, R., & Rueckert, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23629-7_69 · Cited by 16 · see citing works
Robust linear registration of CT images using random regression forests
Konukoğlu, E., Criminisi, A., Pathak, D. S., et al. (2011) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.878085 · Cited by 14 · see citing works
Self-citationCharacterizing Pathological Deviations from Normality Using Constrained Manifold-Learning
Duchateau, N., Craene, M. D., Piella, G., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6_32 · Cited by 8 · see citing works
STARS: A new ensemble partitioning approach
Pauly, O., Mateus, D., & Navab, N. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130407 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Biau, G., & Scornet, E. (2016) · Test · doi:10.1007/s11749-016-0481-7 · Cited by 4,149 · see citing works
Deep convolutional neural networks for multi-modality isointense infant brain image segmentation
Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., & Shen, D. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.061 · Cited by 817 · see citing works
AI in Medical Imaging Informatics: Current Challenges and Future Directions
Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., Foran, D., Do, N., Golemati, S., Kurc, T., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2991043 · Cited by 713 · see citing works
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · see citing works
Machine learning algorithms for social media analysis: A survey
T.K., B., Annavarapu, C. S. R., & Bablani, A. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100395 · Cited by 276 · see citing works
Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models
Jain, A., Koppula, H. S., Raghavan, B., Soh, S., & Saxena, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.364 · Cited by 274 · see citing works
Clinical prediction from structural brain MRI scans: A large-scale empirical study
Initiative, F. T. A. D. N., Sabuncu, M. R., & Konukoglu, E. (2015) · Neuroinformatics · doi:10.1007/s12021-014-9238-1 · Cited by 161 · see citing works
Self-citationReverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2665165 · Cited by 152 · see citing works
Folding clothes autonomously: A complete pipeline
Doumanoglou, A., Stria, J., Peleka, G., Mariolis, I., Petrik, V., Kargakos, A., Wagner, L., Hlavac, V., Kim, T.-K., & Malassiotis, S. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2602376 · Cited by 122 · see citing works
- Authors
- Criminisi, A., Shotton, J., & Konukoglu, E.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Microsoft Research Ltd. , 7 J J Thomson Ave, Cambridge, CB3 0FB,
- Corresponding author
- Criminisi, Antonio
- ORCID
- Criminisi, Antonio
- Venue
- Foundations and Trends® in Computer Graphics and Vision
- Publisher
- Emerald
- Published
- 2012-03-29
- Volume
- 7
- Issue
- 2-3
- Pages
- 81-227
- ISSN
- 1572-2740 (ISSN-L)
- 1572-2759
- OpenAlex
- W4293171766
- Cited by
- 677
- References
- 58
- Citations by year
- 2026: 18 · 2025: 29 · 2024: 30 · 2023: 52 · 2022: 37 · 2021: 55 · 2020: 51 · 2019: 64 · 2018: 50 · 2017: 62 · 2016: 66 · 2015: 66 · 2014: 45 · 2013: 37 · 2012: 15
- FWCI
- 51.33
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Foundations and Trends® in Computer Graphics and Vision (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- decision forest, random forest, classification, regression analysis, density estimation, manifold learning, nonlinear dimensionality reduction, semi-supervised learning, active learning (machine learning), transductive learning, margin maximization, random ferns, extra trees, probabilistic regression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
