An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests.

In the reaction time experiment, the effect of sleep deprivation on reaction time differs between subjects.

“it is illustrated in Figure 9 that the effect of sleep deprivation on reaction time differs between subjects.”

In the artificial smoking example, bagging and even a single tree would perform equally well because the interaction of friends_smoke and alcohol_per_month is the only effect induced in the data.

“In our artificial example, bagging, and even a single tree, would actually perform equally well, because the interaction of friends_smoke and alcohol_per_month, that was already correctly identified by the single tree, is the only effect that was induced in the data.”

A logistic regression model would not be applicable in a sample of this size, even for the reduced data set with only 102 genes, because the estimation algorithm does not converge.

“a logistic regression model would not be applicable in a sample of this size, even for the reduced data set with only 102 genes – not even if forward selection was employed and only main effects were considered, disregarding the possibility of interaction effects – because the estimation algorithm does not converge.”

For the smoking data example, the standard and out-of-bag prediction accuracy of a random forest with ntree=500 and mtry=2 are 74.5% and 71.5%, respectively.

“The standard and out-of-bag prediction accuracy of a random forests with ntree=500 and mtry=2 for our smoking data example is 74.5% and 71.5% respectively”

In the simple smoking example, the two relevant predictor variables friends_smoke and alcohol_per_month are correctly identified by the permutation variable importance of both bagging and random forests.

“in our simple example the two relevant predictor variables friends_smoke and alcohol_per_month are correctly identified by the permutation variable importance of both bagging and random forests”

In the gene data example, the effects of the two artificially modified genes can be clearly identified by permutation importances.

“The effects of the two artificially modified genes can be clearly identified.”

Even in studies with much lower numbers of predictor variables, the combination of all main and interaction effects of interest may lead to cell counts too sparse for parameter convergence.

“However, even in studies with much lower numbers of predictor variables, the combination of all main and interaction effects of interest – especially in the case of categorical predictor variables – may well lead to cell counts too sparse for parameter convergence.”

Random forest variable importance ranking is more stable than stepwise variable selection approaches for logistic regression, which are affected by order effects.

“For detecting relevant predictor variables, Rossi et al. (2005) point out that the random forest variable importance ranking proves to be more stable than stepwise variable selection approaches available for logistic regression, that are known to be affected by order effects”

• cites Rossi et al. (2005)Not checked yet

  Not checked in the free run.

• cites Freedman 1983Not checked yet

  Not checked in the free run.

• cites Derksen and Keselman 1992Not checked yet

  Not checked in the free run.

In the small n large p case, parametric approaches are no longer applicable.

“One situation where parametric approaches are no longer applicable is the so called ”small n large p” case, where the number of predictor variables p is greater than the number of subjects n.”

Classical statistical methods can be subject to severe limitations in certain situations.

“However, in certain situations these classical methods can be subject to severe limitations.”

High-order interaction effects usually cannot be included in standard parametric models.

“Thus, interaction effects of high order usually cannot be included in standard parametric models.”

Dimension reduction followed by standard regression has the disadvantage that original input variables are projected into a reduced set of components, so their individual effect is no longer identifiable.

“However, this approach has the disadvantage that the original input variables are projected into a reduced set of components, so that their individual effect is not longer identifiable.”

A high random forest variable importance of a variable not included in stepwise regression may indicate that the variable works in interactions too complex to be captured by parametric regression models.

“Moreover, a high random forest variable importance of a variable that was not included in stepwise regression may indicate that the variable works in interactions that are too complex to be captured by parametric regression models.”

In stepwise linear regression, extensions including interaction effects are typically limited to two-fold interactions because the number of higher-order interactions is too large.

“However, in stepwise linear regression the predictors still have a linear effect on the dependent variable, while extensions of stepwise procedures including interaction effects are typically limited to the inclusion of two-fold interactions, since the number of higher order interactions – that would have to be created simultaneously when starting the selection procedure – is too large.”

Recursive partitioning methods overcome the limitations of classical parametric methods.

“The aim of this paper is to provide an instructive review of a set of statistical methods adopted from machine learning, that overcome these limitations.”

Classification trees can represent nonlinear and nonmonotone association rules that do not need to be specified in advance but are determined in a data-driven way.

“In particular, this includes nonlinear and even nonmonotone association rules, that do not need to be specified in advance but are determined in a data driven way.”

Classification trees and ensemble methods can be used to estimate propensity scores when treatment assignment is chosen as a working response.

“When the treatment assignment is chosen as a working response, classification trees and ensemble methods can be used to estimate propensity scores.”

• cites Luellen et al. (2005)Not checked yet

  Not checked in the free run.

Ensemble methods have smoother decision boundaries than single trees.

“As described in the next section, ensemble methods have the additional advantage, that their decision boundaries are more smooth than those of single trees.”

The statement in Berk (2006) that a branch not split further indicates a main effect is not correct when splitting continues in the other branch created by the same variable.

“However, when in the other branch created by the same variable splitting continues, as is the case in the example of Berk (2006), this statement is not correct.”

• cites Berk (2006)Not checked yet

  Not checked in the free run.

Due to instability, predictions of single trees show high variability.

“Due to this instability, the predictions of single trees show a high variability.”

Bagging has superiority in prediction accuracy over single classification or regression trees.

“the theoretical results of Bühlmann and Yu (2002), that show the superiority in prediction accuracy of bagging over single classification or regression trees”

• cites Bühlmann and Yu (2002)Not checked yet

  Not checked in the free run.

The main flaw of simple tree models is their instability to small changes in the learning data.

“The main flaw of simple tree models is their instability to small changes in the learning data”

A regression tree creates a piecewise constant prediction function.

“Note that this means that a regression tree creates a piecewise (or rectangle-wise for two dimensions and cuboid-wise in higher dimensions) constant prediction function.”

Overfitting is equally relevant in parametric models: adding variables or parameters improves fit to learning data because the model becomes more flexible.

“However, it should be noted that overfitting is an equally relevant issue in parametric models: With every variable, and thus every parameter, that is added to the regression model, its fit to the learning data improves, because the model becomes more flexible.”

An overfitted model performs poorly on a new test sample because it does not generalize well.

“When such an overfitted model is later applied to a new test sample from the same population, its performance will be poor because it does not generalize well.”

Model-based recursive partitioning offers a way to partition the feature space to detect parameter instabilities in a parametric model via a structural change test framework.

“The model based recursive partitioning approach of Zeileis, Hothorn, and Hornik (2008) offers a way to partition the feature space in order to detect parameter instabilities in the parametric model of interest by means of a structural change test framework.”

• cites Zeileis, Hothorn, and Hornik (2008)Not checked yet

  Not checked in the free run.

Model-based partitioning can offer a heuristic but easy-to-interpret alternative to latent class, random effects, and mixed effects models.

“Thus, model based partitioning can offer a heuristic but easy to interpret alternative to latent class – as well as random or mixed effects – models.”

Information criteria such as AIC and BIC are not applicable to nonparametric models.

“Since information criteria such as the AIC and BIC are, however, not applicable to nonparametric models (see, e.g., Claeskens and Hjort 2008)”

• cites Claeskens and Hjort 2008Not checked yet

  Not checked in the free run.

In recursive partitioning, only those interactions that are actually used in the tree are generated during the fitting process.

“In contrast to this, in recursive partitioning only those interactions that are actually used in the tree are generated during the fitting process.”

Following the principle of impurity reduction, each split in tree building results in daughter nodes that are more pure than the parent node.

“From this example we can see that, following the principle of impurity reduction, each split in the tree building process results in daughter nodes that are more “pure” than the parent node”

If classification trees built on different bootstrap samples vary too strongly in structure, this proves that their interpretability can be severely affected by random variability.

“Thus, if classification trees built on different bootstrap samples vary too strongly in their structure, this proves that their interpretability can be severely affected by the random variability present in any data set.”

In bagging and random forests, trees are usually grown very large without stopping or pruning.

“usually trees are grown very large, without any stopping or pruning involved.”

Ensemble methods solve the instability of single trees by averaging over an ensemble of trees for prediction.

“As a solution to the problem of instability, the average over an ensembles of trees, rather than a single tree, is used for prediction in ensemble methods, as outlined in the following.”

Large trees can become more diverse and include a large variety of combinations of predictor variables.

“large trees can become even more diverse and include a large variety of combinations of predictor variables.”

Because individual classification trees depend highly on the learning sample, trees grown on different random samples can differ substantially.

“Keeping in mind that each individual classification tree depends highly on the learning sample as outlined above, the resulting trees can thus differ substantially.”

The visual representation of classification trees may be misguiding because the actual statistical interpretation of a tree model is not trivial.

“However, the downside of this apparently straightforward interpretability is that the visual representation may be misguiding, because the actual statistical interpretation of a tree model is not trivial.”

Smoothing hard decision boundaries makes ensembles more flexible than single trees in approximating smooth rather than piecewise constant functional forms.

“The smoothing of hard decision boundaries also makes ensembles more flexible than single trees in approximating functional forms that are smooth rather than piecewise constant.”

In random forests, randomly restricting the set of predictor variables to select from in each split introduces another source of diversity, producing more diverse trees.

“In random forests another source of diversity is introduced when the set of predictor variables to select from is randomly restricted in each split, producing even more diverse trees.”

Searching for a single globally best tree is computationally infeasible.

“searching for a single globally best tree is computationally infeasible”

• cites van Os and Meulman 2005Not checked yet

  Not checked in the free run.

Split selection in regular classification trees is only locally optimal in each node: a variable and cutpoint are chosen for the impurity reduction they achieve in a given node, regardless of future splits.

“the split selection process in regular classification trees is only locally optimal in each node: A variable and cutpoint are chosen with respect to the impurity reduction they can achieve in a given node defined by all previous splits, but regardless of all splits yet to come.”

Random forests can improve predictive performance even further compared to bagging because the single trees involved in averaging are even more diverse.

“random forests can improve the predictive performance even further as compared to bagging, because the single trees involved in averaging are even more diverse.”

Random restriction of the splitting variables provides an easy and efficient way to generate locally suboptimal splits that can improve the global performance of an ensemble of trees.

“the random restriction of the splitting variables provides an easy and efficient way to generate locally suboptimal splits that can improve the global performance of an ensemble of trees.”

The number of randomly preselected splitting variables (mtry) and the overall number of trees (ntree) are parameters of random forests that affect the stability of the results.

“The number of randomly preselected splitting variables, termed mtry in most algorithms, as well as the overall number of trees, usually termed ntree, are parameters of random forests that affect the stability of the results”

The upper bound for the generalization error of an ensemble depends on the correlation between the individual trees, such that a low correlation between individual trees results in a low upper bound for the error.

“the upper bound for the generalization error of an ensemble depends on the correlation between the individual trees, such that a low correlation between the individual trees results in a low upper bound for the error.”

• cites Breiman (2001a)Not checked yet

  Not checked in the free run.

In both recursive partitioning and stepwise regression, adding one locally optimal variable at a time does not necessarily lead to the globally best model over all possible combinations of variables.

“In both recursive partitioning and stepwise regression, the approach of adding one locally optimal variable at a time does not necessarily (or rather hardly ever) lead to the globally best model over all possible combinations of variables.”

Even with the simple aggregation schemes used in standard algorithms, ensemble methods reliably outperform single trees and many other advanced methods.

“even with the simple aggregation schemes used in the standard algorithms, ensembles methods reliably outperform single trees and many other advanced methods”

Out-of-bag observations can serve as a built-in test sample for computing the prediction accuracy of a tree.

“they can serve as a “built-in” test sample for computing the prediction accuracy of that tree.”

The TWIX approach is limited to a sequence of locally optimal splits.

“the TWIX approach is limited to a sequence of locally optimal splits.”

The TWIX approach starts in a single starting node but branches to a set of trees at each decision by splitting not only in the best cutpoint but also in reasonable extra cutpoints.

“the building of the tree ensemble starts in a single starting node but branches to a set of trees at each decision by means of splitting not only in the best cutpoint but also in reasonable extra cutpoints.”

• cites Potapov 2007Not checked yet

  Not checked in the free run.

• cites Potapov, Theus, and Urbanek 2006Not checked yet

  Not checked in the free run.

The out-of-bag error is a more realistic estimate of the error rate expected in a new test sample than the naive and over-optimistic estimate from the entire learning sample.

“The advantage of the out-of-bag error is that it is a more realistic estimate of the error rate that is to be expected in a new test sample, than the naive and over-optimistic estimate of the error rate resulting from the prediction of the entire learning sample”

• cites Breiman 1996bNot checked yet

  Not checked in the free run.

• cites Boulesteix, Strobl, Augustin, and Daumer (2008)Not checked yet

  Not checked in the free run.

For linear functions, a single tree's step-function approximation will be rather poor.

“For linear functions, the problem from a practical point of view is that a single tree’s step-function approximation will be rather poor.”

In bagging and random forests, variable importance measures are computed to assess the relevance of each variable over all trees of the ensemble.

“in bagging and random forests variable importance measures are computed to assess the relevance of each variable over all trees of the ensemble.”

Compared to other nonlinear regression approaches such as smoothing splines, ensemble methods do not require prespecifying the shape of the function or the position or number of knots.

“as compared to other nonlinear regression approaches such as smoothing splines, neither the shape of the function nor the position or number of knots needs to be prespecified”

• cites Wood 2006Not checked yet

  Not checked in the free run.

Bagging and random forests can be used to approximate any unknown function, even if it is nonlinear and involves complex interactions.

“Therefore, bagging and random forests can be used to approximate any unknown function, even if it is nonlinear and involves complex interactions.”

Classification and regression trees are provably Bayes consistent, i.e. in principle they can approximate any decision boundary, whether linear or highly nonlinear, given a sufficiently large data set and allowed to grow at a proper rate.

“Classification and regression trees are provably Bayes consistent, i.e. in principle they can approximate any decision boundary, whether linear or highly nonlinear, given a sufficiently large data set and allowed to grow at a proper rate”

• cites Devroye, Györfi, and Lugosi 1996Not checked yet

  Not checked in the free run.

Lunetta et al. (2004) find that genetic markers relevant in interactions with other markers or environmental variables can be detected more efficiently by random forests than by univariate screening methods like Fisher's exact test.

“Lunetta et al. (2004) find that genetic markers relevant in interactions with other markers or environmental variables can be detected more efficiently by means of random forests than by means of univariate screening methods like Fisher’s exact test.”

• cites Lunetta et al. (2004)Not checked yet

  Not checked in the free run.

In many applications involving predictor variables of different types, the Gini importance measure is biased.

“in many applications involving predictor variables of different types, this measure is biased”

Variable selection can be limited to random subsets in random forests, making them particularly well applicable in small n large p problems.

“The fact that variable selection can be limited to random subsets in random forests make them particularly well applicable in “small n large p” problems”

A reasonable measure for variable importance is the difference in prediction accuracy before and after permuting a variable, averaged over all trees.

“a reasonable measure for variable importance is the difference in prediction accuracy (i.e. the number of observations classified correctly; usually the out-of-bag prediction accuracy is used to compute the permutation importance) before and after permuting a variable, averaged over all trees.”

Variables not included in a tree did not contribute to the model, at least not in the context of the previously chosen splitting variables.

“variables that are not included in the tree did not contribute to the model – at least not in the context of the previously chosen splitting variables.”

In the original study of Ryan et al. (2006), no genes were clearly found to be differentially expressed in this sample.

“In the original study of Ryan et al. (2006), no genes were clearly found to be differentially expressed, i.e., to have an effect on the disease, in this sample.”

• cites Ryan et al. (2006)Not checked yet

  Not checked in the free run.

A conditional version of the random forest permutation importance resembles the properties of partial correlations rather than those of dominance analysis.

“A conditional version of the random forest permutation importance, that resembles the properties of partial correlations rather than that of dominance analysis, is suggested by Strobl et al. (2008).”

• cites Strobl et al. (2008)Not checked yet

  Not checked in the free run.

Bureau et al. (2005) and Heidema et al. (2006) point out that the recursive partitioning strategy is a clear advantage of random forests as opposed to more common methods like logistic regression in high dimensional settings.

“Bureau et al. (2005) and Heidema, Boer, Nagelkerke, Mariman, van der A, and Feskens (2006) point out that the recursive partitioning strategy is a clear advantage of random forests as opposed to more common methods like logistic regression in high dimensional settings.”

• cites Bureau et al. (2005)Not checked yet

  Not checked in the free run.

• cites Heidema et al. (2006)Not checked yet

  Not checked in the free run.

The main advantage of random forest permutation accuracy importance compared to univariate screening methods is that it covers the impact of each predictor variable individually as well as in multivariate interactions with other predictor variables.

“the main advantage of the random forest permutation accuracy importance, as compared to univariate screening methods, is that it covers the impact of each predictor variable individually as well as in multivariate interactions with other predictor variables.”

An ensemble of trees gives each variable the chance to appear in different contexts with different covariates and can thus better reflect its potentially complex effect on the response.

“an ensemble of trees has the advantage that it gives each variable the chance to appear in different contexts with different covariates, and can thus better reflect its potentially complex effect on the response.”

Ensembles of trees can approximate functions more smoothly by averaging over the single trees' step-functions.

“Ensembles of trees, however, can approximate functions more smoothly by averaging over the single trees’ step-functions.”

Order effects induced by the recursive variable selection scheme in constructing single trees are eliminated by averaging over the entire ensemble.

“order effects induced by the recursive variable selection scheme employed in constructing the single trees are eliminated by averaging over the entire ensemble.”

The R functions ctree and cforest guarantee unbiased variable selection when used with default parameter settings.

“For applications in R, the functions ctree for classification and regression trees and cforest for bagging and random forests (both freely available in the add-on package party; Hothorn et al. 2006; Hothorn, Hornik, and Zeileis 2008) guarantee unbiased variable selection when used with the default parameter settings, as documented in the supplement to this work.”

• cites Hothorn et al. 2006Not checked yet

  Not checked in the free run.

• cites Hothorn, Hornik, and Zeileis 2008Not checked yet

  Not checked in the free run.

In stepwise variable selection and single trees, order effects are present because only one variable at a time is considered in the context of variables already selected, regardless of variables yet to come.

“In both, stepwise variable selection and single trees, order effects are present, because only one variable at a time is considered – in the context of the variables that were already selected, but regardless of all variables yet to come.”

Ensemble methods that employ several parallel tree models counterbalance the order effects of individual trees, making the overall importance ranking of a variable more reliable than its position in stepwise selection.

“However, the advantage of ensemble methods, that employ several parallel tree models, is that the order effects of all individual trees counterbalance, so that the overall importance ranking of a variable is much more reliable than its position in stepwise selection (see also Rossi et al. 2005).”

• cites Rossi et al. 2005Not checked yet

  Not checked in the free run.

If the number of randomly preselected predictor variables is very low, interactions of high order may be missed in the tree building process.

“In addition to that, if the number of randomly preselected predictor variables is very low, interactions of high order may be missed in the tree building process.”

Reporting the significance of variable importance scores can be highly misleading because the number of variables whose scores exceed a given significance threshold depends on the arbitrary choice of a tuning parameter.

“Thus reporting the significance of variable importance scores (like, e.g., Baca-Garcia et al. 2007, who do not even report the parameter settings they use for fitting the random forest) can be highly misleading, because the number of variables whose scores exceed a given threshold for significance depends on the arbitrary choice of a tuning parameter.”

The choice of tuning parameters in random forests is not as critical as in other computer-intensive approaches such as support vector machines.

“Note, however, that the choice of tuning parameters in random forests is not as critical as in other computerintensive approaches, such as support vector machines (Svetnik, Liaw, Tong, and Wang 2004)”

• cites Svetnik, Liaw, Tong, and Wang 2004Not checked yet

  Not checked in the free run.

Bagging and random forests provide out-of-bag observations as a built-in test sample that gives a fair means of error estimation.

“It was already mentioned, and used in the application example, that bagging and random forests come with their own “built-in” test sample, the out-of-bag observations, that provide a fair means of error estimation (Breiman 1996b).”

• cites Breiman 1996bNot checked yet

  Not checked in the free run.

The simple statistical test based on the supposed normality of the z-score has such alarming statistical properties that any statement of significance made with it is nullified.

“However, it shows such alarming statistical properties that any statement of significance made with this test is nullified (Strobl and Zeileis 2008)”

• cites Strobl and Zeileis 2008Not checked yet

  Not checked in the free run.

In a perfectly symmetric artificial XOR problem, a tree would not find a cutpoint to start with.

“In such a perfectly symmetric, artificial XOR problem, a tree would indeed not find a cutpoint to start with.”

The resulting permutation importance can be interpreted reliably only when subsamples drawn without replacement, instead of bootstrap samples, are used together with unbiased split selection criteria in constructing the forest.

“Only when subsamples drawn without replacement, instead of bootstrap samples, in combination with unbiased split selection criteria, are used in constructing the forest, the resulting permutation importance can be interpreted reliably (Strobl et al. 2007).”

• cites Strobl et al. 2007Not checked yet

  Not checked in the free run.

In situations with few relevant variables, small mtry results in many trees being built that do not incorporate any relevant variables, which would lead to a decrease in prediction accuracy.

“In situations with few relevant variables, “small mtry results in many trees being built that do not incorporate any of the relevant [variables]” (Diaz-Uriarte and Alvarez de Andrés 2006), which would lead to a decrease in prediction accuracy.”

• cites Diaz-Uriarte and Alvarez de Andrés 2006Not checked yet

  Not checked in the free run.

Random forests and bagging are truly random models in the sense that, for the same data set, results may differ between two computation runs.

“One special characteristic of random forests and bagging, that new users are often not entirely aware of, is that they are truly “random” models in the sense that, for the same data set, the results may differ between two computation runs.”

Random preselection of splitting variables in random forests increases the chance that a variable with a weak marginal effect is still selected, at least in some trees, because some of its competitors are not available.

“the random preselection of splitting variables in random forests again increases the chance that a variable with a weak marginal effect is still selected, at least in some trees, because some of its competitors are not available.”

The power of the test based on the z-score depends on the number of trees in the ensemble ntree over which the importance is averaged.

“The power of this test depends on the number of trees in the ensemble ntree, over which the importance is averaged (cf. Equations 2 and 3 in section “Variable importance”; see also Lunetta et al. 2004).”

• cites Lunetta et al. 2004Not checked yet

  Not checked in the free run.

A tree, and even better an ensemble of trees, can approximate the XOR problem by means of a sequence of cutpoints driven by random fluctuations present in any real data set.

“a tree, and even better an ensemble of trees, is able to approximate the XOR problem by means of a sequence of cutpoints driven by random fluctuations that are present in any real data set.”

When predictor variables are highly correlated, a higher number of randomly preselected predictor variables is better suited to reflect conditional importance.

“When predictor variables are highly correlated, the results of Strobl et al. (2008) indicate that a higher number of randomly preselected predictor variables is better suited to reflect conditional importance.”

• cites Strobl et al. 2008Not checked yet

  Not checked in the free run.

The number of randomly preselected predictor variables mtry may affect the stability of the model and the reliability of the variable importance.

“The number of randomly preselected predictor variables mtry may also affect the stability of the model and the reliability of the variable importance.”

The study referred to in Breiman (2001b), which states that random forests do not overfit, may be a prominent example of a premature conclusion drawn from an unrepresentative sample.

“The study referred to in Breiman (2001b), where it is stated (and has been extensively cited ever since) that random forests do not overfit, may be a prominent example for a premature conclusion drawn from an unrepresentative sample.”

• cites Breiman 2001bNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: A variety of studies exploring the characteristics of machine learning tools such as random forests are based on only a few real data sets that happen to be freely available in some data repository.

The real data case studies referred to in Breiman (2001b) do not exclude the possibility that they overfit due to other reasons.

“However, the real data “case studies” referred to in Breiman (2001b) do not exclude the possibility that they overfit due to other reasons.”

• cites Breiman 2001bNot checked yet

  Not checked in the free run.

For problems with small sample sizes or even small n large p problems, growing large trees usually leads to the best performance.

“On the other hand for problems with small sample sizes or even “small n large p” problems, growing large trees usually does lead to the best performance.”

• cites Lin and Jeon 2006Not checked yet

  Not checked in the free run.

The recent results of Lin and Jeon (2006) show that creating large trees is not necessarily the optimal strategy.

“While most previous publications have argued that in an ensemble each individual tree should be grown as large as possible and that trees should not be pruned, the recent results of Lin and Jeon (2006) also show that creating large trees is not necessarily the optimal strategy.”

• cites Lin and Jeon 2006Not checked yet

  Not checked in the free run.

The theoretical results of Breiman (1996a) support the fact that ensemble methods do not overfit with an increasing number of trees.

“The theoretical results of Breiman (1996a) do support the fact that ensemble methods do not overfit with an increasing number of trees.”

• cites Breiman 1996aNot checked yet

  Not checked in the free run.

In problems with a high number of observations and few variables, a better convergence rate of the mean squared error can be achieved when the terminal node size increases with the sample size, i.e. when smaller trees are grown for larger samples.

“In problems with a high number of observations and few variables a better convergence rate (of the mean squared error as a measure of prediction accuracy) can be achieved when the terminal node size increases with the sample size (i.e. when smaller trees are grown for larger samples).”

• cites Lin and Jeon 2006Not checked yet

  Not checked in the free run.

Empirical comparisons between different supervised learning methods indicate that random forests are among the best performing methods even without extra tuning.

“With respect to ease of application, the results of the empirical comparisons between different supervised learning methods conducted by Caruana and Niculescu-Mizil (2006) and Svetnik et al. (2004) indicate that random forests are among the best performing methods even without extra tuning.”

• cites Caruana and Niculescu-Mizil 2006Not checked yet

  Not checked in the free run.

• cites Svetnik et al. 2004Not checked yet

  Not checked in the free run.

In contrast to random forests, shrinkage methods such as LASSO, the elastic net, and the Dantzig selector require the assumption that the model is linear or additive and that the problem is sparse.

“However, in contrast to random forests, for these methods it has to be assumed that the model is linear or additive and that the problem is sparse (meaning that only few predictor variables have an effect).”

• cites Hastie et al. 2001Not checked yet

  Not checked in the free run.

• cites Zou and Hastie 2005Not checked yet

  Not checked in the free run.

• cites Candes and Tao 2007Not checked yet

  Not checked in the free run.