Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection
Open accessThe DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Abstract
Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state-of-the-art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) measures. We demonstrate that although these implementation demonstrate highly competitive predictive performance, they still, surprisingly, suffer from bias in FI. By utilizing cross-validated (CV) unbiased base learners, we fix this flaw at a relatively low computational cost. We demonstrate the suggested framework in a variety of synthetic and real-world setups, showing a significant improvement in all GBM FI measures while maintaining relatively the same level of prediction accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also kept the top 3 edges for each downstream gene to reduce any gene specific bias caused by feature importance.81”
From SCING: Inference of robust, interpretable gene regulatory networks from single cell and spatial transcriptomics (2023) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 4×
“GBM (Friedman, 2001) is an ensemble of base (weak) learners.”
BackgroundInfluentialA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · cited 2×
“We randomly sample 30,000 observations, such that half of them have a positive label and the others are negative (following [37]).”
BackgroundMethodFriedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067
Feature Selection with the Boruta Package
Kursa, M. B., & Rudnicki, W. R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i11 · Cited by 5,659 · cited 2×
“For example, methods that rely on greedy backward elimination of features (such as (Kursa et al., 2010; Gregorutti et al., 2017)), still need to use a FI measure to score each variable in the final selected feature subset.”
MethodUnbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021 · cited 4×
“These approaches are built on a well-defined statistical theory, which either assumes a priori modeling assumptions on the distribution of the features [31] or using permutation or non-parametric tests such as in CIT [32].”
BackgroundMethodInfluentialBias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,688
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
From RankNet to LambdaRank to LambdaMART: An Overview
Burges, C. J. C. (2010) · Cited by 1,448
Consistent Individualized Feature Attribution for Tree Ensembles
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018) · arXiv · doi:10.48550/arxiv.1802.03888 · Cited by 1,430
CatBoost: unbiased boosting with categorical features
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09516 · Cited by 1,110
Interpretable Machine Learning
Molnar, C. (2020) · Cited by 1,104
Correlation and variable importance in random forests
Gregorutti, B., Michel, B., & Saint-Pierre, P. (2017) · Statistics and Computing · doi:10.1007/s11222-016-9646-1 · Cited by 1,073
Predicting clicks: estimating the click-through rate for new ads
Richardson, M., Dominowska, E., & Ragno, R. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242643 · Cited by 1,049
Distribution-Free Predictive Inference for Regression
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1307116 · Cited by 1,049 · cited 2×
“The leave one covariate out (LOCO) (Lei et al., 2018) on the prediction (Molnar, 2020).”
BackgroundSPLIT SELECTION METHODS FOR CLASSIFICATION TREES
Loh, W., & Shih, Y. (1997) · Cited by 977
Bishop, C. M. (2004) · Handbook of Neural Computation · doi:10.1887/0750303123/b365c43 · Cited by 833
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
Fifty Years of Classification and Regression Trees
Loh, W. (2014) · International Statistical Review · doi:10.1111/insr.12016 · Cited by 649
Nicodemus, K. K., Malley, J. D., Strobl, C., & Ziegler, A. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-110 · Cited by 376
Parallel boosted regression trees for web search ranking
Tyree, S., Weinberger, K. Q., Agrawal, K., & Paykin, J. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963461 · Cited by 161
Classification Trees With Bivariate Linear Discriminant Node Models
Kim, H., & Loh, W. (2003) · Journal of Computational and Graphical Statistics · doi:10.1198/1061860032049 · Cited by 97
Song, C., & Zhang, H. (2014) · International Statistical Review · doi:10.1111/insr.12060 · Cited by 90
Feature selection for ranking using boosted trees
Pan, F., Converse, T., Ahn, D., Salvetti, F., & Donato, G. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1646292 · Cited by 85
Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance
Painsky, A., & Rosset, S. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2636831 · Cited by 84
Self-citationUsing a Permutation Test for Attribute Selection in Decision Trees
Frank, E., & Witten, I. H. (1998) · Research Commons (University of Waikato) · Cited by 55
On the Universality of the Logistic Loss Function
Painsky, A., & Wornell, G. W. (2018) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437786 · Cited by 50
Self-citationSelecting multiway splits in decision trees
Frank, E., & Witten, I. H. (1996) · Research Commons (University of Waikato) · Cited by 17
Ranking Categorical Features Using Generalization Properties
Sivan Sabato, A., & Shalev‐Shwartz, S. (2008) · Journal of Machine Learning Research · Cited by 10
Research works printed but not linked: 1
- Loh W.Y. Regression tress with unbiased variable selection and interaction detection Stat. Sin. 2002 12 361 386
Cited by
Chiu, C., Wu, C.-M., Chien, T.-N., Kao, L., Li, C., & Chu, C.-M. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20054340 · Cited by 57
Efficient permutation testing of variable importance measures by the example of random forests
Hapfelmeier, A., Hornung, R., & Haller, B. (2023) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2022.107689 · Cited by 47
Yin, H., Wang, K., Yang, R., Tan, Y., Li, Q., Zhu, W., & Sung, S. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.108005 · Cited by 40
Machine Learning-Based Crop Stress Detection in Greenhouses
Elvanidi, A., & Katsoulas, N. (2022) · Plants · doi:10.3390/plants12010052 · Cited by 39
Debiasing SHAP scores in random forests
Loecher, M. (2023) · AStA Advances in Statistical Analysis · doi:10.1007/s10182-023-00479-7 · Cited by 40
Metadata
- Authors
- Adler, A. I., & Painsky, A.
- Affiliations
- Tel Aviv University (IL)
- Corresponding author
- Painsky, Amichai
- ORCID
- Painsky, Amichai
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2022-05-13
- Volume
- 24
- Issue
- 5
- Pages
- 687
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC9140774
- OpenAlex
- W3200824799
- Cited by
- 149
- References
- 39
- Citations by year
- 2026: 35 · 2025: 59 · 2024: 32 · 2023: 18 · 2022: 5
- FWCI
- 9.38
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- gradient boosting, feature importance, categorical variables, decision tree, tabular data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.