Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection
Open accessThe DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Abstract
Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state-of-the-art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) measures. We demonstrate that although these implementation demonstrate highly competitive predictive performance, they still, surprisingly, suffer from bias in FI. By utilizing cross-validated (CV) unbiased base learners, we fix this flaw at a relatively low computational cost. We demonstrate the suggested framework in a variety of synthetic and real-world setups, showing a significant improvement in all GBM FI measures while maintaining relatively the same level of prediction accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also kept the top 3 edges for each downstream gene to reduce any gene specific bias caused by feature importance.81”
From SCING: Inference of robust, interpretable gene regulatory networks from single cell and spatial transcriptomics (2023) · cites this paper as [81]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 4×
“GBM (Friedman, 2001) is an ensemble of base (weak) learners.”
BackgroundInfluentialA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · cited 2×
“We randomly sample 30,000 observations, such that half of them have a positive label and the others are negative (following [37]).”
BackgroundMethod
Research works printed but not linked: 1
- Loh W.Y. Regression tress with unbiased variable selection and interaction detection Stat. Sin. 2002 12 361 386
Cited by
Items 101–186 of 186
Niu, Y., LaBass, I., Tye, Y. Y., Smith, T., Wang, S., Li, Y., Han, T., & Leh, C. P. (2026) · ACS Omega · doi:10.1021/acsomega.6c04765 · Cited by 2
Takefuji, Y. (2025) · Journal of Hepatology · doi:10.1016/j.jhep.2025.05.025 · Cited by 2
Exploiting Data Distribution: A Multi-Ranking Approach
Zielosko, B. M., Jabloński, K., & Dmytrenko, A. (2025) · Entropy · doi:10.3390/e27030278 · Cited by 2
Zhao, Z., Cheng, X., Gao, Y., Tan, F., Xue, Q., Gao, S., & He, J. (2025) · Translational Lung Cancer Research · doi:10.21037/tlcr-24-331 · Cited by 2
AI-powered deep ultraviolet laser diode design for resource-efficient optimization
Sarwar, A., Usman, M., Hussain, M., Jadoon, K. K., Manzoor, T., & Ali, S. (2025) · Journal of Computational Electronics · doi:10.1007/s10825-025-02352-z · Cited by 2
Zhang, Q., Wang, F., Zhang, S., Luo, L., & Shi, Z. (2025) · Contemporary Mathematics · doi:10.37256/cm.6420257205 · Cited by 2
Bakasa, W., & Viriri, S. (2023) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2023.1175 · Cited by 2
Sun, Q. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.04409 · Cited by 2
Deep incremental learning models for financial temporal tabular datasets with distribution shiftsPreprint
Wong, T., & Barahona, M. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2303.07925 · Cited by 2
Machine Learning Approaches to Predict Alcohol Consumption from Biomarkers in the UK BiobankPreprint
Hassan, M. F., Gentry, A. E., Prom‐Wormley, E. C., Peterson, R. E., & Webb, B. T. (2024) · medRxiv · doi:10.1101/2024.12.22.24319486 · Cited by 2
FaKy: A Feature Extraction Library to Detect the Truthfulness of a Text
Hamers, S. B., & Ceolin, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47896-3_3 · Cited by 1
Grzybowska, U., & Karwański, M. (2024) · Metody Ilościowe w Badaniach Ekonomicznych · doi:10.22630/mibe.2023.24.4.18 · Cited by 1
EFFECTIVENESS OF VARIABLE SELECTION METHODS FOR MACHINE LEARNING AND CLASSICAL STATISTICAL MODELS
Grzybowska, U., & Karwański, M. (2024) · Metody Ilościowe w Badaniach Ekonomicznych · doi:10.22630/mibe.2024.25.2.6 · Cited by 1
Enhancing Smart Grid Security: An Data-Driven Anomaly Detection Framework
Son, N. K., Sangaiah, A. K., Medhane, D. V., Alenazi, M. J. F., & AlQahtani, S. A. (2024) · IEEE Conference on Communications and Network Security · doi:10.1109/cns62487.2024.10735564 · Cited by 1
Tretyakov, A. K., Vrazhnov, D. A., Shkurinov, A. P., Zasedatel, V. S., & Kistenev, Y. V. (2024) · Applied Sciences · doi:10.3390/app142411521 · Cited by 1
Comment on “Reassessing predictive modeling for emergency department return in COVID-19 patients”
Hieu, N. M., & Takefuji, Y. (2025) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2025.01.009 · Cited by 1
Enhancing Agile Defect Prediction with Optimized Machine Learning and Feature Selection
Wicaksono, F. D., & Siahaan, D. (2025) · Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) · doi:10.29207/resti.v9i4.6713 · Cited by 1
Oka, S., & Takefuji, Y. (2025) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2025.08.035 · Cited by 1
Asad, A. B., Paudel, A., Kshetri, S., Kang, C., Khanal, S. R., Shcherbatyuk, N., Davadant, P., Schreiner, R. P., Kalauni, S., Karkee, M., et al. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.112093 · Cited by 1
Nakajima, K., & Sekine, A. (2025) · AI in Clinical Medicine · doi:10.14740/aicm10 · Cited by 1
Liu, Y., Zhang, Z., Liu, J., & Li, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131955 · Cited by 1
Liao, W., Wei, Z., Wei, M., & Wei, Y. (2025) · Environmental Monitoring and Assessment · doi:10.1007/s10661-025-14768-y · Cited by 1
Beyond accuracy: Stabilizing feature importance in GWRF/RF for soil heavy metal mapping
Takefuji, Y. (2025) · Environmental Pollution · doi:10.1016/j.envpol.2025.127447 · Cited by 1
Mustafa, A. I., & Alyozbaky, O. S. (2025) · International Conference on Power Systems and Electrical Technology (PSET) · doi:10.1109/pset66242.2025.11296520 · Cited by 1
Feature Classification Methods for Measuring User Engagement in Social Media Campaigns
Krishnamurthy, S., Misopoulos, F., Leung, E. K. H., & Antonopoulou, K. (2026) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2026.3664701 · Cited by 1
Machine learning-driven cancer diagnostics with improved robustness and interpretability
Li, P., & Liu, Z. (2026) · Chemical Science · doi:10.1039/d6sc02368a · Cited by 1
Duan, C., Chen, Y., He, J., & Ng, K. H. (2026) · Asian Academy of Management Journal of Accounting and Finance · doi:10.21315/aamjaf2026.22.1.3 · Cited by 1
Jin, X., VARQUEZ, A. C. G., Khanh, D. N., Ihara, T., Itsubo, N., Kanda, M., & Junnaedhi, I. D. G. A. (2026) · Earth and Space Science · doi:10.1029/2025ea004856 · Cited by 1
Machine Learning Techniques for Electricity Theft Detection in Smart Grids: A Comprehensive Review
Apata, O., Ratshitanga, M., & Davidson, I. E. (2026) · Energies · doi:10.3390/en19163877 · Cited by 2
Coral reef exposure increases aerosol and cloud condensation nuclei over the Great Barrier Reef
Sulo, J., Okuljar, M., Alroe, J., Li, Z., Horchler, E. J., Cravigan, L. T., Miljevic, B., Harrison, L. P., Harrison, D. P., & Ristovski, Z. (2026) · Aerosol Research · doi:10.5194/ar-4-413-2026 · Cited by 1
Takefuji, Y. (2026) · Civil Engineering Design · doi:10.1002/cend.70026 · Cited by 1
Data driven approach for weight restricted data envelopment analysis models with single output
Şenol, K. U. R. T., Yüksel, M. K., & Dinçergök, B. (2023) · Journal of Turkish Operations Management · doi:10.56554/jtom.1333333 · Cited by 1
Yoshioka, N., Takeuchi, H., Shu, Y., Okamatsu, T., Araki, N., Kamakura, Y., & Ohsuga, M. (2024) · Sensors · doi:10.3390/s24196465 · Cited by 1
Cognitive Frailty Classification Models for Older Adults in a Point-of-Care System
Cao, L., Lim, Y. W., Tan, M. P., & Rokhani, F. Z. (2024) · IEEE Asia Pacific Conference on Circuits and Systems (APCCAS) · doi:10.1109/apccas62602.2024.10808371 · Cited by 1
Kannan, V., Shuff, J., Acharya, A., Venkatesh, R., Shekhawat, N. S., & Parikh, K. S. (2025) · medRxiv · doi:10.1101/2025.11.14.25340278 · Cited by 1
Informing decision-making in single-objective, mixed-variable design problems
Fang, D. L. (2024) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1
Algoritmos de selección de características en aprendizaje automático
Sánchez Martín, D. (2024) · Gredos (University of Salamanca) · Cited by 1
Amin, M. A., Naznin, F., Yeasmin, M. N., Sarkar, M. S., Mia, M. M., Chowdhury, A., & Islam, M. Z. (2024) · F1000Research · doi:10.12688/f1000research.144871.1
The European Film Industry as a Market for LemonsPreprint
Pfeifer, M., & Lagarias, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4993704
Determinants Shaping Human Preference for Thermal Springs in Palaeolithic Europe and Asia Minor
Trájer, A. J. (2025) · Environmental Archaeology · doi:10.1080/14614103.2025.2452724
Utilizing Machine Learning Techniques to Analyze Primary Cilia in Postmortem Tissue Samples
Mazumder, A., & Mazumder, A. (2024) · IEEE International Conference on Technology, Informatics, Management, Engineering and Environment (TIME-E) · doi:10.1109/time-e62724.2024.10919807
Privacy-Aware Income Prediction Using Deep Neural Networks on the UCI Adult Dataset
Benyamina, O. N. A. (2025) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v10i57s.12273
Oka, S., & Takefuji, Y. (2025) · Clinical Lung Cancer · doi:10.1016/j.cllc.2025.09.005
Bhatia, A., Kumar, R., & Manteghi, G. (2025) · International Journal of Emerging Science and Engineering · doi:10.35940/ijese.l2620.13111025
Ensemble recursive feature elimination-based ensemble classification for medical diagnosis
Ramanathan, T. T., Hossen, M. J., Mamun, A. A., & Raja, J. E. (2025) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v40.i2.pp758-771
Takefuji, Y. (2025) · Journal of environmental chemical engineering · doi:10.1016/j.jece.2025.120094
Oka, S., & Takefuji, Y. (2025) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2025.09.008
Subhadarsini, J., Chattopadhyaya, J., Bhardwaj, S., Sahu, M., Jhariya, D. C., & Vidyarthi, V. K. (2025) · doi:10.1109/cises66934.2025.11265515
AFŞİN, Y. (2023) · OpenMETU (Middle East Technical University)
Araig, Y., & Takefuji, Y. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106238
Towards reliable feature interpretation in machine learning-based longevity prediction
Oka, S., Takahashi, Y., & Takefuji, Y. (2026) · Annals of Epidemiology · doi:10.1016/j.annepidem.2026.01.005
Ajvad Haneef, K., Singh, K. K., & D, M. K. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.15754
Ozkan, M., Atakan, M., & YILDIZ, K. (2026) · IEEE Access · doi:10.1109/access.2026.3664515
Algarra, D. F. G., & Díaz, L. N. G. (2023) · Repositorio Institucional FULL
Mankad, M. D. (2026) · INTERNATIONAL JOURNAL OF RESEARCH AND ANALYTICAL REVIEWS · doi:10.56975/ijrar.v13i1.328565
Handika, F. S., Yulianto, L. D., & Andryana, S. (2026) · Journal of Information Systems and Informatics · doi:10.63158/journalisi.v8i1.1456
Sélection de variables et analyse d'association pour les données cliniques du Myélome Multiple
Kaddour, R. G. E. H. (2023) · Dépôt Institutionnel de lUniversité de Tlemcen
Addressing biases and limitations in feature attribution for circRNA modification profiling
Oka, S., Takemura, K., & Takefuji, Y. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag168
The BES–GDP Nexus: A Panel Econometric and Machine Learning Analysis of Italian Regions
Leogrande, A., Arnone, M., Drago, C., Costantiello, A., & Anobile, F. (2026) · Land · doi:10.3390/land15050825
AI home monitoring for behavioral markers of cerebrovascular disease
Baek, J., Cho, K.-H., Lim, L., & Chong, J. W. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02836-7
Aji, B. W., Raharjo, A. A. R., Hikmah, F. N., Afifah, L., Qoyyimah, Surarso, B., & Sumanto, Y. D. (2026) · AIP conference proceedings · doi:10.1063/5.0331691
Multi-Stage Evaluation Framework for Identifying Deployment-Ready Prediabetes Prediction Models
Sher, M., & Toma, M. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8070192
Oka, S., Takemura, K., & Takefuji, Y. (2026) · Diabetes Obesity and Metabolism · doi:10.1111/dom.71140
Frequency-Domain Feature Analysis for Early Gear Damage Detection in Planetary Gearboxes
Binanzer, L., & Dazer, M. (2026) · PHM Society European Conference · doi:10.36001/phme.2026.v9i1.5021
Predicting the Workability of PCE-Modified Cement–Fly Ash Pastes with Explainable Machine Learning
Gider, V., Ekinci, S., İzci, D., Güneş, B. B., Budak, C., ÖZTEBER, S., & Mardani, A. (2026) · Buildings · doi:10.3390/buildings16152965
Strengthening Feature-Importance Interpretation in Pediatric Telemedicine Prediction Models
Oka, S., Saito, S., & Takefuji, Y. (2026) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2026.115268
Zhu, F., Cui, H., Guo, X., Xiong, H., Li, X., Cui, D., He, Y., Shi, Y., Zhi, F., Xiong, R., et al. (2026) · Geocarto International · doi:10.1080/10106049.2026.2712662
Vasić, M. V. (2026) · Ceramics International · doi:10.1016/j.ceramint.2026.08.125
Lončar, B. L., Radosavljević, M., Filipović, J., Djalović, I. G., Košutić, M., Filipović, V., & Nićetin, M. R. (2026) · Foods · doi:10.3390/foods15162854
Comparative analysis of classical and deep machine learning architectures in wildfire forecasting
Goncharov, A., Александрович, Б. А., & Matveev, A. (2026) · National Security and Strategic Planning · doi:10.37468/2307-1400-2026-1-78-89
iBitter-HF: A Method for Bitter Peptide Sequence Identification Based on Hybrid Feature Embedding
Yan, F., Xiang, S., Yi, T., Kuang, Z., Liu, H., Luo, X., & Lv, Z. (2026) · Foods · doi:10.3390/foods15173016
Oka, S., Suzuki, M., & Takefuji, Y. (2026) · Geriatric Nursing · doi:10.1016/j.gerinurse.2026.104318
Sakti, A. D., Komara, H. Y., Sholihah, A., Santoso, C., Barrand, N. E., Pradhan, B., Takeuchi, W., Ward, R. D., Lotfata, A., Abdillah, M. R., et al. (2026) · Ecological Informatics · doi:10.1016/j.ecoinf.2026.104028
The revival of bagged trees: hierarchical shrinkage as a regularization tool
Loecher, M., Gevaert, A., Pfeifer, B., Adhikari, S., & Holzinger, A. (2026) · Machine Learning Science and Technology · doi:10.1088/2632-2153/aea39d
Takefuji, Y. (2026) · Next research. · doi:10.1016/j.nexres.2026.102457
Ahumada‐García, R., Zabala‐Blanco, D., Monzón, V. H., Sánchez, I., da Silva, N. F. F., Rosa, T. C., Ferreira, A. I. S., López-Cortés, X. A., Flores-Calero, M. J., & Vásquez-Iglesias, P. (2026) · Smart Agricultural Technology · doi:10.1016/j.atech.2026.102575
Machine learning-driven strategies for biofuel production and process optimization
Rupesh, S., & Acharya, S. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-34088-8.00145-2
AI-Driven Decision Support System for Enhancing Business Operations
Almehrzi, M., Copiaco, A., & Mukhtar, H. (2026) · Student Multidisciplinary Innovation and Research Conference (SMIRC) · doi:10.1109/smirc70459.2026.11706680
Stefaniak, F. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c02217
Dai, J., Xu, Y., Wang, X., Zhang, L., Ma, M., 饶凯锋, & 王子健 (2026) · ACS ES&T Toxicology · doi:10.1021/acsesttox.6c00052
Elsayad, A. M., & Elsayad, K. A. (2026) · Scientific Reports · doi:10.1038/s41598-026-76515-8
Islam, T., Saha, S., Ganguly, S., Bhattacharya, P., Naskar, S., khatun, J., Mandi, A. K., Bhandari, P., Dutta, S. S., Koley, M., et al. (2026) · Indian Journal of Homeopathy and Integrative Medicine · doi:10.4103/ijhim.ijhim_16_26
Olawore, A. S., Oladosu, K. O., Wong, K. Y., Kareem, M. W., Kolawole, M. Y., Saidu, N. S., & Adesope, W. A. (2026) · BioEnergy Research · doi:10.1007/s12155-026-11069-1
Machine-learning exploration of isotope-informed near-term temperature forecast at European sites
Samborska-Goik, K. (2026) · Atmospheric Research · doi:10.1016/j.atmosres.2026.109448
Metadata
- Authors
- Adler, A. I., & Painsky, A.
- Affiliations
- Tel Aviv University (IL)
- Corresponding author
- Painsky, Amichai
- ORCID
- Painsky, Amichai
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2022-05-13
- Volume
- 24
- Issue
- 5
- Pages
- 687
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC9140774
- OpenAlex
- W3200824799
- Cited by
- 149
- References
- 39
- Citations by year
- 2026: 35 · 2025: 59 · 2024: 32 · 2023: 18 · 2022: 5
- FWCI
- 9.38
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- gradient boosting, feature importance, categorical variables, decision tree, tabular data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.