The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Bias in random forest variable importance measures: Illustrations, sources and a solution
Open accessAbstract
BACKGROUND: Variable importance measures for random forests have been receiving increased attention as a means of variable selection in many classification tasks in bioinformatics and related scientific fields, for instance to select a subset of genetic markers relevant for the prediction of a certain disease. We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories. This is particularly important in genomics and computational biology, where predictors often include variables of different types, for example when predictors include both sequence data and continuous variables such as folding energy, or when amino acid sequence data show different numbers of categories. RESULTS: Simulation studies are presented illustrating that, when random forest variable importance measures are used with data of varying types, the results are misleading because suboptimal predictor variables may be artificially preferred in variable selection. The two mechanisms underlying this deficiency are biased variable selection in the individual classification trees used to build the random forest on one hand, and effects induced by bootstrap sampling with replacement on the other hand. CONCLUSION: We propose to employ an alternative implementation of random forests, that provides unbiased variable selection in the individual classification trees. When this method is applied using subsampling without replacement, the resulting variable importance measures can be used reliably for variable selection even in situations where the potential predictor variables vary in their scale of measurement or their number of categories. The usage of both random forest algorithms and their variable importance measures in the R system for statistical computing is illustrated and documented thoroughly in an application re-analyzing data from a study on RNA editing. Therefore the suggested method can be applied straightforwardly by scientists in bioinformatics research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
Self-citation
Cited by
Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
Wager, S., & Athey, S. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1319839 · Cited by 2,917
Variable selection using random forests
Genuer, R., Poggi, J.-M., & Tuleau-Malot, C. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2010.03.014 · Cited by 2,619
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583
Self-citationInferring Regulatory Networks from Expression Data Using Tree-Based Methods
Huynh-Thu, V. A., Irrthum, A., Wehenkel, L., & Geurts, P. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012776 · Cited by 2,131
Detecting gene–gene interactions that underlie human diseases
Cordell, H. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2579 · Cited by 1,350
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
Wang, J.-X., Wu, J.-L., & Xiao, H. (2017) · Physical Review Fluids · doi:10.1103/physrevfluids.2.034603 · Cited by 737
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Improved Random Forest for Classification
Paul, A., Mukherjee, D. P., Das, P., Gangopadhyay, A., Chintha, A. R., & Kundu, S. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2834830 · Cited by 483
Opening the black box: interpretable machine learning for geneticists
Azodi, C. B., Tang, J., & Shiu, S.-H. (2020) · Trends in Genetics · doi:10.1016/j.tig.2020.03.005 · Cited by 473
An Empirical Evaluation of Explanations for State Repression
HILL, D. W., & JONES, Z. M. (2014) · American Political Science Review · doi:10.1017/s0003055414000306 · Cited by 450
Coupling machine learning and crop modeling improves crop yield prediction in the US corn belt
Shahhosseini, M., Hu, G., Huber, I., & Archontoulis, S. V. (2021) · Scientific Reports · doi:10.1038/s41598-020-80820-1 · Cited by 452
A comparative study of machine learning classifiers for modeling travel mode choice
Hagenauer, J., & Helbich, M. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.01.057 · Cited by 424
Integrative Analysis of Drug Response and Clinical Outcome in Acute Myeloid Leukemia
Bottomly, D., Long, N., Schultz, A. R., Kurtz, S. E., Tognon, C. E., Johnson, K., Abel, M., Agarwal, A., Avaylon, S., Benton, E., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.002 · Cited by 403
Sensitive periods of amygdala development: the role of maltreatment in preadolescence
Pechtel, P., Lyons-Ruth, K., Anderson, C. M., & Teicher, M. H. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.04.025 · Cited by 342
“In addition, the conditional random forest technique used is not biased by independent variable parameters, such as variability, range or mean (Strobl et al., 2007).”
MethodInfluentialSchalinski, I., Teicher, M. H., Nischk, D., Hinderer, E., Müller, O., & Rockstroh, B. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-1004-5 · Cited by 327
Deep learning algorithm predicts diabetic retinopathy progression in individual patients
Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., & Prunotto, M. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0172-3 · Cited by 305
Modelling species presence‐only data with random forests
Valavi, R., Elith, J., Lahoz‐Monfort, J. J., & Guillera‐Arroita, G. (2021) · Ecography · doi:10.1111/ecog.05615 · Cited by 307
Post-fire vegetation and fuel development influences fire severity patterns in reburns
Coppoletta, M., Merriam, K. E., & Collins, B. M. (2016) · Ecological Applications · doi:10.1890/15-0225 · Cited by 295
Auditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280
On safari to Random Jungle: a fast implementation of Random Forests for high-dimensional data
Schwarz, D. F., König, I. R., & Ziegler, A. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq257 · Cited by 269
Mitchard, E. T., Saatchi, S. S., Baccini, A., Asner, G. P., Goetz, S. J., Harris, N. L., & Brown, S. (2013) · Carbon Balance and Management · doi:10.1186/1750-0680-8-10 · Cited by 250
Hooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264
Interpretation of Quantitative Structure-Activity Relationship Models: Past, Present, and Future
Polishchuk, P. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00274 · Cited by 236
Big data hurdles in precision medicine and precision public health
Prosperi, M., Min, J. S., Bian, J., & Modave, F. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0719-2 · Cited by 200
Bias-corrected random forests in regression
Zhang, G., & Lu, Y. (2012) · Journal of Applied Statistics · doi:10.1080/02664763.2011.578621 · Cited by 196
Tomoda, A., Polcari, A., Anderson, C. M., & Teicher, M. H. (2012) · PLoS ONE · doi:10.1371/journal.pone.0052528 · Cited by 190
White matter damage in frontotemporal lobar degeneration spectrum
Agosta, F., Scola, E., Canu, E., Marcone, A., Magnani, G., Sarro, L., Copetti, M., Caso, F., Cerami, C., Comi, G., et al. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr288 · Cited by 180
Berk, R., Sherman, L., Barnes, G., Kurtz, E., & Ahlman, L. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00556.x · Cited by 178
Lydersen, J. M., North, M. P., & Collins, B. M. (2014) · Forest Ecology and Management · doi:10.1016/j.foreco.2014.06.005 · Cited by 161
Marchese Robinson, R. L., Palczewska, A., Palczewski, J., & Kidley, N. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00753 · Cited by 167
'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
Zhang, K., Li, Y., Schwartz, J. D., & O׳Neill, M. S. (2014) · Environmental Research · doi:10.1016/j.envres.2014.04.004 · Cited by 152
Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection
Adler, A. I., & Painsky, A. (2022) · Entropy · doi:10.3390/e24050687 · Cited by 149
Knudby, A., LeDrew, E., & Brenning, A. (2010) · Remote Sensing of Environment · doi:10.1016/j.rse.2010.01.007 · Cited by 129
Weng, S. F., Vaz, L., Qureshi, N., & Kai, J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0214365 · Cited by 118
Kosaka, H., Okamoto, Y., Munesue, T., Yamasue, H., Inohara, K., Fujioka, T., Anme, T., Orisaka, M., Ishitobi, M., Jung, M., et al. (2016) · Translational Psychiatry · doi:10.1038/tp.2016.152 · Cited by 117
Adams, S. E., Arnold, D., Murphy, B., Carroll, P., Green, A. K., Smith, A. M., Marsh, P. D., Chen, T., Marriott, R. E., & Brading, M. G. (2017) · Scientific Reports · doi:10.1038/srep43344 · Cited by 111
Designing a feature selection method based on explainable artificial intelligence
Zacharias, J., von Zahn, M., Chen, J., & Hinz, O. (2022) · Electronic Markets · doi:10.1007/s12525-022-00608-1 · Cited by 109
Land-surface parameters for spatial predictive mapping and modeling
Maxwell, A. E., & Shobe, C. M. (2022) · Earth-Science Reviews · doi:10.1016/j.earscirev.2022.103944 · Cited by 105
A peek into the black box: exploring classifiers by randomization
Henelius, A., Puolamäki, K., Boström, H., Asker, L., & Papapetrou, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0368-8 · Cited by 98
Atabaki-Pasdar, N., Ohlsson, M., Viñuela, A., Frau, F., Pomares-Millan, H., Haid, M., Jones, A. G., Thomas, E. L., Koivula, R. W., Kurbasic, A., et al. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003149 · Cited by 97
Environmental adversities and psychotic symptoms: The impact of timing of trauma, abuse, and neglect
Schalinski, I., Breinlinger, S., Hirt, V., Teicher, M. H., Odenwald, M., & Rockstroh, B. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2017.10.034 · Cited by 95
Lindberg, D. S., Prosperi, M., Bjarnadottir, R. I., Thomas, J., Crane, M., Chen, Z., Shear, K., Solberg, L. M., Snigurska, U. A., Wu, Y., et al. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104272 · Cited by 95
Jaxa-Rozen, M., & Kwakkel, J. (2018) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2018.06.011 · Cited by 91
Everson, T. M., Lyons, G., Zhang, H., Soto-Ramírez, N., Lockett, G. A., Patil, V. K., Merid, S. K., Sӧderhӓll, C., Melén, E., Holloway, J. W., et al. (2015) · Genome Medicine · doi:10.1186/s13073-015-0213-8 · Cited by 78
Breitmeyer, S. E., Williams, A. M., Duris, J. W., Eicholtz, L. W., Shull, D. R., Wertz, T. A., & Woodward, E. E. (2023) · Science of The Total Environment · doi:10.1016/j.scitotenv.2023.164161 · Cited by 78
Prediction of hypoglycemia during aerobic exercise in adults with type 1 diabetes
Reddy, R., Resalat, N., Wilson, L. M., Castle, J. R., El Youssef, J., & Jacobs, P. G. (2019) · Journal of Diabetes Science and Technology · doi:10.1177/1932296818823792 · Cited by 75
Blyuss, O., Zaikin, A., Cherepanova, V., Munblit, D., Kiseleva, E. M., Prytomanova, O. M., Duffy, S. W., & Crnogorac-Jurcevic, T. (2020) · British Journal of Cancer · doi:10.1038/s41416-019-0694-0 · Cited by 67
Takiguchi, S., Fujisawa, T. X., Mizushima, S., Saito, D. N., Okamoto, Y., Shimada, K., Koizumi, M., Kumazaki, H., Jung, M., Kosaka, H., et al. (2015) · BJPsych Open · doi:10.1192/bjpo.bp.115.001586 · Cited by 63
RetractedWitnessing Violence Toward Siblings: An Understudied but Potent Form of Early Adversity
Teicher, M. H., & Vitaliano, G. D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028852 · Cited by 63
How Interpretable and Trustworthy are GAMs?
Chang, C.-H., Tan, S., Lengerich, B., Goldenberg, A., & Caruana, R. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467453 · Cited by 66
Canu, E., Agosta, F., Mandic-Stojmenovic, G., Stojković, T., Stefanova, E., Inuggi, A., Imperiale, F., Copetti, M., Kostic, V. S., & Filippi, M. (2017) · NeuroImage: Clinical · doi:10.1016/j.nicl.2017.05.018 · Cited by 62
Response of the human gut and saliva microbiome to urbanization in Cameroon
Lokmer, A., Aflalo, S., Amougou, N., Lafosse, S., Froment, A., Tabe, F. E., Poyet, M., Groussin, M., Said-Mohamed, R., & Ségurel, L. (2020) · Scientific Reports · doi:10.1038/s41598-020-59849-9 · Cited by 46
Lessons and tips for designing a machine learning study using EHR data
Arbet, J., Brokamp, C., Meinzen-Derr, J., Trinkley, K. E., & Spratt, H. M. (2021) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.513 · Cited by 46
Bracconi, M., & Maestri, M. (2020) · Chemical Engineering Journal · doi:10.1016/j.cej.2020.125469 · Cited by 48
From observation to information: data-driven understanding of on farm yield variation
Jiménez, D., Dorado, H., Cock, J., Prager, S. D., Delerce, S., Grillon, A., Andrade Bejarano, M., Benavides, H., & Jarvis, A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150015 · Cited by 47
DELL, J., WILCOX, C., & HOBDAY, A. J. (2011) · Fisheries Oceanography · doi:10.1111/j.1365-2419.2011.00591.x · Cited by 46
Data consistency in the English Hospital Episodes Statistics database
Hardy, F., Heyl, J., Tucker, K., Hopper, A., Marchã, M. J., Briggs, T. W. R., Yates, J., Day, J., Wheeler, A., Eve-Jones, S., et al. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100633 · Cited by 38
Zhang, R., Walker, A. R., & Datta, S. (2021) · Biology Direct · doi:10.1186/s13062-020-00284-1 · Cited by 30
Landscape Analysis of Adult Florida Panther Habitat
Frakes, R. A., Belden, R. C., Wood, B. E., & James, F. E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133044 · Cited by 30
Decay radius of climate decision for solar panels in the city of Fresno, USA
Barton-Henry, K., Wenz, L., & Levermann, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-87714-w · Cited by 28
Comparing Conventional and Machine‐Learning Approaches to Risk Assessment in Domestic Abuse Cases
Grogger, J., Gupta, S., Ivandic, R., & Kirchmaier, T. (2021) · Journal of Empirical Legal Studies · doi:10.1111/jels.12276 · Cited by 26
Wolfe, M. B., Schagat, T. L., Paulsen, M. T., Magnuson, B., Ljungman, M., Park, D., Zhang, C., Campbell, Z. T., Goldstrohm, A. C., & Freddolino, P. L. (2020) · RNA · doi:10.1261/rna.077362.120 · Cited by 22
Lebowitz, E. R., Zilcha‐Mano, S., Orbach, M., Shimshoni, Y., & Silverman, W. K. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13386 · Cited by 19
What’s all that racket! Soundscapes, phenology, and biodiversity in estuaries
Monczak, A., McKinney, B., Mueller, C., & Montie, E. W. (2020) · PLOS ONE · doi:10.1371/journal.pone.0236874 · Cited by 17
Pasca, D. P., Aloisio, A., De Santis, Y., Burkart, H., & Øvrum, A. (2025) · Engineering Structures · doi:10.1016/j.engstruct.2024.119218 · Cited by 15
Xie, Z., Gu, W., & Ming, T. (2023) · Combinatorial Chemistry & High Throughput Screening · doi:10.2174/1386207325666220404123433 · Cited by 6
Bitto, V., Hönscheid, P., Besso, M. J., Sperling, C., Kurth, I., Baumann, M., & Brors, B. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00385-x · Cited by 6
Parsimonious statistical learning models for low flow estimation
Laimighofer, J., Melcher, M., & Laaha, G. (2021) · doi:10.5194/hess-2021-481 · Cited by 30
Metadata
- Authors
- Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T.
- Venue
- BMC Bioinformatics
- Published
- 2007-01-25
- Volume
- 8
- Issue
- 1
- Pages
- 25-25
- ISSN
- 1471-2105 (ISSN-L)
- OpenAlex
- W1875061881
- Cited by
- 3,688
- References
- 3
- FWCI
- 37.84
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
