The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Permutation importance: a corrected feature importance measure
Open accessAbstract
Motivation: In life sciences, interpretability of machine learning models is as important as their prediction accuracy. Linear models are probably the most frequently used methods for assessing feature relevance, despite their relative inflexibility. However, in the past years effective estimators of feature relevance have been derived for highly complex or non-parametric models such as support vector machines and RandomForest (RF) models. Recently, it has been observed that RF models are biased in such a way that categorical variables with a large number of categories are preferred. Results: In this work, we introduce a heuristic for normalizing feature importance measures that can correct the feature importance bias. The method is based on repeated permutations of the outcome vector for estimating the distribution of measured importance for each variable in a non-informative setting. The P-value of the observed importance provides a corrected measure of feature importance. We apply our method to simulated data and demonstrate that (i) non-informative predictors do not receive significant P-values, (ii) informative variables can successfully be recovered among non-informative variables and (iii) P-values computed with permutation importance (PIMP) are very helpful for deciding the significance of variables, and therefore improve model interpretability. Furthermore, PIMP was used to correct RF-based importance measures for two real-world case studies. We propose an improved RF model that uses the significant variables with respect to the PIMP measure and show that its prediction accuracy is superior to that of other existing models. Availability: R code for the method presented in this article is available at http://www.mpi-inf.mpg.de/∼altmann/download/PIMP.R Contact: altmann@mpi-inf.mpg.de, laura.tolosi@mpi-inf.mpg.de Supplementary information: Supplementary data are available at Bioinformatics online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
Cited by
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167
A comparison of random forest variable selection methods for classification prediction modeling
Speiser, J. L., Miller, M. E., Tooze, J., & Ip, E. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.05.028 · Cited by 1,561
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268
Wang, J.-X., Wu, J.-L., & Xiao, H. (2017) · Physical Review Fluids · doi:10.1103/physrevfluids.2.034603 · Cited by 737
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
Cell-of-origin chromatin organization shapes the mutational landscape of cancer
Polak, P., Karlić, R., Koren, A., Thurman, R., Sandstrom, R., Lawrence, M. S., Reynolds, A., Rynes, E., Vlahoviček, K., Stamatoyannopoulos, J. A., et al. (2015) · Nature · doi:10.1038/nature14221 · Cited by 557
Opening the black box: the promise and limitations of explainable machine learning in cardiology
Petch, J., Di, S., & Nelson, W. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.09.004 · Cited by 552
Coupling machine learning and crop modeling improves crop yield prediction in the US corn belt
Shahhosseini, M., Hu, G., Huber, I., & Archontoulis, S. V. (2021) · Scientific Reports · doi:10.1038/s41598-020-80820-1 · Cited by 452
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
Zhang, Z., Hamadi, H. A., Damiani, E., Yeun, C. Y., & Taher, F. (2022) · IEEE Access · doi:10.1109/access.2022.3204051 · Cited by 450
Artificial intelligence for aging and longevity research: Recent advances and perspectives
Zhavoronkov, A., Mamoshina, P., Vanhaelen, Q., Scheibye-Knudsen, M., Moskalev, A., & Aliper, A. (2019) · Ageing Research Reviews · doi:10.1016/j.arr.2018.11.003 · Cited by 246
“…prior to the training of the model, the predictions obtained from AI algorithms can be better interpreted by using methods such as the permutation feature importance (PFI) technique (Altmann et al., 2010) which allows evaluating the relative importance of features to DNN prediction accuracy.”
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211
Biohorology and biomarkers of aging: Current state-of-the-art, challenges and opportunities
Galkin, F., Mamoshina, P., Aliper, A., de Magalhães, J. P., Gladyshev, V. N., & Zhavoronkov, A. (2020) · Ageing Research Reviews · doi:10.1016/j.arr.2020.101050 · Cited by 189
The ConTraSt database for analysing and comparing empirical studies of consciousness theories
Yaron, I., Melloni, L., Pitts, M., & Mudrik, L. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01284-5 · Cited by 147
Opoku Asare, K., Terhorst, Y., Vega, J., Peltonen, E., Lagerspetz, E., & Ferreira, D. (2021) · JMIR mHealth and uHealth · doi:10.2196/26540 · Cited by 145
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143
Z-Inspection ® : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134
Ukwaththa, J., Herath, S., & Meddage, D. P. P. (2024) · Materials Today Communications · doi:10.1016/j.mtcomm.2024.110294 · Cited by 118
Jaxa-Rozen, M., & Kwakkel, J. (2018) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2018.06.011 · Cited by 91
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Johnson, M., Albizri, A., & Harfouche, A. (2023) · Information Systems Frontiers · doi:10.1007/s10796-021-10137-5 · Cited by 88
Conceptual challenges for interpretable machine learning
Watson, D. S. (2022) · Synthese · doi:10.1007/s11229-022-03485-5 · Cited by 87
Prosperi, M. C. F., Belgrave, D., Buchan, I., Simpson, A., & Custovic, A. (2014) · Pediatric Allergy and Immunology · doi:10.1111/pai.12139 · Cited by 60
Gut microbiome strain-sharing within isolated village social networks
Beghini, F., Pullman, J., Alexander, M., Shridhar, S. V., Prinster, D., Singh, A., Matute Juárez, R., Airoldi, E. M., Brito, I. L., & Christakis, N. A. (2025) · Nature · doi:10.1038/s41586-024-08222-1 · Cited by 51
Explainability Methods for Identifying Root-Cause of SLA Violation Prediction in 5G Network
Terra, A., Inam, R., Baskaran, S., Batista, P., Burdick, I., & Fersman, E. (2020) · GLOBECOM 2020 - 2020 IEEE Global Communications Conference · doi:10.1109/globecom42002.2020.9322496 · Cited by 50
A data-driven approach for predicting the impact of drugs on the human microbiome
Algavi, Y. M., & Borenstein, E. (2023) · Nature Communications · doi:10.1038/s41467-023-39264-0 · Cited by 54
Explainable AI in Industry: Practical Challenges and Lessons Learned
Gade, K., Geyik, S., Kenthapadi, K., Mithal, V., & Taly, A. (2020) · Companion Proceedings of the Web Conference 2020 · doi:10.1145/3366424.3383110 · Cited by 46
Cho, S., Nam, H. J., Shi, C., Kim, C. Y., Byun, S.-H., Agno, K.-C., Lee, B. C., Xiao, J., Sim, J. Y., & Jeong, J.-W. (2023) · Biosensors and Bioelectronics · doi:10.1016/j.bios.2022.115018 · Cited by 46
Jabbour, S., Fouhey, D., Kazerooni, E., Wiens, J., & Sjoding, M. W. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac030 · Cited by 39
Marrying Fairness and Explainability in Supervised Learning
Grabowicz, P. A., Perello, N., & Mishra, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533236 · Cited by 37
Lin, J., Zhao, D., Lu, S., Li, R., Xu, X., Wang, Z., Li, W., Ji, Y., Zhang, C., Shi, L., et al. (2025) · ACS Nano · doi:10.1021/acsnano.5c04880 · Cited by 36
Decay radius of climate decision for solar panels in the city of Fresno, USA
Barton-Henry, K., Wenz, L., & Levermann, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-87714-w · Cited by 28
Peña-Bautista, C., Vigor, C., Galano, J.-M., Oger, C., Durand, T., Ferrer, I., Cuevas, A., López-Cuevas, R., Baquero, M., López-Nogueroles, M., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-50837-2 · Cited by 28
Qi, L., Zhong, F., Chen, Y., Mao, S., Yan, Z., & Ma, Y. (2020) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2019.12.004 · Cited by 26
Personalised analytics for rare disease diagnostics
Anderson, D., Baynam, G., Blackwell, J. M., & Lassmann, T. (2019) · Nature Communications · doi:10.1038/s41467-019-13345-5 · Cited by 24
Martínez-Pascual, D., Catalán, J. M., Blanco-Ivorra, A., Sanchís, M., Arán-Ais, F., & García-Aracil, N. (2023) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2023.1199459 · Cited by 21
Yaron, I., Melloni, L., Pitts, M., & Mudrik, L. (2021) · doi:10.1101/2021.06.10.447863 · Cited by 21
What’s all that racket! Soundscapes, phenology, and biodiversity in estuaries
Monczak, A., McKinney, B., Mueller, C., & Montie, E. W. (2020) · PLOS ONE · doi:10.1371/journal.pone.0236874 · Cited by 17
Pan, J., Fan, Z., Smith, G. E., Guo, Y., Bian, J., & Xu, J. (2025) · Alzheimer's & Dementia · doi:10.1002/alz.70128 · Cited by 19
An interpretable deep learning framework identifies proteomic drivers of Alzheimer’s disease
Panizza, E., & Cerione, R. A. (2024) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2024.1379984 · Cited by 11
Prediction of Protein B-Factor Profiles Based on Bidirectional Long Short-Term Memory Network
Wang, Q., Xiao, X., Miao, Z., Zhang, X., Yang, D., Jiang, B., & Liu, M. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3564284 · Cited by 5
Nomological Deductive Reasoning for Trustworthy, Human-Readable, and Actionable AI Outputs
Hakizimana, G., & Ledezma Espino, A. (2025) · Algorithms · doi:10.3390/a18060306 · Cited by 2
Metadata
- Authors
- Altmann, A., Toloşi, L., Sander, O., & Lengauer, T.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-04-12
- Volume
- 26
- Issue
- 10
- Pages
- 1340-1347
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W2102636708
- Cited by
- 2,843
- References
- 3
- FWCI
- 6.12
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
