From local explanations to global understanding with explainable AI for trees
Open accessAbstract
Tree-based machine learning models such as random forests, decision trees, and gradient boosted trees are popular non-linear predictive models, yet comparatively little attention has been paid to explaining their predictions. Here, we improve the interpretability of tree-based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non-linear mortality risk factors in the US population, ii) highlight distinct population sub-groups with shared risk characteristics, iii) identify non-linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SHAP popularized Shapley values for machine learning feature importance, with extensions to tree models.
“SHAP [6] popularized this for machine learning feature importance, with extensions to tree models [5].”
From AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation (2025) · cites this paper as [5]
Tree SHAP provides interpretable explanations suitable for regression and classification of models with tree structure applied to tabular data.
“The algorithm provides interpretable explanations suitable for regression and classification of models with tree structure applied to tabular data [22,39].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [39]
ML and AI techniques, such as social network analysis and agent-based modeling, can model the complex interactions between individuals and their social environments to identify key leverage points for interventions that promote health and well-being across communities.
“ML and AI techniques, such as social network analysis and agent-based modeling, can model the complex interactions between individuals and their social environments to identify key leverage points for interventions that promote health and well-being across communities [40, 47].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Self-citation"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698 · see citing works
Heart Disease and Stroke Statistics—2016 Update: A Report From the American Heart Association
Mozaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., Das, S. R., de Ferranti, S., Després, J.-P., Fullerton, H. J., et al. (2016) · Circulation · doi:10.1161/cir.0000000000000350 · Cited by 6,596 · see citing works
A Randomized Trial of Intensive versus Standard Blood-Pressure Control
The SPRINT Research Group (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1511939 · Cited by 6,325 · see citing works
Heart Disease and Stroke Statistics—2015 Update: A Report From the American Heart Association
Mozaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., de Ferranti, S., Després, J.-P., Fullerton, H. J., Howard, V. J., et al. (2015) · Circulation · doi:10.1161/cir.0000000000000152 · Cited by 5,797 · see citing works
Repeated observation of breast tumor subtypes in independent gene expression data sets
Sørlie, T., Tibshirani, R. J., Parker, J. S., Hastie, T. J., Marron, J. S., Nobel, A. B., Deng, S., Johnsen, H., Pesich, R., Geisler, S. B., et al. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0932692100 · Cited by 5,428 · see citing works
Variable selection using random forests
Genuer, R., Poggi, J.-M., & Tuleau-Malot, C. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2010.03.014 · Cited by 2,619 · see citing works
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1× · see citing works
“We show that combining many local explanations lets us represent global structure while retaining local faithfulness [14] to the original model, which produces detailed and accurate representations of model behavior.”
BackgroundExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · see citing works
Self-citationUnderstanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
On the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F. C., Görgen, K., Dähne, S., Haynes, J., Blankertz, B., & Bießmann, F. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,367 · see citing works
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047 · see citing works
Clinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920 · see citing works
Monotonic solutions of cooperative games
Young, H. P. (1985) · International Journal of Game Theory · doi:10.1007/bf01769885 · Cited by 914 · see citing works
How to Explain Individual Classification DecisionsPreprint
Baehrens, D., Schroeter, T. S., Stefan Harmeling, B., Kawanabe, M., Hansen, K., & Mueller, K.-R. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0912.1128 · Cited by 743 · see citing works
Tangri, N., Grams, M. E., Levey, A. S., Coresh, J., Appel, L. J., Astor, B. C., Chodick, G., Collins, A. J., Djurdjev, O., Elley, C. R., et al. (2016) · JAMA · doi:10.1001/jama.2015.18202 · Cited by 737 · see citing works
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · see citing works
Not Just a Black Box: Learning Important Features Through Propagating Activation DifferencesPreprint
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 704 · see citing works
Towards better understanding of gradient-based attribution methods for Deep Neural NetworksPreprint
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 634 · see citing works
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580 · see citing works
Hastie, T., Friedman, J., & Tibshirani, R. J. (2001) · Springer series in statistics · doi:10.1007/978-0-387-21606-5_1 · Cited by 480 · see citing works
Screening large-scale association study data: exploiting interactions using random forests
Lunetta, K. L., Hayward, B., Segal, J., & Eerdewegh, P. V. (2004) · BMC Genetics · doi:10.1186/1471-2156-5-32 · Cited by 464 · see citing works
A Benchmark for Interpretability Methods in Deep Neural NetworksPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 438 · see citing works
The many Shapley values for model explanationPreprint
Sundararajan, M., & Najmi, A. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1908.08474 · Cited by 415 · see citing works
Plan and operation of the NHANES I Epidemiologic Followup Study, 1986.
Cox, C. S., Rothwell, S. T., Madans, J. H., Finucane, F. F., Freid, V. M., Kleinman, J. C., Barbano, H. E., & Feldman, J. J. (1987) · PubMed · Cited by 311 · see citing works
A random forest approach to the detection of epistatic interactions in case-control studies
Jiang, R., Tang, W., Wu, X., & Fu, W. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-s1-s65 · Cited by 306 · see citing works
The identification of Parkinson's disease subtypes using cluster analysis: A systematic review
van Rooden, S. M., Heiser, W. J., Kok, J. N., Verbaan, D., van Hilten, J. J., & Marinus, J. (2010) · Movement Disorders · doi:10.1002/mds.23116 · Cited by 238 · see citing works
Feature relevance quantification in explainable AI: A causal problem
Janzing, D., Minorics, L., & Blöbaum, P. (2019) · arXiv · doi:10.48550/arxiv.1910.13413 · Cited by 207 · see citing works
Clever Hans (the horse of Mr. Von Osten) a contribution to experimental animal and human psychology,
Pfungst, O., & Rahn, C. L. (1911) · Henry Holt and Company eBooks · doi:10.5962/bhl.title.56164 · Cited by 189 · see citing works
Axiomatic characterizations of probabilistic and cardinal-probabilistic interaction indices
Fujimoto, K., Kojadinovic, I., & Marichal, J. (2005) · Games and Economic Behavior · doi:10.1016/j.geb.2005.03.002 · Cited by 170 · see citing works
Model Agnostic Supervised Local Explanations
Plumb, G., Molitor, D., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1807.02910 · Cited by 149 · see citing works
NP-completeness for calculating power indices of weighted majority games
Matsui, Y., & Matsui, T. (2001) · Theoretical Computer Science · doi:10.1016/s0304-3975(00)00251-6 · Cited by 121 · see citing works
Association between Monocyte Count and Risk of Incident CKD and Progression to ESRD
Bowe, B. C., Xie, Y., Xian, H., Li, T., & Al‐Aly, Z. (2017) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.09710916 · Cited by 78 · see citing works
The Judicial Demand for Explainable Artificial IntelligencePreprint
Deeks, A. S. (2019) · SSRN Electronic Journal · Cited by 74 · see citing works
Influence-Directed Explanations for Deep Convolutional Networks
Leino, K. R. M., Sen, S., Datta, A., Fredrikson, M., & Li, L. (2018) · IEEE International Test Conference (ITC) · doi:10.1109/test.2018.8624792 · Cited by 60 · see citing works
Fan, F., Jia, J., Li, J., Huo, Y., & Zhang, Y. (2017) · BMC Nephrology · doi:10.1186/s12882-017-0608-4 · Cited by 41 · see citing works
Explainable machine learning predictions to help anesthesiologists prevent hypoxemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2017) · doi:10.1101/206540 · Cited by 34 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [2]Friedman J, Hastie T & Tibshirani R The elements of statistical learning (Springer series in statistics Springer, Berlin, 2001).
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
“sex of the participant (figure is adapted from [115]).”
BackgroundMethodInfluentialExplainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386 · see citing works
An interpretable mortality prediction model for COVID-19 patients
Li, Y., Zhang, H., Gonçalves, J. M., Xiao, Y., Wang, M., Guo, Y., Sun, C., Tang, X., Jing, L., Zhang, M., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0180-7 · Cited by 1,304 · see citing works
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,188 · see citing works
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979 · see citing works
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932 · see citing works
Bobadilla, A. V. P., Schmitt, V., Maier, C. S., Mensing, S., & Stodtmann, S. (2024) · Clinical and Translational Science · doi:10.1111/cts.70056 · Cited by 931 · see citing works
Viral epitope profiling of COVID-19 patients reveals cross-reactivity and correlates of severity
Shrock, E., Fujimura, E., Kula, T., Timms, R. T., Lee, I.-H., Leng, Y., Robinson, M. L., Sie, B. M., Li, M. Z., Chen, Y., et al. (2020) · Science · doi:10.1126/science.abd4250 · Cited by 678 · see citing works
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717 · see citing works
Rodríguez-Pérez, R., & Bajorath, J. (2020) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-020-00314-0 · Cited by 718 · see citing works
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660 · see citing works
The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
Razavi, S., Jakeman, A. J., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Piano, S. L., Iwanaga, T., Becker, W., et al. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 651 · see citing works
Understanding the diversity of the metal-organic framework ecosystem
Moosavi, S. M., Nandy, A., Jablonka, K. M., Ongari, D., Janet, J. P., Boyd, P. G., Lee, Y., Smit, B., & Kulik, H. J. (2020) · Nature Communications · doi:10.1038/s41467-020-17755-8 · Cited by 622 · see citing works
Explainable Machine Learning in Credit Risk Management
Bussmann, N., Giudici, P., Marinelli, D., & Papenbrock, J. (2020) · Computational Economics · doi:10.1007/s10614-020-10042-0 · Cited by 635 · see citing works
From explanations to feature selection: assessing SHAP values as feature selection mechanism
Marcilio, W. E., & Eler, D. M. (2020) · 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi51738.2020.00053 · Cited by 504 · see citing works
Algorithms to estimate Shapley value feature attributions
Chen, H., Covert, I., Lundberg, S., & Lee, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00657-x · Cited by 510 · see citing works
Self-citationOn the Tractability of SHAP Explanations
den Broeck, G. V., Lykov, A., Schleich, M., & Suciu, D. M. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13283 · Cited by 501 · see citing works
Uçar, A., Karaköse, M., & Kırımça, N. (2024) · Applied Sciences · doi:10.3390/app14020898 · Cited by 453 · see citing works
Interpretable XGBoost-SHAP Machine-Learning Model for Shear Strength Prediction of Squat RC Walls
Feng, D., Wang, W., Mangalathu, S., & Taciroğlu, E. (2021) · Journal of Structural Engineering · doi:10.1061/(asce)st.1943-541x.0003115 · Cited by 457 · see citing works
Explainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_2 · Cited by 480 · see citing works
Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448 · see citing works
CatBoost model and artificial intelligence techniques for corporate failure prediction
Jabeur, S. B., Gharib, C., Mefteh‐Wali, S., & Arfi, W. B. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120658 · Cited by 402 · see citing works
Deterministic Local Interpretable Model-Agnostic Explanations for Stable Explainability
Zafar, M. R., & Khan, N. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030027 · Cited by 408 · see citing works
Kavzoğlu, T., & Teke, A. (2022) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-022-06560-8 · Cited by 379 · see citing works
On scientific understanding with artificial intelligence
Krenn, M., Pollice, R., Guo, S. Y., Aldeghi, M., Cervera-Lierta, A., Friederich, P., Gomes, G., Häse, F., Jinich, A., Nigam, A., et al. (2022) · Nature Reviews Physics · doi:10.1038/s42254-022-00518-3 · Cited by 396 · see citing works
An open-source, high-performance tool for automated sleep staging
Vallat, R., & Walker, M. P. (2021) · eLife · doi:10.7554/elife.70092 · Cited by 366 · see citing works
Argentieri, M. A., Xiao, S., Bennett, D., Winchester, L. M., Nevado‐Holgado, A. J., Ghose, U., Albukhari, A., Yao, P., Mazidi, M., Lv, J., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03164-7 · Cited by 378 · see citing works
Wang, K., Tian, J., Zheng, C., Yang, H., Ren, J., Liu, Y., Han, Q., & Zhang, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104813 · Cited by 361 · see citing works
Explanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355 · see citing works
Tran, V. Q., Dang, V. Q., & Ho, L. S. (2022) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2022.126578 · Cited by 351 · see citing works
Hossain, M. S., Muhammad, G., & Guizani, N. (2020) · IEEE Network · doi:10.1109/mnet.011.2000458 · Cited by 347 · see citing works
Shapley values for feature selection: the good, the bad, and the axioms
Fryer, D., Strumke, I., & Nguyen, H. (2021) · IEEE Access · doi:10.1109/access.2021.3119110 · Cited by 347 · see citing works
Moncada‐Torres, A., van Maaren, M. C., Hendriks, M. P., Siesling, S., & Geleijnse, G. (2021) · Scientific Reports · doi:10.1038/s41598-021-86327-7 · Cited by 340 · see citing works
Wen, X. L., Xie, Y., Wu, L., & Jiang, L. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106261 · Cited by 327 · see citing works
Senoner, J., Netland, T., & Feuerriegel, S. (2022) · Management Science · doi:10.1287/mnsc.2021.4190 · Cited by 317 · see citing works
Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 339 · see citing works
Wang, D., Thunéll, S., Lindberg, U., Jiang, L., Trygg, J., & Tysklind, M. (2021) · Journal of Environmental Management · doi:10.1016/j.jenvman.2021.113941 · Cited by 302 · see citing works
Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00343-w · Cited by 300 · see citing works
Self-citationHaimovich, A. D., Ravindra, N. G., Stoytchev, S., Young, H. P., Wilson, F. P., van Dijk, D., Schulz, W. L., & Taylor, R. A. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.07.022 · Cited by 289 · see citing works
Revealing Drivers of Haze Pollution by Explainable Machine Learning
Hou, L., Dai, Q., Song, C., Liu, B., Guo, F., Dai, T., Li, L., Liu, B., Bi, X., Zhang, Y., et al. (2022) · Environmental Science & Technology Letters · doi:10.1021/acs.estlett.1c00865 · Cited by 289 · see citing works
de Almeida, B. P., Reiter, F., Pagani, M., & Stark, A. (2021) · doi:10.1101/2021.10.05.463203 · Cited by 284
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302 · see citing works
Explaining a series of models by propagating Shapley values
Chen, H., Lundberg, S., & Lee, S. (2022) · Nature Communications · doi:10.1038/s41467-022-31384-3 · Cited by 289 · see citing works
Self-citationde Almeida, B. P., Reiter, F., Pagani, M., & Stark, A. (2022) · Nature Genetics · doi:10.1038/s41588-022-01048-5 · Cited by 283 · see citing works
A microbiome-dependent gut–brain pathway regulates motivation for exercise
Dohnalová, L., Lundgren, P., Carty, J. R. E., Goldstein, N., Wenski, S. L., Nanudorn, P., Thiengmag, S., Huang, K.-P., Litichevskiy, L., Descamps, H. C., et al. (2022) · Nature · doi:10.1038/s41586-022-05525-z · Cited by 292 · see citing works
Xiao, L., Lo, S., Liu, J., Zhou, J., & Li, Q. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103063 · Cited by 286 · see citing works
Predicting the efficiency of prime editing guide RNAs in human cells
Kim, H. K., Yu, G., Park, J., Min, S., Lee, S., Yoon, S., & Kim, S. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-0677-y · Cited by 272 · see citing works
Self-citationAdvancements and challenges of digital twins in industry
Tao, F., Zhang, H., & Zhang, C. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00603-w · Cited by 286 · see citing works
Interpretable machine learning with an ensemble of gradient boosting machines
Konstantinov, A. V., & Utkin, L. V. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106993 · Cited by 287 · see citing works
Interpretable tree-based ensemble model for predicting beach water quality
Li, L., Qiao, J., Yu, G., Wang, L., Li, H.-Y., Liao, C., & Zhu, Z. (2022) · Water Research · doi:10.1016/j.watres.2022.118078 · Cited by 273 · see citing works
A Holistic Approach to the Design of Artificial Intelligence (AI) Education for K-12 Schools
Chiu, T. K. F. (2021) · TechTrends · doi:10.1007/s11528-021-00637-1 · Cited by 271 · see citing works
Muhammad, D., & Bendechache, M. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.08.005 · Cited by 310 · see citing works
Artificial intelligence-powered electronic skin
Xu, C., Solomon, S. A., & Gao, W. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00760-z · Cited by 259 · see citing works
Oukawa, G. Y., Krecl, P., & Targino, A. C. (2022) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.152836 · Cited by 248 · see citing works
Machine learning models to accelerate the design of polymeric long-acting injectables
Bannigan, P., Bao, Z., Hickman, R. J., Aldeghi, M., Häse, F., Aspuru‐Guzik, A., & Allen, C. J. (2023) · Nature Communications · doi:10.1038/s41467-022-35343-w · Cited by 246 · see citing works
Machine learning enables interpretable discovery of innovative polymers for gas separation membranes
Yang, J., Tao, L., He, J., McCutcheon, J. R., & Li, Y. (2022) · Science Advances · doi:10.1126/sciadv.abn9545 · Cited by 244 · see citing works
Decision trees: from efficient prediction to responsible AI
Blockeel, H., Devos, L., Frénay, B., Nanfack, G., & Nijssen, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1124553 · Cited by 267 · see citing works
Machine learning model to predict mental health crises from electronic health records
Garriga, R., Mas, J., Abraha, S., Nolan, J., Harrison, O., Tadros, G., & Matic, A. (2022) · Nature Medicine · doi:10.1038/s41591-022-01811-5 · Cited by 229 · see citing works
Nemesure, M. D., Heinz, M. V., Huang, R., & Jacobson, N. C. (2021) · Scientific Reports · doi:10.1038/s41598-021-81368-4 · Cited by 232 · see citing works
Wang, F., Wang, Y., Zhang, K., Hu, M., Weng, Q., & Zhang, H. J. (2021) · Environmental Research · doi:10.1016/j.envres.2021.111660 · Cited by 237 · see citing works
Vaid, A., Somani, S. S., Russak, A. J., Freitas, J. K. D., Chaudhry, F. F., Paranjpe, I., Johnson, K. W., Lee, S. J., Miotto, R., Zhao, S., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/24018 · Cited by 230 · see citing works
Self-citationElectricity price forecasting on the day-ahead market using machine learning
Tschora, L., Pierre, E., Plantevit, M., & Robardet, C. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.118752 · Cited by 233 · see citing works
A novel construction cost prediction model using hybrid natural and light gradient boosting
Chakraborty, D., Elhegazy, H. M., Elzarka, H. M., & Gutierrez, L. (2020) · Advanced Engineering Informatics · doi:10.1016/j.aei.2020.101201 · Cited by 227 · see citing works
Wang, F., Huang, S., Gao, R., Zhou, Y., Lai, C., Li, Z., Xian, W., Qian, X., Li, Z., Huang, Y., et al. (2020) · Cell Discovery · doi:10.1038/s41421-020-00231-4 · Cited by 222 · see citing works
Bello, M. D., Lee, H., Goyal, A., & Gore, J. (2021) · Nature Ecology & Evolution · doi:10.1038/s41559-021-01535-8 · Cited by 225 · see citing works
Dynamics of particle network in composite battery cathodes
Li, J., Sharma, N., Jiang, Z., Yang, Y., Monaco, F., Xu, Z., Hou, D., Ratner, D. F., Pianetta, P., Cloetens, P., et al. (2022) · Science · doi:10.1126/science.abm8962 · Cited by 220 · see citing works
Shapley value: from cooperative game to explainable artificial intelligence
Li, M., Sun, H., Huang, Y., & Chen, H. (2024) · Autonomous Intelligent Systems · doi:10.1007/s43684-023-00060-8 · Cited by 242 · see citing works
Self-citationExplainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216 · see citing works
Gut microbiota, inflammation, and molecular signatures of host response to infection
Gou, W., Fu, Y., Yue, L., Chen, G., Cai, X., Shuai, M., Xu, F., Yi, X., Chen, H., Zhu, Y., et al. (2021) · Journal of genetics and genomics/Journal of Genetics and Genomics · doi:10.1016/j.jgg.2021.04.002 · Cited by 216 · see citing works
Self-citationUnderstanding Global Feature Contributions With Additive Importance MeasuresPreprint
Covert, I., Lundberg, S., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00668 · Cited by 237 · see citing works
Self-citationNilsson, S. R. O., Goodwin, N. L., Choong, J. J., Hwang, S., Wright, H. R., Norville, Z. C., Tong, X. F., Lin, D., Bentzley, B. S., Eshel, N., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.19.049452 · Cited by 217 · see citing works
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · see citing works
Prediction of the sequence-specific cleavage activity of Cas9 variants
Kim, N., Kim, H. K., Lee, S., Seo, J. H., Choi, J. W., Park, J., Min, S., Yoon, S., Cho, S.-R., & Kim, H. H. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-0537-9 · Cited by 198 · see citing works
Self-citationAbbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236 · see citing works
“In contrast, model-specific approaches such as Grad-CAM, attention mechanisms, and Integrated Gradients are tailored to deep neural network architectures and require access to gradients or attention weights [106].”
InfluentialSelf-citationA Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 235 · see citing works
Understanding deep learning in land use classification based on Sentinel-2 time series
Campos‐Taberner, M., Garcı́a-Haro, F. J., Martínez, B., Izquierdo‐Verdiguier, E., Atzberger, C., Camps‐Valls, G., & Navarro, M. A. G. (2020) · Scientific Reports · doi:10.1038/s41598-020-74215-5 · Cited by 207 · see citing works
Zhou, F., Yang, X. J., & de Winter, J. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3069776 · Cited by 206 · see citing works
Zhou, X., Wen, H., Li, Z., Zhang, H., & Zhang, W. (2022) · Geocarto International · doi:10.1080/10106049.2022.2076928 · Cited by 204 · see citing works
Post-hoc vs ante-hoc explanations: xAI design guidelines for data scientists
Retzlaff, C. O., Angerschmid, A., Saranti, A., Schneeberger, D. M., Röttger, R., Müller, H., & Holzinger, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101243 · Cited by 206 · see citing works
Goodwin, N. L., Choong, J. J., Hwang, S., Pitts, K. M., Bloom, L., Islam, A., Zhang, Y. Y., Szelenyi, E. R., Tong, X., Newman, E. L., et al. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01649-9 · Cited by 204 · see citing works
Creativity and artificial intelligence—a student perspective
Marrone, R., Taddeo, V., & Hill, G. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10030065 · Cited by 208 · see citing works
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199 · see citing works
Machine Learning Solutions for Osteoporosis—A Review
Smets, J., Shevroja, E., Hügle, T., Leslie, W. D., & Hans, D. B. (2020) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.4292 · Cited by 202 · see citing works
Operational Research and Artificial Intelligence Methods in Banking
Doumpos, M., Zopounidis, C., Gounopoulos, D., Platanakis, E., & Zhang, W. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2022.04.027 · Cited by 235 · see citing works
Prevalence and drivers of abrupt vegetation shifts in global drylands
Berdugo, M., Gaitán, J., Delgado‐Baquerizo, M., Crowther, T. W., & Dakos, V. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123393119 · Cited by 199 · see citing works
Lewis, J. E., & Kemp, M. L. (2021) · Nature Communications · doi:10.1038/s41467-021-22989-1 · Cited by 187 · see citing works
Hussain, S., Mustafa, M. W., Jumani, T. A., Baloch, S. K., Alotaibi, H., Khan, I., & Khan, A. (2021) · Energy Reports · doi:10.1016/j.egyr.2021.07.008 · Cited by 190 · see citing works
Interpretable machine learning for imbalanced credit scoring datasets
Chen, Y., Calabrese, R., & Martin-Barragan, B. (2024) · European Journal of Operational Research · doi:10.1016/j.ejor.2023.06.036 · Cited by 190 · see citing works
Interpretable deep learning for automatic diagnosis of 12-lead electrocardiogram
Zhang, D., Yang, S., Yuan, X., & Zhang, P. (2021) · iScience · doi:10.1016/j.isci.2021.102373 · Cited by 192 · see citing works
Recent Applications of Explainable AI (XAI): A Systematic Literature Review
Saarela, M., & Podgorelec, V. (2024) · Applied Sciences · doi:10.3390/app14198884 · Cited by 219 · see citing works
COVID-19 mortality risk assessment: An international multi-center study
Bertsimas, D., Lukin, G., Mingardi, L., Nohadani, O., Orfanoudaki, A., Stellato, B., Wiberg, H., González-García, S., Parra-Calderón, C. L., Robinson, K. J., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243262 · Cited by 187 · see citing works
Ma, Z., Mei, G., & Cuomo, S. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106322 · Cited by 187 · see citing works
Heino, M., Kinnunen, P., Anderson, W., Ray, D. K., Puma, M. J., Varis, O., Siebert, S., & Kummu, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-29378-2 · Cited by 193 · see citing works
XGBoost-SHAP-based interpretable diagnostic framework for alzheimer’s disease
Yi, F., Yang, H., Chen, D., Qin, Y., Han, H., Cui, J., Bai, W., Ma, Y., Zhang, R., & Yu, H. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02238-9 · Cited by 188 · see citing works
Mitrentsis, G., & Lens, H. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118473 · Cited by 192 · see citing works
Comprehensive comparative study of multi-label classification methods
Bogatinovski, J., Todorovski, L., Džeroski, S., & Kocev, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117215 · Cited by 185 · see citing works
How Interpretable Machine Learning Can Benefit Process Understanding in the Geosciences
Jiang, S., Sweet, L., Blougouras, G., Brenning, A., Li, W., Reichstein, M., Denzler, J., Shangguan, W., Yu, G., Huang, F., et al. (2024) · Earth s Future · doi:10.1029/2024ef004540 · Cited by 197 · see citing works
- Authors
- Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I.
- Affiliations
- Microsoft (United States) (US)
- University of Washington (US)
- Corresponding author
- Lee, Su-In
- ORCID
- Venue
- Nature Machine Intelligence
- Published
- 2020-01-17
- Volume
- 2
- Issue
- 1
- Pages
- 56-67
- ISSN
- 2522-5839 (ISSN-L)
- PMC
- PMC7326367
- OpenAlex
- W2999615587
- Cited by
- 10,551
- References
- 38
- Citations by year
- 2026: 3607 · 2025: 2330 · 2024: 1650 · 2023: 1197 · 2022: 959 · 2021: 614 · 2020: 187 · 2019: 4 · 2017: 1 · 2016: 1
- FWCI
- 291.51
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Science Foundation (DGE-1256082, DBI-1759487, DBI-1552309, DBI-1355899)
- American Cancer Society (127332-RSG-15-097-01-TBG)
- U.S. Department of Health & Human Services | National Institutes of Health (R35 GM 128638, R01 NIA AG 061132)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, tree-based models, local explanations, mortality risk, chronic kidney disease, model monitoring, machine learning in medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
