The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
From local explanations to global understanding with explainable AI for trees
Open accessAbstract
Tree-based machine learning models such as random forests, decision trees, and gradient boosted trees are popular non-linear predictive models, yet comparatively little attention has been paid to explaining their predictions. Here, we improve the interpretability of tree-based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non-linear mortality risk factors in the US population, ii) highlight distinct population sub-groups with shared risk characteristics, iii) identify non-linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“SHAP [6] popularized this for machine learning feature importance, with extensions to tree models [5].”
From AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation (2025) · cites this paper as [5]
In the citing paper
“The algorithm provides interpretable explanations suitable for regression and classification of models with tree structure applied to tabular data [22,39].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [39]
In the citing paper
“ML and AI techniques, such as social network analysis and agent-based modeling, can model the complex interactions between individuals and their social environments to identify key leverage points for interventions that promote health and well-being across communities [40, 47].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Self-citation"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623
Shapley, L. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511528446.003 · Cited by 10,698
Heart Disease and Stroke Statistics—2016 Update: A Report From the American Heart Association
Mozaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., Das, S. R., de Ferranti, S., Després, J.-P., Fullerton, H. J., et al. (2016) · Circulation · doi:10.1161/cir.0000000000000350 · Cited by 6,596
A Randomized Trial of Intensive versus Standard Blood-Pressure Control
The SPRINT Research Group (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1511939 · Cited by 6,325
Heart Disease and Stroke Statistics—2015 Update: A Report From the American Heart Association
Mozaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., de Ferranti, S., Després, J.-P., Fullerton, H. J., Howard, V. J., et al. (2015) · Circulation · doi:10.1161/cir.0000000000000152 · Cited by 5,797
Repeated observation of breast tumor subtypes in independent gene expression data sets
Sørlie, T., Tibshirani, R. J., Parker, J. S., Hastie, T. J., Marron, J. S., Nobel, A. B., Deng, S., Johnsen, H., Pesich, R., Geisler, S. B., et al. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0932692100 · Cited by 5,508
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Variable selection using random forests
Genuer, R., Poggi, J.-M., & Tuleau-Malot, C. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2010.03.014 · Cited by 2,619
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“We show that combining many local explanations lets us represent global structure while retaining local faithfulness [14] to the original model, which produces detailed and accurate representations of model behavior.”
BackgroundExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Self-citationUnderstanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,918
On the interpretation of weight vectors of linear models in multivariate neuroimaging
Haufe, S., Meinecke, F. C., Görgen, K., Dähne, S., Haynes, J., Blankertz, B., & Bießmann, F. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.067 · Cited by 1,423
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Clinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920
Monotonic solutions of cooperative games
Young, H. P. (1985) · International Journal of Game Theory · doi:10.1007/bf01769885 · Cited by 914
Not Just a Black Box: Learning Important Features Through Propagating Activation DifferencesPreprint
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 853
Tangri, N., Grams, M. E., Levey, A. S., Coresh, J., Appel, L. J., Astor, B. C., Chodick, G., Collins, A. J., Djurdjev, O., Elley, C. R., et al. (2016) · JAMA · doi:10.1001/jama.2015.18202 · Cited by 737
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728
Towards better understanding of gradient-based attribution methods for Deep Neural NetworksPreprint
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 634
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
A Benchmark for Interpretability Methods in Deep Neural NetworksPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10758 · Cited by 531
Hastie, T., Friedman, J., & Tibshirani, R. J. (2001) · Springer series in statistics · doi:10.1007/978-0-387-21606-5_1 · Cited by 480
Screening large-scale association study data: exploiting interactions using random forests
Lunetta, K. L., Hayward, B., Segal, J., & Eerdewegh, P. V. (2004) · BMC Genetics · doi:10.1186/1471-2156-5-32 · Cited by 478
The many Shapley values for model explanationPreprint
Sundararajan, M., & Najmi, A. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1908.08474 · Cited by 415
Plan and operation of the NHANES I Epidemiologic Followup Study, 1986.
Cox, C. S., Rothwell, S. T., Madans, J. H., Finucane, F. F., Freid, V. M., Kleinman, J. C., Barbano, H. E., & Feldman, J. J. (1987) · PubMed · Cited by 311
A random forest approach to the detection of epistatic interactions in case-control studies
Jiang, R., Tang, W., Wu, X., & Fu, W. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-s1-s65 · Cited by 310
The identification of Parkinson's disease subtypes using cluster analysis: A systematic review
van Rooden, S. M., Heiser, W. J., Kok, J. N., Verbaan, D., van Hilten, J. J., & Marinus, J. (2010) · Movement Disorders · doi:10.1002/mds.23116 · Cited by 238
Feature relevance quantification in explainable AI: A causal problem
Janzing, D., Minorics, L., & Blöbaum, P. (2019) · arXiv · doi:10.48550/arxiv.1910.13413 · Cited by 207
Clever Hans (the horse of Mr. Von Osten) a contribution to experimental animal and human psychology,
Pfungst, O., & Rahn, C. L. (1911) · Henry Holt and Company eBooks · doi:10.5962/bhl.title.56164 · Cited by 196
Axiomatic characterizations of probabilistic and cardinal-probabilistic interaction indices
Fujimoto, K., Kojadinovic, I., & Marichal, J. (2005) · Games and Economic Behavior · doi:10.1016/j.geb.2005.03.002 · Cited by 170
Model Agnostic Supervised Local Explanations
Plumb, G., Molitor, D., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1807.02910 · Cited by 149
NP-completeness for calculating power indices of weighted majority games
Matsui, Y., & Matsui, T. (2001) · Theoretical Computer Science · doi:10.1016/s0304-3975(00)00251-6 · Cited by 118
Association between Monocyte Count and Risk of Incident CKD and Progression to ESRD
Bowe, B. C., Xie, Y., Xian, H., Li, T., & Al‐Aly, Z. (2017) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.09710916 · Cited by 78
The Judicial Demand for Explainable Artificial IntelligencePreprint
Deeks, A. S. (2019) · SSRN Electronic Journal · Cited by 74
Influence-Directed Explanations for Deep Convolutional Networks
Leino, K. R. M., Sen, S., Datta, A., Fredrikson, M., & Li, L. (2018) · IEEE International Test Conference (ITC) · doi:10.1109/test.2018.8624792 · Cited by 70
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54
Fan, F., Jia, J., Li, J., Huo, Y., & Zhang, Y. (2017) · BMC Nephrology · doi:10.1186/s12882-017-0608-4 · Cited by 41
Explainable machine learning predictions to help anesthesiologists prevent hypoxemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2017) · doi:10.1101/206540 · Cited by 34
Self-citation
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“sex of the participant (figure is adapted from [115]).”
BackgroundMethodInfluentialExplainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
An interpretable mortality prediction model for COVID-19 patients
Li, Y., Zhang, H., Gonçalves, J. M., Xiao, Y., Wang, M., Guo, Y., Sun, C., Tang, X., Jing, L., Zhang, M., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0180-7 · Cited by 1,328
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,400
Metadata
- Authors
- Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I.
- Affiliations
- Microsoft (United States) (US)
- University of Washington (US)
- Corresponding author
- Lee, Su-In
- ORCID
- Venue
- Nature Machine Intelligence
- Published
- 2020-01-17
- Volume
- 2
- Issue
- 1
- Pages
- 56-67
- ISSN
- 2522-5839 (ISSN-L)
- PMC
- PMC7326367
- OpenAlex
- W2999615587
- Cited by
- 10,551
- References
- 38
- Citations by year
- 2026: 3607 · 2025: 2330 · 2024: 1650 · 2023: 1197 · 2022: 959 · 2021: 614 · 2020: 187
- FWCI
- 291.51
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Science Foundation (DGE-1256082, DBI-1759487, DBI-1552309, DBI-1355899)
- American Cancer Society (127332-RSG-15-097-01-TBG)
- U.S. Department of Health & Human Services | National Institutes of Health (R35 GM 128638, R01 NIA AG 061132)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- explainable AI, tree-based models, local explanations, mortality risk, chronic kidney disease, model monitoring, machine learning in medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
