Interpretable and explainable machine learning: A methods‐centric overview with concrete examples
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Interpretability and explainability are crucial for machine learning (ML) and statistical applications in medicine, economics, law, and natural sciences and form an essential principle for ML model design and development. Although interpretability and explainability have escaped a precise and universal definition, many models and techniques motivated by these properties have been developed over the last 30 years, with the focus currently shifting toward deep learning. We will consider concrete examples of state‐of‐the‐art, including specially tailored rule‐based, sparse, and additive classification models, interpretable representation learning, and methods for explaining black‐box models post hoc. The discussion will emphasize the need for and relevance of interpretability and explainability, the divide between them, and the inductive biases behind the presented “zoo” of interpretable models and explanation methods. This article is categorized under: Fundamental Concepts of Data and Knowledge > Explainable AI Technologies > Machine Learning Commercial, Legal, and Ethical Issues > Social Considerations
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Khanna and Tan (2020) apply the approach to a more advanced neural network architecture, namely, the long short-term memory (Hochreiter & Schmidhuber, 1997).”
MethodGreedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · cited 1×
“Peritonitis = generalized 6 are reminiscent of conventional medical scoring systems, such as APACHE (Knaus et al., 1985) or SOFA (Vincent et al., 1996).”
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 11×
“It is natural to question the utility of interpretable and explainable ML, especially given a widespread belief that a trade-off exists between accuracy and interpretability (Rudin, 2019; Semenova et al., 2019).”
BackgroundMethodInfluentialFrom local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · cited 1×
“There also exist model-specific implementations of SHAP, for example, for decision trees and gradient boosted decision trees (Lundberg et al., 2020).”
MethodBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 16×
“In practice, interpretability and explainability are typically most useful when auditing ML systems and confirming auxiliary desiderata beyond predictive performance (Carvalho et al., 2019; Doshi-Velez & Kim, 2017; Lipton, 2018).”
BackgroundMethodInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“They recruit human subjects on Amazon Mechanical Turk (Paolacci et al., 2010) and compare their ability to choose the best text classification model based on explanations provided by the proposed method versus baseline techniques.”
MethodThe EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001 · cited 2×
“It is worth mentioning that the GDPR does not prohibit black-box predictive models and that the right to an explanation is not legally binding (Carvalho et al., 2019; Wachter et al., 2017).”
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · cited 1×
“Similar results have been attained with variational autoencoders (VAE) (Higgins et al., 2016; Kingma & Welling, 2014; Kingma & Welling, 2019) by introducing statistical independence constraints on embedding dimensions via a factorizing prior distribution.”
ResultA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173 · cited 1×
“Another noteworthy extension is causal falling rule lists (F. Wang & Rudin, 2015a) that leverage FRLs to estimate treatment effects in the potential outcomes framework (Rubin, 2005).”
BackgroundDefinitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · cited 2×
“One such approach is concept bottleneck models (N. Kumar et al., 2009; Lampert et al., 2009), recently reexplored by Koh et al.”
BackgroundMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 15×
“Nevertheless, the notion of interpretability often depends on the domain of application (Rudin, 2019) and the target explainee (Carvalho et al., 2019), that is, the recipient of interpretations and explanations.”
BackgroundMethodInfluentialCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1×
“For instance, Ghassemi et al. (2021) overview healthcare applications of explainability techniques and their failure cases and argue that XAI is unlikely to address the real needs of practitioners.”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 12×
“Other terms that are synonymous with interpretability and also appear in the ML literature are “intelligibility” (Caruana et al., 2015; Lou et al., 2012) and “understandability” (Lipton, 2018). These concepts are often used interchangeably. Yet another term prevalent in the literature is “explainability,” giving rise…”
BackgroundMethodInfluentialSimon, N., Friedman, J., Hastie, T., & Tibshirani, R. (2013) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2012.681250 · Cited by 1,380 · cited 1×
“SPINNs are fully connected neural networks characterized by sparse weights in the input layer and are trained by minimizing the following loss function: (cid:1) , j refers to the j -th column of the input layer weight matrix; and Þ is the sparse group Lasso penalty (Simon et al., 2013).”
BackgroundDARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 1×
“Yet another term prevalent in the literature is “ explainability , ” giving rise to the direction of explainable artificial intelligence (XAI) (Gunning & Aha, 2019).”
BackgroundPredictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315 · cited 1×
“…include repeated incremental pruning to produce error reduction (RIPPER) (Cohen, 1995), which keeps the number of rules small, RuleFit (Friedman & Popescu, 2008), which induces rules from a sparse linear model with pairwise interactions, and falling rule lists (FRL) (F. Wang & Rudin, 2015b),…”
MethodTechniques for Interpretable Machine Learning
Du, M., Liu, N., & Hu, X. (2018) · arXiv · doi:10.48550/arxiv.1808.00033 · Cited by 1,261
Explaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233 · cited 3×
“Mothilal et al. (2020) extend the framework above to multiple diverse counterfactual explanations.”
BackgroundExplainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
An Introduction to Causal Inference
Pearl, J. (2010) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1203 · Cited by 755
Explainable Recommendation: A Survey and New Perspectives
Zhang, Y., & Chen, X. (2020) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000066 · Cited by 755
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · cited 2×
“Nauta et al. (2022) provide the most comprehensive survey to date of qualitative and quantitative methods.”
BackgroundIntelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · cited 5×
“In addition to the standard use of spline-based shape functions (Hastie & Tibshirani, 1986), Lou et al. (2012) consider single, bagged, boosted, and boosted bagged decision trees.”
BackgroundMethodInfluentialInterpretability of machine learning-based prediction models in healthcare
Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., & Cilar, L. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1379 · Cited by 541
Sequential regulatory activity prediction across chromosomes with convolutional neural networks
Kelley, D. R., Reshef, Y. A., Bileschi, M., Belanger, D., McLean, C. Y., & Snoek, J. (2018) · Genome Research · doi:10.1101/gr.227819.117 · Cited by 537 · cited 1×
“The use of attributions as a model diagnostic has become ubiquitous in ML applications (Arcadu et al., 2019; Kelley et al., 2018; Y. Liu et al., 2020; Parsa et al., 2020), especially for image data, since attributions can be readily visualized as a heat map and are helpful for understanding on…”
MethodRight for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · cited 2×
“Attribution priors (Erion et al., 2021; Ross et al., 2017) facilitate the use of post hoc explanations, such as EG, to make the neural network more interpretable, thus, building a connection with approaches described in Section 4.”
MethodInfluentialSoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502 · cited 2×
“When ML algorithms are incorporated into decision-making, for example, social, economic, or medical, and use sensitive personal data, we have to scrutinize their fairness (Barocas et al., 2019; Dignum, 2019) and privacy (Papernot et al., 2018).”
MethodThe Shapley Value in Machine Learning
Rozemberczki, B., Watson, L., Bayer, P., Yang, H.-T., Kiss, O., Nilsson, S., & Sarkar, R. (2022) · arXiv · doi:10.48550/arxiv.2202.05594 · Cited by 312 · cited 1×
“Rozemberczki et al. (2022) provide an in-depth overview of the cooperative game theory and numerous applications of the Shapley value in machine learning.”
BackgroundErion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00343-w · Cited by 300 · cited 3×
“Among more recent developments, Erion et al. (2021) introduce expected gradients (EG), which require fewer hyper-parameters than the measure in Equation (21): where D is the reference distribution, for example, x 0 could be sampled from the training dataset with replacement, and U 0,1 ð Þ is the…”
BackgroundMethodInfluentialCounterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning
Byrne, R. M. J. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/876 · Cited by 285
Algorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274
Causal interpretability for machine learning-problems, methods and evaluation
Moraffah, R., Karami, M., Guo, R., Raglin, A., & Liu, H. (2020) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3400051.3400058 · Cited by 238
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 2×
“I. E. Kumar et al. (2020) criticize Shapley-value-based explanations, such as described above, for their reliance on the additivity axiom (Hart, 1989) and lack of human-groundedness and contrastiveness.”
BackgroundAlgorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76
Müller, H., & Holzinger, A. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103546 · Cited by 30
Cited by
Shobayo, O., & Saatchi, R. (2025) · Diagnostics · doi:10.3390/diagnostics15091072 · Cited by 53
Buszydlik, A., Altmeyer, P., Dobbe, R., & Liem, C. C. S. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-02515-9_8 · Cited by 2
Pranata, Y., Mirwanti, R., Bahtiar, Y. A., & Purwanti, H. (2026) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s621469 · Cited by 1
“13,15,16 This distinction is especially consequential in the ICU, where the accountability demanded of clinical decisions calls for a deeper understanding than a mere summary of model patterns or trends.”
Edwards, T., Long, Q., & Shen, L. (2026) · doi:10.64898/2026.10.17.759869
Metadata
- Authors
- Marcinkevičs, R., & Vogt, J. E.
- Venue
- WIREs Data Mining and Knowledge Discovery
- Publisher
- Wiley
- Published
- 2023-02-28
- Volume
- 13
- Issue
- 3
- ISSN
- 1942-4787 (ISSN-L)
- 1942-4795
- OpenAlex
- W4322621694
- Cited by
- 161
- References
- 53
- FWCI
- 14.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.