Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · cited 1×
“This effect is called confirmation bias [112].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 7×
“The main goal is to create a suite of interpretable models and methods that produce more explainable models whilst preserving high predictive performance levels [2].”
BackgroundMethodInfluentialLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 7×
“Intrinsic interpretability is also called transparency and answers the question of how the model works [64].”
BackgroundMethodInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 4×
“However, it is noteworthy that this property is somewhat tricky [70], since the two models could use different features but could get similar predictions, which is described by the “Rashomon Effect” [108].”
BackgroundInfluentialAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 3×
“However, it is worth noting that this axiom only holds if the model does not have more degrees of freedom than needed to represent the prediction function [127].”
BackgroundMethodStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 1×
“There are many notorious examples of such DNN explanation methods [7], most of which are used in computer vision [125], including guided backpropagation [126], integrated gradients [127], SmoothGrad saliency maps [128], Grad-CAM [129], and more recently, testing with Concept Activation Vectors (TCAV) [130].”
MethodAn Experimental Study of Apparent Behavior
Heider, F., & Simmel, M. (1944) · The American Journal of Psychology · doi:10.2307/1416950 · Cited by 3,340 · cited 1×
“A few decades ago, Heider and Simmel [97] have shown that people attribute beliefs and intentions to abstract objects.”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 16×
“[41], explaining a ML model’s decisions provides a way to check the desiderata of ML systems, as noted in Section 3.”
BackgroundInfluentialEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“• Multi-objective trade-offs—Two well-defined desiderata in ML systems may compete with each other, such as privacy and prediction quality [90].”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundExplaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“With respect to interpretability itself, historically speaking, there has been sporadic interest in explanations of intelligent systems since the 1970s, beginning with attention on expert systems [11–13]; to, a decade after, neural networks [14]; and then to recommendation systems in the 2000s [15,16].”
BackgroundKahneman, D., & Tversky, A. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.015 · Cited by 1,476 · cited 1×
“• Focus on the abnormal—People focus more on abnormal causes to explain events [111].”
BackgroundSmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 1×
“There are many notorious examples of such DNN explanation methods [7], most of which are used in computer vision [125], including guided backpropagation [126], integrated gradients [127], SmoothGrad saliency maps [128], Grad-CAM [129], and more recently, testing with Concept Activation Vectors (TCAV) [130].”
MethodExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Predictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundWachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
The Magical Mystery Four: How Is Working Memory Capacity Limited, and Why?
Cowan, N. (2010) · Current Directions in Psychological Science · doi:10.1177/0963721409359277 · Cited by 1,255
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 1×
“There are many notorious examples of such DNN explanation methods [7], most of which are used in computer vision [125], including guided backpropagation [126], integrated gradients [127], SmoothGrad saliency maps [128], Grad-CAM [129], and more recently, testing with Concept Activation Vectors (TCAV) [130].”
MethodSurvey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225 · cited 1×
“With respect to interpretability itself, historically speaking, there has been sporadic interest in explanations of intelligent systems since the 1970s, beginning with attention on expert systems [11–13]; to, a decade after, neural networks [14]; and then to recommendation systems in the 2000s [15,16].”
BackgroundJolliffe, I. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat06472 · Cited by 1,221
Jolliffe, I. (2011) · International Encyclopedia of Statistical Science · doi:10.1007/978-3-642-04898-2_455 · Cited by 1,206
Interpretable Machine Learning
Molnar, C. (2020) · Cited by 1,104
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
The General Data Protection Regulation: An Opportunity for the HCI Community?
Thelisson, E., Sharma, K., Salam, H., & Dignum, V. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3170632 · Cited by 991
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · cited 2×
“Recently, the achievements of ML systems for many domains of high interest, as well as the use of increasingly complex and nontransparent algorithms, such as deep learning, calls for another wave of interest in the need to have a greater understanding of the aforementioned systems’ outputs [17].”
BackgroundInterpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · cited 1×
“These include decision sets [113], rule-based classifiers [114–117], and scorecards [37,118].”
BackgroundComprehensible classification models: a position paper
Freitas, A. A. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2594473.2594475 · Cited by 572
Google Vizier: A Service for Black-Box Optimization
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098043 · Cited by 571 · cited 2×
“In other words, any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand [8].”
BackgroundMethod(2016) · neural information processing systems · Cited by 556
Explainability Methods for Graph Convolutional Neural Networks
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01103 · Cited by 553 · cited 1×
“There are CVPR 2019 papers with a great focus on interpretability and explainability for computer vision, such as explainability methods for graph CNNs [138], interpretable and fine-grained visual explanations for CNNs [139], interpreting CNNs via Decision Trees [140], and learning to explain with complemental example…”
MethodIntelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · cited 1×
“[106] describes this as intelligibility, arguing that a Generative Additive Model (GAM) fulfills it.”
BackgroundDARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546
Explainable Artificial Intelligence (XAI)
Hossain, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6371435 · Cited by 541
Accountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · cited 2×
“However, it is important to mitigate concerns around trade secrets: Interpretability is a reasonable solution for this problem because explanations can be provided without revealing the internal logic of the system [78].”
BackgroundThe effects of transparency on trust in and acceptance of a content-based art recommender
Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-008-9051-3 · Cited by 448 · cited 1×
“With respect to interpretability itself, historically speaking, there has been sporadic interest in explanations of intelligent systems since the 1970s, beginning with attention on expert systems [11–13]; to, a decade after, neural networks [14]; and then to recommendation systems in the 2000s [15,16].”
BackgroundDistilling a neural network into a soft decision tree
Frosst, N., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1711.09784 · Cited by 407
The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype ClassificationPreprint
Kim, B., Rudin, C., & Shah, J. A. (2014) · Neural Information Processing Systems · doi:10.48550/arxiv.1503.01161 · Cited by 346 · cited 1×
“These include, for example, classifiers that are comprised of a small number of short rules [119], Bayesian case-based reasoning models [120], and neural networks with L1 penalties to their input gradients for sparse local explanations [121].”
BackgroundActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P. Y., Kalro, A., & Chau, D. H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 337 · cited 1×
“Facebook, in collaboration with Georgia Tech, published a paper where it shows a visual exploration tool of industry-scale DNN models [39].”
MethodWhat Does Explainable AI Really Mean? A New Conceptualization of Perspectives
Doran, D., Schulz, S., & Besold, T. R. (2017) · arXiv · doi:10.48550/arxiv.1710.00794 · Cited by 333 · cited 1×
“Despite the interchangeability concerning the use of both terms by many authors, there are others that develop some distinguishability between them [9,66,67].”
BackgroundInterpreting CNNs via Decision Trees
Zhang, Q., Yang, Y., Ma, H., & Wu, Y. N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00642 · Cited by 330 · cited 1×
“There are CVPR 2019 papers with a great focus on interpretability and explainability for computer vision, such as explainability methods for graph CNNs [138], interpretable and fine-grained visual explanations for CNNs [139], interpreting CNNs via Decision Trees [140], and learning to explain with complemental example…”
MethodLipton, P. (1990) · Royal Institute of Philosophy Supplement · doi:10.1017/s1358246100005130 · Cited by 281
Learning how to explain neural networks: PatternNet and PatternAttribution
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 264 · cited 1×
“There are many notorious examples of such DNN explanation methods [7], most of which are used in computer vision [125], including guided backpropagation [126], integrated gradients [127], SmoothGrad saliency maps [128], Grad-CAM [129], and more recently, testing with Concept Activation Vectors (TCAV) [130].”
MethodInterpreting Blackbox Models via Model ExtractionPreprint
Bastani, O., Kim, C., & Bastani, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08504 · Cited by 258
Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models
Apley, D. W., & Zhu, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08468 · Cited by 247 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundManifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
Zhang, J., Wang, Y., Molino, P., Li, L., & Ebert, D. S. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864499 · Cited by 220 · cited 1×
“The tool is not production ready at the time of writing, but a prototype has been developed in one of the most recently published papers by their research team [40].”
BackgroundAn explainable artificial intelligence system for small-unit tactical behavior
van Lent, M., Fisher, W. M., & Mancuso, M. R. (2004) · Cited by 215
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213
Model compression via distillation and quantization
Polino, A., Pascanu, R., & Alistarh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05668 · Cited by 212 · cited 1×
“This can be achieved, for example, through model compression [131] or tree regularization [132] or even by combining model compression with dimension reduction [133].”
MethodOn the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · cited 1×
“incidents in other domains, such as healthcare and finance [74].”
BackgroundConsiderations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176 · cited 11×
“Doshi-Velez and Kim answer this question by stating that “the problem is that a single metric, such as classification accuracy, is an incomplete description of most real-world tasks” [41].”
BackgroundInfluentialBoolean Decision Rules via Column GenerationPreprint
Dash, S., Günlük, O., & Wei, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09901 · Cited by 175
Explanations of Model Predictions with live and breakDown Packages
Staniak, M., & Biecek, P. (2019) · The R Journal · doi:10.32614/rj-2018-072 · Cited by 172 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundPerturbation-Based Explanations of Prediction Models
Robnik-Šikonja, M., & Bohanec, M. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_9 · Cited by 171 · cited 2×
“[109] have also defined some properties for the explanations (i.”
BackgroundOptimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice
Rudin, C. D., & Ustun, B. (2018) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2018.0957 · Cited by 154 · cited 1×
“These include decision sets [113], rule-based classifiers [114–117], and scorecards [37,118].”
BackgroundScalable Bayesian Rule ListsPreprint
Yang, H., Rudin, C. D., & Seltzer, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08610 · Cited by 147
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks
Wagner, J., Köhler, J. M., Gindele, T., Hetzel, L., Wiedemer, J. T., & Behnke, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00931 · Cited by 144 · cited 1×
“There are CVPR 2019 papers with a great focus on interpretability and explainability for computer vision, such as explainability methods for graph CNNs [138], interpretable and fine-grained visual explanations for CNNs [139], interpreting CNNs via Decision Trees [140], and learning to explain with complemental example…”
MethodPlease Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138
The Devil is in the Tails: Actuarial Mathematics and the Subprime Mortgage Crisis
Donnelly, C., & Paul Embrechts, A. (2010) · Astin Bulletin · doi:10.2143/ast.40.1.2049222 · Cited by 134 · cited 1×
“On top of that, there is evidence that incorrect modeling assumptions were, at least, partially responsible for the recent mortgage crisis [75].”
BackgroundEU regulations on algorithmic decision-making and a "right to explanation".Preprint
Goodman, B., & Flaxman, S. (2016) · arXiv (Cornell University) · Cited by 134
Explanations of Model Predictions with Live and breakDown Packages
Staniak, M., & Biecek, P. (2018) · Lincoln (University of Nebraska) · Cited by 128
Collision Avoidance for Unmanned Aircraft using Markov Decision Processes*
Temizer, S., Kochenderfer, M. J., Kaelbling, L. P., Lozano‐Pérez, T., & Kuchar, J. K. (2010) · AIAA Guidance, Navigation, and Control Conference and Exhibit · doi:10.2514/6.2010-8040 · Cited by 126 · cited 1×
“The latter refers to well-studied systems that have been used for a while, including postal code sorting and aircraft collision systems [71], which compute their output without human intervention.”
BackgroundHuman-in-the-Loop Interpretability Prior
Lage, I., Ross, A. S., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2018) · PubMed · doi:10.57702/j7gpfk6v · Cited by 125
Fisher, A., Rudin, C. D., & Dominici, F. (2018) · arXiv (Cornell University) · Cited by 117
(2018) · arXiv (Cornell University) · Cited by 113
Beyond Sparsity: Tree Regularization of Deep Models for InterpretabilityPreprint
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Röth, V., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06178 · Cited by 106 · cited 1×
“This can be achieved, for example, through model compression [131] or tree regularization [132] or even by combining model compression with dimension reduction [133].”
Method(2017) · doi:10.1145/3097983.3098047 · Cited by 104
Automatic Rule Extraction from Long Short Term Memory NetworksPreprint
Murdoch, W. J., & Szlam, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02540 · Cited by 102
Pereira, S., Meier, R., Alves, V., Reyes, M., & Silva, C. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02628-8 · Cited by 98 · cited 1×
“ICML Workshop on Human Interpretability in Machine Learning (WHI) [43–45] 2016–2018 NIPS Workshop on Interpretable Machine Learning for Complex Systems [46] 2016 NIPS Symposium on Interpretable Machine Learning [47] 2017 XCI: Explainable Computational Intelligence Workshop [48] 2017 IJCNN Explainability of Learning Ma…”
Background(2017) · arXiv (Cornell University) · Cited by 86
A multi-objective genetic optimization of interpretability-oriented fuzzy rule-based classifiers
Rudziński, F. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.09.038 · Cited by 82
Interpretability of machine learning models and representations: an introduction.
Bibal, A., & Frénay, B. (2016) · Repository of the University of Namur · Cited by 77
Stefan Rüping, A. (2006) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-8863 · Cited by 71 · cited 9×
“5, the main cause for the interpretability problem to remain unsolved is that interpretability is a very subjective concept and, thus, hard to formalize [91].”
BackgroundMethodInfluentialTowards Complementary Explanations Using Deep Neural Networks
Silva, W., Fernandes, K., Cardoso, M. J., & Cardoso, J. S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02628-8_15 · Cited by 53
Self-citationEMYCIN: A Knowledge Engineer’s Tool for Constructing Rule-Based Expert Systems
van Melle, W., Shortliffe, E. H., & Buchanan, B. G. (2005) · Cited by 52
Case, N. (2018) · Journal of design and science · doi:10.21428/61b2215c · Cited by 47 · cited 1×
“A strength of humans is intuition and reasoning [93].”
BackgroundQuestion-Answering with Grammatically-Interpretable Representations
Palangi, H., Smolensky, P., He, X., & Deng, L. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12004 · Cited by 47 · cited 1×
“It is also worth mentioning existing works that make use of tensor product representation to perform knowledge encoding and logical reasoning based on common-sense inference [123], which is useful for increasing interpretability through question-answering with the model [124].”
BackgroundBayesian Rule Sets for Interpretable Classification
Wang, T., Rudin, C. D., Doshi‐Velez, F., Liu, Y., Klampfl, E., & MacNeille, P. (2016) · Proceedings · doi:10.1109/icdm.2016.0171 · Cited by 46
sbrl: Scalable Bayesian Rule Lists Model
Yang, H., Rudin, C. D., & Seltzer, M. (2016) · doi:10.32614/cran.package.sbrl · Cited by 42
Learning to Explain With Complemental Examples
Kanehira, A., & Harada, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00880 · Cited by 39 · cited 1×
“There are CVPR 2019 papers with a great focus on interpretability and explainability for computer vision, such as explainability methods for graph CNNs [138], interpretable and fine-grained visual explanations for CNNs [139], interpreting CNNs via Decision Trees [140], and learning to explain with complemental example…”
MethodDetecting Bias in Black-Box Models Using Transparent Model Distillation.Preprint
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2017) · arXiv (Cornell University) · Cited by 36
Holzinger, A., Kieseberg, P., Tjoa, A., & Weippl, E. (2018) · International Cross-Domain Conference on Machine Learning and Knowledge Extraction · doi:10.1007/978-3-319-99740-7 · Cited by 35
Fahner, G. (2018) · Cited by 30
Müller, H., & Holzinger, A. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103546 · Cited by 30
What lies beneath? Understanding the limits of understanding
Keil, F. C., Rozenblit, L., & Mills, C. M. (2004) · Cited by 21
Explain to whom? Putting the User in the Center of Explainable AIPreprint
Kirsch, A. (2017) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 18
Varshney, K. R., Khanduri, P., Sharma, P., Zhang, S., & Varshney, P. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09710 · Cited by 16 · cited 1×
“Thus, for the success of AI as an indispensable tool for decision support, it must not only bring forth relevant information for the decision making task but also communicate at an appropriate rate and in a way that allows a human recipient of the information to tap into the strengths of intuition and reasoning [94].”
BackgroundXPLAIN: a system for creating and explaining expert consulting programs
Swartout, W. (1983) · Computer Compacts · doi:10.1016/0167-7136(83)90280-9 · Cited by 9
An Introduction on Interpretable Machine Learning
(2020) · International Journal of Innovative Technology and Exploring Engineering · doi:10.35940/ijitee.g1023.0597s20 · Cited by 8 · cited 1×
“Interpretability methods can be grouped regarding when these methods are applicable: before (pre-model), during (in-model), or after (post-model) building the ML model [99].”
MethodExSS: explainable smart systems 2019
Lim, B. Y., Sarkar, A., Smith, A., & Stumpf, S. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3308557.3313112 · Cited by 4 · cited 1×
“IPMU 2018—Advances on Explainable Artificial Intelligence [53] 2018 CD-MAKE Workshop on explainable Artificial Intelligence [54,55] 2018–2019 Workshop on Explainable Smart Systems (ExSS) [56,57] 2018–2019 ICAPS—Workshop on Explainable AI Planning (XAIP) [58,59] 2018–2019 AAAI-19 Workshop on Network Interpretability fo…”
Background
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
“According to Carvalho et al. (2019), such transparency is pivotal for enhancing understanding of AI dialogue systems, laying a solid foundation for the challenges and advantages of algorithmic decisions, and ensuring that decision-making processes are both accountable and fair.”
BackgroundInfluentialExplainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
“, model accuracy) do not always exhibit true prediction decisions [9].”
BackgroundA comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127
“Interpretabil-ity (Carvalho et al., 2019) refers to the ability of a user to understand the ensemble outcomes.”
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.