Definitions, methods, and applications in interpretable machine learning
Open accessThe DOI matches its Crossref record; 123 of the 123 references it cites were found in a registry or an index.
Abstract
Significance: The recent surge in interpretability research has led to confusion on numerous fronts. In particular, it is unclear what it means to be interpretable and how to select, evaluate, or even discuss methods for producing interpretations of machine-learning models. We aim to clarify these concerns by defining interpretable machine learning and constructing a unifying framework for existing methods which highlights the underappreciated role played by human audiences. Within this framework, methods are organized into 2 classes: model based and post hoc. To provide guidance in selecting and evaluating interpretation methods, we introduce 3 desiderata: predictive accuracy, descriptive accuracy, and relevancy. Using our framework, we review existing work, grounded in real-world studies which exemplify our desiderata, and suggest directions for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Open problems include: a remaining lack of agreement on what explainability means [3, 14, 16], no clear guidance how to choose amongst explainable AI methods [19], and the absence of standardized evaluation methods [14, 16, 19].”
In the citing paper
“Integrating ML and AI findings with existing domain knowledge and expert interpretation is essential for ensuring the validity and relevance of the results [59, 60].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · cited 1×
“Methods for obtaining sparsity often utilize a penalty on a loss function, such as LASSO (31) and sparse coding (32), or on a model selection criteria such as AIC or BIC (33, 34).”
MethodHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4 · Cited by 33,115
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“One line of work focuses on providing an overview of different interpretation methods with a strong emphasis on post hoc interpretations of deep learning models (7, 8), sometimes pointing out similarities between various methods (9, 10).”
BackgroundClassification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“wolf classifier simply learns to identify whether there is snow in the image, ignoring the animals themselves (78).”
BackgroundVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“When visualizing convolutional neural networks trained on image data, work has been done on visualizing filters (62, 63), maximally activating responses of individual neurons or classes (64), understanding intra-class variation (65), and grouping different neurons (66).”
MethodA survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
ggplot2: Elegant Graphics for Data Analysis
Ginestet, C. (2011) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00676_9.x · Cited by 13,726
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Multimodel Inference: Understanding AIC and BIC in Model Selection
Burnham, K. P., & Anderson, D. R. (2004) · Sociological Methods & Research · doi:10.1177/0049124104268644 · Cited by 11,648 · cited 1×
“Methods for obtaining sparsity often utilize a penalty on a loss function, such as LASSO (31) and sparse coding (32), or on a model selection criteria such as AIC or BIC (33, 34).”
MethodData Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-00a · Cited by 11,032 · cited 1×
“Examples include interactive environments (89–91), tools for visualization (92–94), and data exploration tools (95, 96).”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986 · cited 1×
“Methods such as principal components analysis (48), independent components analysis (49), and canonical correlation analysis (50) can often identify a few interpretable dimensions, which can then be used as input to a model or to provide insights in their own right.”
BackgroundGoodall, C. (1988) · Technometrics · doi:10.1080/00401706.1988.10488412 · Cited by 8,675
Jolliffe, I. T. (1986) · Springer Series in Statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,273
seaborn: statistical data visualization
Waskom, M. (2021) · Journal of Open Source Software · doi:10.21105/joss.03021 · Cited by 7,310
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
(2009) · Cited by 6,153
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Other work has focused on the narrower problem of evaluating interpretations (12, 13) and what properties they should satisfy (14).”
BackgroundRELATIONS BETWEEN TWO SETS OF VARIATES
HOTELLING, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,263
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 1×
“If the constructed model does not accurately represent the underlying problem, any subsequent analysis will be suspect (28, 29).”
BackgroundPati, Y. C., Rezaiifar, R., & Krishnaprasad, P. S. (2002) · Proceedings of 27th Asilomar Conference on Signals, Systems and Computers · doi:10.1109/acssc.1993.342465 · Cited by 4,323
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 2×
“This is normally done by introducing a novel form of output, such as feature heatmaps (71), rationales (90), or feature hierarchies (84), or identifying important elements in the training set (86).”
BackgroundAkaike, H. (1987) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_29 · Cited by 4,027 · cited 1×
“Methods for obtaining sparsity often utilize a penalty on a loss function, such as LASSO (31) and sparse coding (32), or on a model selection criteria such as AIC or BIC (33, 34).”
MethodIPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009
Robust Statistics: The Approach Based on Influence Functions
Ruppert, D. (1987) · Technometrics · doi:10.2307/1269782 · Cited by 3,880 · cited 1×
“Recently, stability has been shown to be important in applied statistical problems, for example when trying to make conclusions about a scientific problem (21) and in more general settings (22).”
BackgroundSparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · cited 1×
“Methods for obtaining sparsity often utilize a penalty on a loss function, such as LASSO (31) and sparse coding (32), or on a model selection criteria such as AIC or BIC (33, 34).”
MethodDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“, medicine (1), policymaking (2), and science (3, 4), as well as in auditing the predictions themselves in response to issues such as regulatory pressure (5) and fairness (6).”
BackgroundStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 1×
“In the deep learning literature, a number of different approaches have been proposed to address this problem (70–79), with some methods for other models as well (80).”
MethodHastie, T., & Tibshirani, R. (1986) · Statistical Science · doi:10.1214/ss/1177013604 · Cited by 3,409 · cited 1×
“Generalized additive models (40) force the relationship between variables in the model to be additive.”
BackgroundConditional variable importance for random forests
Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-307 · Cited by 3,254
Jupyter Notebooks – a publishing format for reproducible computational workflows
Thomas, K., Benjamin, R.-K., Fernando, P., Brian, G., Matthias, B., Jonathan, F., Kyle, K., Jessica, H., Jason, G., Sylvain, C., et al. (2016) · Positioning and Power in Academic Publishing: Players, Agents and Agendas · doi:10.3233/978-1-61499-649-1-87 · Cited by 3,063
Permutation importance: a corrected feature importance measure
Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq134 · Cited by 2,843
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“For example, interpretability is a major topic when considering bias and fairness in ML models (14, 15), with examples given throughout the paper (16).”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 2×
“medicine (1), policy-making (2), and science (3, 4), as well as in auditing the predictions themselves in response to issues such as regulatory pressure (5) and fairness (6).”
BackgroundBox, G. E. P. (1976) · Journal of the American Statistical Association · doi:10.1080/01621459.1976.10480949 · Cited by 2,114 · cited 1×
“Otherwise, they up-date something in the chain (problem, data, and/or model) and iterate, potentially multiple times (27).”
BackgroundRobust Statistics: The Approach Based on Influence Functions.
Jennison, C., Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1987) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981480 · Cited by 1,753
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Biometrika · doi:10.2307/2333955 · Cited by 1,645
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 4×
“Researchers have proposed a variety of interpretation forms, including feature heatmaps (71), feature hierarchies (82) and identifying important elements in the training set (69).”
BackgroundMethodInfluentialDeep learning for computational biology
Angermueller, C., Pärnamaa, T., Parts, L., & Stegle, O. (2016) · Molecular Systems Biology · doi:10.15252/msb.20156651 · Cited by 1,473 · cited 1×
“, medicine (1), policymaking (2), and science (3, 4), as well as in auditing the predictions themselves in response to issues such as regulatory pressure (5) and fairness (6).”
BackgroundSmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Consistent Individualized Feature Attribution for Tree Ensembles
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018) · arXiv · doi:10.48550/arxiv.1802.03888 · Cited by 1,430 · cited 1×
“In the deep learning literature, a number of different approaches have been proposed to address this problem (70–79), with some methods for other models as well (80).”
Methodtidyverse: Easily Install and Load the 'Tidyverse'
Wickham, H. (2016) · CRAN: Contributed Packages · doi:10.32614/cran.package.tidyverse · Cited by 1,381
Keil, F. C. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.57.102904.190100 · Cited by 1,350
Predictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315 · cited 1×
“rules (38, 39), which can easily be simulated.”
BackgroundSanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294 · cited 1×
“While feature importance scores can provide useful insights, they also have a number of limitations (80, 82).”
BackgroundGeneralized additive models for medical research
Hastie, T., & Tibshirani, R. (1995) · Statistical Methods in Medical Research · doi:10.1177/096228029500400302 · Cited by 1,266
Olden, J. D., Joy, M. K., & Death, R. G. (2004) · Ecological Modelling · doi:10.1016/j.ecolmodel.2004.03.013 · Cited by 1,029 · cited 1×
“Methods have been developed to score individual features in many models including neural networks (52), random forests, (53, 54), and generic classifiers (55).”
BackgroundAndreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.12 · Cited by 908 · cited 1×
“In deep learning, specific methods such as attention (41) and modular network architectures (42) provide limited insight into a network’s inner workings.”
BackgroundOlah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · cited 1×
“…Term Memory Networks (LSTMs), researchers have focused on analyzing the state vector, identifying individual dimensions that correspond to meaningful features (e.g. position in line, within quotes) (67), and building tools to track the model’s decision process over the course of a sequence (68).”
BackgroundCredit scoring with a data mining approach based on support vector machines
Huang, C.-L., Chen, M.-C., & Wang, C.-J. (2007) · Expert Systems with Applications · doi:10.1016/j.eswa.2006.07.007 · Cited by 804 · cited 1×
“In industrial settings, the problem often concerns the predictive performance or other qualities of a model, such as how to assign credit scores with high accuracy (26), or do so fairly with respect to gender and race (16).”
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 1×
“This is normally done by introducing a novel form of output, such as feature heatmaps (71), rationales (87), feature hierarchies (82) or identifying important elements in the training set (69).”
BackgroundLetham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · cited 5×
“In order to construct a model that has high predictive and descriptive accuracy, (39) introduce a method for learning lists of if-then rules that are predictive of one year stroke risk.”
BackgroundMethodInfluentialDatta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 3×
“For example, interpretability is a major topic when considering bias and fairness in ML models (14, 15), with examples given throughout the paper (16).”
BackgroundThe structure and function of explanations
Lombrozo, T. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.08.004 · Cited by 639 · cited 1×
“In psychology, the general notions of interpretability and explanations have been studied at a more abstract level (17, 18), providing relevant conceptual perspectives.”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“One line of work focuses on providing an overview of different interpretation methods with a strong emphasis on post hoc interpretations of deep learning models (7, 8), sometimes pointing out similarities between various methods (9, 10).”
BackgroundProceedings of the AMIA Symposium
Schulz, S., Daumke, P., Smith, B., & Hahn, U. (2005) · Cited by 606
Extracting Tree-Structured Representations of Trained Networks
Craven, M. W., & SHAVLIK, J. W. (1995) · neural information processing systems · Cited by 602
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · cited 1×
“Given the challenges surrounding simply generating post hoc interpretations, research on their uses has been limited (97, 98).”
BackgroundNot Just a Black Box: Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 496
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · cited 1×
“In the deep learning literature, a number of different approaches have been proposed to address this problem (70–79), with some methods for other models as well (80).”
MethodInterpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M., Preece, A., Julier, S., Rao, R. M., et al. (2017) · 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 422 · cited 1×
“One line of work focuses on providing an overview of different interpretation methods with a strong emphasis on post hoc interpretations of deep learning models (7, 8), sometimes pointing out similarities between various methods (9, 10).”
BackgroundGrounding of Textual Phrases in Images by Reconstruction
Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., & Schiele, B. (2016) · Max Planck Digital Library · Cited by 422
Distilling a neural network into a soft decision tree
Frosst, N., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1711.09784 · Cited by 407 · cited 1×
“In addition to the areas below, we note that there are other emerging techniques, such as model distillation (51, 52).”
MethodUsing “annotator rationales” to improve machine learning for text categorization
Zaidan, O. F., Eisner, J. M., & Piatko, C. (2007) · Cited by 393
Toward a Unified Theory of Visual Area V4
Roe, A. W., Chelazzi, L., Connor, C. E., Conway, B. R., Fujita, I., Gallant, J. L., Lu, H., & Vanduffel, W. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.011 · Cited by 374 · cited 1×
“In a scientific setting, the practitioner may be interested in relationships contained in the data, such as how brain cells in a particular area of the visual system relate to visual stimuli (25).”
BackgroundLSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 371
Towards better understanding of gradient-based attribution methods for Deep Neural Networks
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 368 · cited 1×
“One line of work focuses on providing an overview of different interpretation methods with a strong emphasis on post hoc interpretations of deep learning models (8, 9), sometimes pointing out similarities between various methods (10, 11).”
BackgroundInterpretable Learning for Self-Driving Cars by Visualizing Causal Attention
Kim, J., & Canny, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.320 · Cited by 366 · cited 1×
“In deep learning, specific methods such as attention (41) and modular network architectures (42) provide limited insight into a network’s inner workings.”
BackgroundBox, G. E. P. (1976) · Journal of the American Statistical Association · doi:10.2307/2286841 · Cited by 366
Proceedings of the 9th Python in Science Conference
(2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-012 · Cited by 342
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 335 · cited 1×
“These methods focus on explicitly capturing and displaying the interactions learned by a neural network (81, 82), alternative forms of interpretations such as textual explanations (83), identifying influential data points (69), and analyzing nearest neighbors (84, 85).”
BackgroundStatistical models and shoe leather
Freedman, D. A. (1991) · Sociological Methodology · doi:10.2307/270939 · Cited by 333
Iterative random forests to discover predictive and stable high-order interactions
Basu, S., Kumbier, K., Brown, J. B., & Yu, B. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711236115 · Cited by 317 · cited 6×
“For instance, in one example discussed above (20), the authors evaluated a new interpretation method (iterative random forests) by demonstrating that it could be used to identify meaningful biological Boolean interactions for use in experiments.”
BackgroundMethodInfluentialSelf-citationGrad-CAM: Why did you say that?
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07450 · Cited by 309
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1999) · Unsupervised Learning · doi:10.7551/mitpress/7011.003.0009 · Cited by 284
Statistical models and shoe leather
Freedman, D. (1989) · Mathematical Social Sciences · doi:10.1016/0165-4896(89)90047-4 · Cited by 242 · cited 1×
“If the constructed model does not accurately represent the underlying problem, any subsequent analysis will be suspect (28, 29).”
BackgroundAmaratunga, D., Cabrera, J., & Lee, Y.-S. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn356 · Cited by 228
Generalized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables
Hooker, G. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x237892 · Cited by 220 · cited 2×
“While feature importance scores can provide useful insights, they also have a number of limitations (80, 82).”
BackgroundInterpreting CNN Knowledge via an Explanatory Graph
Zhang, Q., Cao, R., Shi, F., Wu, Y. N., & Zhu, S.-C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11819 · Cited by 214
A Shared Vision for Machine Learning in Neuroscience
Vu, M.-A. T., Adalı, T., Ba, D., Buzsáki, G., Carlson, D., Heller, K., Liston, C., Rudin, C., Sohal, V. S., Widge, A. S., et al. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0508-17.2018 · Cited by 191 · cited 2×
“medicine (1), policy-making (2), and science (3, 4), as well as in auditing the predictions themselves in response to issues such as regulatory pressure (5) and fairness (6).”
BackgroundDetecting Statistical Interactions from Neural Network WeightsPreprint
Tsang, M., Cheng, D., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04977 · Cited by 189 · cited 2×
“For example, (57, 86) train neural networks on a suite of generative models with certain built-in interactions, and test whether their method successfully recovers them.”
BackgroundBeyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
Murdoch, W. J., Liu, P. J., & Yu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05453 · Cited by 150 · cited 1×
“These methods focus on explicitly capturing and displaying the interactions learned by a neural network (81, 82), alternative forms of interpretations such as textual explanations (83), identifying influential data points (69), and analyzing nearest neighbors (84, 85).”
BackgroundSelf-citationWu, S., Joseph, A., Hammonds, A. S., Celniker, S., Yu, B., & Frise, E. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521171113 · Cited by 139 · cited 1×
“A recent study (50) uses unsupervised learning to learn spatial patterns of gene expression in Drosophila (fruit fly) embryos.”
BackgroundSelf-citationPlease Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138
Hierarchical interpretations for neural network predictionsPreprint
Singh, C., Murdoch, W. J., & Yu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05337 · Cited by 127 · cited 4×
“Researchers have proposed a variety of interpretation forms, including feature heatmaps (71), feature hierarchies (82) and identifying important elements in the training set (69).”
BackgroundInfluentialSelf-citationA Roadmap for a Rigorous Science of Interpretability.Preprint
Doshi‐Velez, F., & Kim, B. (2017) · arXiv (Cornell University) · Cited by 119
(2018) · arXiv (Cornell University) · Cited by 113
Yu, B. (2013) · Bernoulli · doi:10.3150/13-bejsp14 · Cited by 113 · cited 1×
“Recently, stability has been shown to be important in applied statistical problems, for example when trying to make conclusions about a scientific problem (21) and in more general settings (22).”
BackgroundSelf-citationUnderstanding Intra-Class Knowledge Inside CNN
Wei, D., Zhou, B., Torrabla, A., & Freeman, W. (2015) · arXiv · doi:10.48550/arxiv.1507.02379 · Cited by 98 · cited 1×
“When visualizing convolutional neural networks trained on image data, work has been done on visualizing filters (62, 63), maximally activating responses of individual neurons or classes (64), understanding intra-class variation (65), and grouping different neurons (66).”
MethodEstimation Stability With Cross-Validation (ESCV)
Lim, C., & Yu, B. (2016) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2015.1020159 · Cited by 97 · cited 1×
“For example, if the set of sparse parameters changes due to small perturbations in the data set, the coefficients should not be interpreted (30).”
BackgroundSelf-citationCase-based explanation of non-case-based learning methods.
Caruana, R., Kangarloo, H., Dionisio, J. D. N., Sinha, U., & Johnson, D. (1999) · PubMed · Cited by 94
Real Time Image Saliency for Black Box Classifiers
Dabkowski, P., & Gal, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07857 · Cited by 77
Brennan, T., & Oliver, W. L. (2013) · Criminology & Public Policy · doi:10.1111/1745-9133.12055 · Cited by 69 · cited 3×
“and Graphical Statistics 25(2):464–492.”
BackgroundAutomatic Rule Extraction from Long Short Term Memory Networks
Murdoch, W. J., & Szlam, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02540 · Cited by 67
Self-citationVisual Analysis of Hidden State Dynamics in Recurrent Neural NetworksPreprint
Strobelt, H., Gehrmann, S., Huber, B., Pfister, H., & Rush, A. M. (2016) · RePEc: Research Papers in Economics · Cited by 57
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54
Olden, J. D. (2004) · Ecological Modelling · doi:10.1016/s0304-3800(04)00156-5 · Cited by 54
A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
Nie, W., Zhang, Y., & Patel, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07039 · Cited by 51 · cited 1×
“Moreover, it has been shown that some feature importance scores for CNNs are doing (partial) image recovery which is unrelated to the network decisions (81).”
BackgroundInterpretable Learning for Self-Driving Cars by Visualizing Causal AttentionPreprint
Kim, J., & Canny, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10631 · Cited by 48 · cited 1×
“In deep learning, specific methods such as attention (41) and modular network architectures (42) provide limited insight into a network’s inner workings.”
BackgroundThe DeepTune framework for modeling and characterizing neurons in visual cortex area V4Preprint
Abbasi-Asl, R., Chen, Y., Bloniarz, A. E., Oliver, M., Willmore, B. D. B., Gallant, J. L., & Yu, B. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/465534 · Cited by 39 · cited 2×
“A recent study visualizes learned information from deep neural networks to understand individual brain cells (24).”
BackgroundMethodSelf-citationInterpreting CNN Knowledge via an Explanatory GraphPreprint
Zhang, Q., Cao, R., Shi, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01785 · Cited by 33 · cited 1×
“When visualizing convolutional neural networks trained on image data, work has been done on visualizing filters (62, 63), maximally activating responses of individual neurons or classes (64), understanding intra-class variation (65), and grouping different neurons (66).”
MethodStructural Compression of Convolutional Neural Networks Based on Greedy Filter PruningPreprint
Abbasi-Asl, R., & Yu, B. (2017) · arXiv (Cornell University) · Cited by 32
Self-citationCan I trust you more? Model-Agnostic Hierarchical ExplanationsPreprint
Tsang, M., Sun, Y., Ren, D., & Liu, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.04801 · Cited by 24 · cited 1×
“For example, (57, 86) train neural networks on a suite of generative models with certain built-in interactions, and test whether their method successfully recovers them.”
BackgroundSigned iterative random forests to identify enhancer-associated transcription factor bindingPreprint
Kumbier, K., Basu, S., Frise, E., Celniker, S., Brown, J. B., Celniker, S., & Yu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07287 · Cited by 21
Self-citationRefining interaction search through signed iterative Random ForestsPreprint
Kumbier, K., Basu, S., Brown, J. B., Celniker, S., & Yu, B. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/467498 · Cited by 20 · cited 2×
“A previous line of work considers the problem of searching for biological interactions associated with important biological processes (20, 56).”
BackgroundSelf-citationDaytime Arctic Cloud Detection Based on Multi-Angle Satellite Data With Case Studies
Shi, T., Yu, B., Clothiaux, E. E., & Braverman, A. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001283 · Cited by 19 · cited 1×
“In (47), the authors use an unlabeled dataset of arctic satellite imagery to build a model predicting whether each pixel in an image contains clouds or not.”
BackgroundSelf-citationDisentangled Attribution Curves for Interpreting Random Forests and Boosted TreesPreprint
Devlin, S., Singh, C., Murdoch, W. J., & Yu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07631 · Cited by 14 · cited 1×
“Methods exist to extract interactions from many ML models, including random forests (21, 57, 58) and neural networks (59, 60).”
MethodSelf-citationBiclustering by sparse canonical correlation analysis
Pimentel, H., Hu, Z., & Huang, H.-Y. (2018) · Quantitative Biology · doi:10.1007/s40484-017-0127-0 · Cited by 11 · cited 2×
“This is demonstrated through examples in the text (20, 23, 24).”
BackgroundMethodSuperheat: Supervised heatmaps for visualizing complex dataPreprint
Barter, R., & Yu, B. (2015) · arXiv (Cornell University) · Cited by 6
Self-citation
Research works printed but not linked: 5
- Z. C. Lipton The mythos of model interpretability. arXiv:1606.03490 (6 March 2017). · arXiv
- J. Ramos, “Using tf-idf to determine word relevance in document queries” in Proceedings of the First Instructional Conference on Machine Learning, T. Fawcett, N. Mishra, Eds. (ICML, 2003), vol. 242, pp. 133–142.
- Office of Institutional Research HU Exhibit 157: Demographics of Harvard college applicants. http://samv91khoyt2i553a2t1s05i-wpengine.netdna-ssl.com/wp-content/uploads/2018/06/Doc-421-157-May-30-2013-Report.pdf (2018) pp. 8–9.
- A. Mordvintsev, C. Olah, M. Tyka, Deepdream-a code example for visualizing neural networks. Google Res. 2, 5 (2015).
- RStudio Team, RStudio: Integrated Development Environment for R (RStudio, Inc., Boston, MA, 2016).
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“[20] published a survey in 2019, in which they created an interpretability framework in the hope that it would help to bridge the aforementioned gap in the field.”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Metadata
- Authors
- Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B.
- Affiliations
- University of California, Berkeley (US)
- Allen Institute for Brain Science (US)
- Corresponding author
- Yu, Bin
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-10-16
- Volume
- 116
- Issue
- 44
- Pages
- 22071-22080
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6825274
- OpenAlex
- W2910705748
- Cited by
- 2,167
- References
- 121
- Citations by year
- 2026: 296 · 2025: 371 · 2024: 386 · 2023: 363 · 2022: 298 · 2021: 268 · 2020: 145 · 2019: 34
- FWCI
- 84.50
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1741340, DMS-1613002, CCF-0939370)
- Gouvernement du Canada | Natural Sciences and Engineering Research Council of Canada (CGSD3-487534-2016)
- DOD | United States Navy | Office of Naval Research (N00014-16-1-2664)
- DOD | United States Army | RDECOM | Army Research Office (W911NF1710005)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- interpretability, model interpretability, model-based explanation, post-hoc explanation, sparsity, modularity, simulatability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.