The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Review Study of Interpretation Methods for Future Interpretable Machine Learning
Open accessAbstract
In recent years, black-box models have developed rapidly because of their high accuracy. Balancing the interpretability and accuracy is increasingly important. The lack of interpretability severely limits the application of the model in academia and industry. Despite the various interpretable machine learning methods, the perspective and meaning of the interpretation are also different. We provide a review of the current interpretable methods and divide them based on the model being applied. We divide them into two categories: interpretable methods with the self-explanatory model and interpretable methods with external co-explanation. And the interpretable methods with external co-explanation are further divided into subbranch methods based on instances, SHAP, knowledge graph, deep learning, and clustering model. The classification aims to help us understand the model characteristics applied in the interpretable method better. This survey makes the researcher find a suitable model to solve interpretability problems more easily. And the comparison experiments contribute to discovering complementary features from different methods. At the same time, we explore the future challenges and trends of interpretable machine learning to promote the development of interpretable machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“Existing works include clarifying concepts related to interpretability [2] and distinguishing between interpretability and explainability [3].”
BackgroundConvolutional 2D Knowledge Graph Embeddings
Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11573 · Cited by 2,486
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“We collect five meth-ods including LIME [29], Anchors [30], CAM [71], Guided Grad-CAM [70], and meaningful perturbation [28] to show the interpretability.”
MethodMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 2×
“For the classification of interpretable methods, most of the articles are based on the intrinsic inter-pretable model and Post-hoc explanation [7], [8].”
BackgroundMethodUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Intelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534
Quinlan, J. R., & Cameron-Jones, R. M. (1993) · Lecture Notes in Computer Science · doi:10.1007/3-540-56602-3_124 · Cited by 528
Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425
Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05806 · Cited by 335
Explainable decision forest: Transforming a decision forest into an interpretable tree
Sagi, O., & Rokach, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.03.013 · Cited by 189
Perturbation-Based Explanations of Prediction Models
Robnik-Šikonja, M., & Bohanec, M. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_9 · Cited by 171
PALM: Machine Learning Explanations For Iterative Debugging
Krishnan, S., & Wu, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077271 · Cited by 81 · cited 1×
“Furthermore, the partition aware local model (PALM) was proposed to dispose of the misprediction in Deep Neural Network (DNN) [31] which helps machine learning to debug the reason for incorrect classifications.”
MethodAdditive Explanations for Anomalies Detected from Multivariate Temporal Data
Giurgiu, I., & Schumann, A. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3358121 · Cited by 27 · cited 1×
“Not only for individual prediction but also in intrusion detection systems [44] and anomaly detection [45], the SHAP framework provides great help for interpretability.”
Background
Research works printed but not linked: 76
- 10.1080/00401706.2000.10485981 · DOI
- 10.1177/001872089606380203 · DOI · Paper page
- 10.1109/ICCV.2017.371 · DOI · Paper page
- kim, 2016, Examples are not enough, learn to criticize! criticism for interpretability, Proc Adv Neural Inf Process Syst, 2280
- kim, 2014, The Bayesian case model: A generative approach for case-based reasoning and prototype classification, Proc Adv Neural Inf Process Syst, 1952
- 10.1016/j.artmed.2005.10.008 · DOI
- 10.1145/2623330.2623637 · DOI · Paper page
- 10.1016/j.ins.2019.10.050 · DOI
- wang, 2015, Falling rule lists, J Mach Learn Res, 38, 1013
- bertsimas, 2018, Interpretable clustering via optimal trees, arXiv 1812 00539
- craven, 1996, Extracting tree-structured representations of trained networks, Proc Adv Neural Inf Process Syst, 24
- 10.1109/ICDM.2016.0097 · DOI
- 10.1145/775047.775113 · DOI
- 10.1016/S0031-3203(98)00181-2 · DOI
- zhou, 2003, Extracting symbolic rules from trained neural network ensembles, AI Commun, 16, 3
- 10.1016/j.procs.2018.09.020 · doi:10.1016/j.procs.2018.09.020
- 10.1145/3308558.3313607 · DOI · Paper page
- 10.1609/aaai.v33i01.33013036 · DOI
- sarker, 2017, Explaining trained neural networks with semantic Web technologies: First steps, Proc CEUR Workshop, 1
- geng, 2019, Human-centric transfer learning explanation via knowledge graph, arXiv 1901 08547
- 10.1145/3077136.3080777 · DOI · Paper page
- 10.1145/2939672.2939673 · DOI · Paper page
- 10.1609/aaai.v33i01.33015329 · DOI · Paper page
- 10.1145/3240323.3240361 · DOI · Paper page
- catherine, 2017, Explainable entity-based recommendations with knowledge graphs, Proc CEUR Workshop, 1
- 10.1109/UIC-ATC.2017.8397411 · DOI · Paper page
- 10.1145/3359786 · DOI · Paper page
- 10.1145/3219819.3220072 · DOI
- 10.1145/3236009 · DOI · Paper page
- 10.1007/s00778-015-0394-1 · DOI · Paper page
- minervini, 2017, Regularizing Knowledge Graph Embeddings via Equivalence and Inversion Axioms, 668
- janzing, 2019, Feature relevance quantification in explainable AI: A causal problem, arXiv 1910 13413
- lundberg, 2018, Consistent individualized feature attribution for tree ensembles, arXiv 1802 03888
- 10.1109/ACCESS.2020.2988359 · DOI · Paper page
- sundararajan, 2019, The many shapley values for model explanation, arXiv 1908 08474
- 10.1007/978-3-030-01237-3_8 · DOI · Paper page
- shrikumar, 2017, Learning important features through propagating activation differences, Proc Int Conf Mach Learn (ICML), 4844
- 10.1109/CVPR.2016.319 · DOI · Paper page
- singh, 2019, Hierarchical interpretations for neural network predictions, Proc Int Conf Learn Represent (ICLR), 1
- zharov, 2018, YASENN: Explaining neural networks via partitioning activation sequences, arXiv 1811 02783
- zhang, 2018, Interpreting CNN knowledge via an explanatory graph, Proc AAAI Conf Artif Intell, 4454
- 10.1109/CVPR.2018.00920 · DOI · Paper page
- huang, 2017, Densely connected convolutional networks, Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 4700
- 10.1007/978-3-319-10590-1_53 · DOI · Paper page
- yosinski, 2015, Understanding neural networks through deep visualization, arXiv 1506 06579
- bibal, 2016, Interpretability of machine learning models and representations: an introduction, Proc Eur Symp Artif Neural Netw (ESANN), 77
- binder, 2016, Layer-Wise Relevance Propagation for Deep Neural Network Architectures, 913
- 10.1109/CVPR.2018.00907 · DOI · Paper page
- 10.1109/CVPR.2016.308 · DOI · Paper page
- kinkeldey, 2019, Towards Supporting Interpretability of Clustering Results with Uncertainty Visualization, Proc EuroVis Workshop Trustworthy Visualizat (TrustVis), 1
- alaa ahmed, 2019, Demystifying black-box models with symbolic metamodels, Proc Adv Neural Inf Process Syst, 11304
- 10.1016/j.cag.2014.01.006 · DOI
- yoon, 2019, INVASE: Instance-wise variable selection using neural networks, Proc Int Conf Learn Represent (ICLR), 1
- 10.1016/j.neucom.2006.11.018 · DOI
- 10.1109/VAST.2018.8802509 · DOI
- laurens, 2008, Visualizing data using t-SNE, J Mach Learn Res, 9, 2579
- ilyas, 2019, Adversarial examples are not bugs, they are features, Proc Adv Neural Inf Process Syst, 125
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- simonyan, 2015, Very deep convolutional networks for large-scale image recognition, Proc 3rd Int Conf Learn Represent ICLR Conf Track, 1
- lecun, 1999, The MNIST Database of Handwritten Digits
- 10.1109/CVPR.2017.544 · DOI · Paper page
- molnar, 2019, Example-based explanations, Interpretable Machine Learning A Guide for Making Black Box Models Explainable
- 10.1007/978-3-319-45654-6_1 · doi:10.1007/978-3-319-45654-6_1
- ravikumar, 2009, SpAM: Sparse additive models, Proc Adv Neural Inf Process Syst, 1
- 10.1145/2939672.2939778 · DOI · Paper page
- 10.1145/3243734.3243792 · DOI · Paper page
- 10.1109/TPAMI.2018.2858759 · DOI · Paper page
- 10.1109/IS.2006.348467 · DOI
- oramas, 2019, Visual explanation by interpretation: Improving visual feedback capabilities of deep neural networks, Proc Int Conf Learn Represent (ICLR), 1
- 10.1007/s11634-013-0129-3 · DOI
- bau, 2019, GAN dissection: Visualizing and understanding generative adversarial networks, Proc Int Conf Learn Represent (ICLR), 1
- 10.1145/3375627.3375843 · DOI
- hsu, 2017, Unsupervised learning of disentangled and interpretable representations from sequential data, Proc Adv Neural Inf Process Syst, 1879
- 10.1007/978-3-319-49055-7_38 · doi:10.1007/978-3-319-49055-7_38
- antwarg, 2019, Explaining anomalies detected by autoencoders using SHAP, arXiv 1903 02407
- shankaranarayana, 2019, ALIME Autoencoder Based Approach for Local Interpretability, 454
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606
ŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214
Machine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174
Metadata
- Authors
- Mi, J.-X., Li, A.-D., & Zhou, L.-F.
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 191969-191985
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3093681468
- Cited by
- 158
- References
- 29
- FWCI
- 5.11
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Natural Science Foundation of Chongqing (cstc2018jcyjAX0532, cstc2019jcyj-msxmX0461)
- National Natural Science Foundation of China (61906024)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.