The DOI matches its Crossref record; 245 of the 245 references it cites were found in a registry or an index.
Explainable Deep Learning: A Field Guide for the Uninitiated
Open accessAbstract
Deep neural networks (DNNs) are an indispensable machine learning tool despite the difficulty of diagnosing what aspects of a model’s input drive its decisions. In countless real-world domains, from legislation and law enforcement to healthcare, such diagnosis is essential to ensure that DNN decisions are driven by aspects appropriate in the context of its use. The development of methods and studies enabling the explanation of a DNN’s decisions has thus blossomed into an active and broad area of research. The field’s complexity is exacerbated by competing definitions of what it means “to explain” the actions of a DNN and to evaluate an approach’s “ability to explain”. This article offers a field guide to explore the space of explainable deep learning for those in the AI/ML field who are uninitiated. The field guide: i) Introduces three simple dimensions defining the space of foundational methods that contribute to explainable deep learning, ii) discusses the evaluations for model explanations, iii) places explainability in the context of other related deep learning research areas, and iv) discusses user-oriented explanation design and future directions. We hope the guide is seen as a starting point for those embarking on this research field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–245 of 245
Explaining Predictions of Non-Linear Classifiers in NLP
Arras, L., Horn, F., Montavon, G., Müller, K., & Samek, W. (2016) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w16-1601 · Cited by 118
A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based VisualizationsPreprint
Nie, W., Zhang, Y., & Patel, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07039 · Cited by 115
The Doctor Just Won't Accept That!Preprint
Lipton, Z. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08037 · Cited by 112
Learning With Interpretable Structure From Gated RNN
Hou, B., & Zhou, Z. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2967051 · Cited by 112
How can I choose an explainer?: An Application-grounded Evaluation of Post-hoc Explanations
Jesus, S., Belém, C., Balayan, V., Bento, J., Saleiro, P., Bizarro, P., & Gama, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445941 · Cited by 112
Pérez-Suay, A., Laparra, V., Mateo‐García, G., Muñoz-Marí, J., Gómez‐Chova, L., & Camps‐Valls, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71249-9_21 · Cited by 110
Heskes, T., Sijben, E., Bucur, I. G., & Claassen, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01625 · Cited by 106
Investigating the influence of noise and distractors on the interpretation of neural networksPreprint
Kindermans, P.-J., Schütt, K. T., Müller, K., & Dähne, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07270 · Cited by 104
Automatic Rule Extraction from Long Short Term Memory NetworksPreprint
Murdoch, W. J., & Szlam, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02540 · Cited by 102
Characterizing Driving Styles with Deep Learning
Dong, W., Li, J., Yao, R., Li, C., Yuan, T., & Wang, L. (2016) · arXiv · doi:10.48550/arxiv.1607.03611 · Cited by 100
Do Semantic Parts Emerge in Convolutional Neural Networks?
González-García, A., Modolo, D., & Ferrari, V. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1048-0 · Cited by 96
Attentive Explanations: Justifying Decisions and Pointing to the EvidencePreprint
Park, D. H., Hendricks, L. A., Akata, Z., Schiele, B., Darrell, T., & Rohrbach, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04757 · Cited by 94
Obtaining fairness using optimal transport theoryPreprint
del Barrio, E., Gamboa, F., Gordaliza, P., & Loubes, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03195 · Cited by 91
Uncertainty-Aware Attention for Reliable Interpretation and PredictionPreprint
Heo, J., Lee, H., Kim, S., Lee, J., Kim, K. J., Yang, E., & Hwang, S. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09653 · Cited by 86
Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction InvariancePreprint
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05817 · Cited by 84
Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov ModelsPreprint
Krakovna, V., & Doshi‐Velez, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05320 · Cited by 83
Learning Global Additive Explanations for Neural Nets Using Model DistillationPreprint
Tan, S., Caruana, R., Hooker, G., Koch, P., & Gordo, A. (2018) · arXiv (Cornell University) · Cited by 83
Deep learning for HRRP-based target recognition in multistatic radar systems
Lunden, J., & Koivunen, V. (2016) · 2016 IEEE Radar Conference (RadarConf) · doi:10.1109/radar.2016.7485271 · Cited by 82
ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking
Groth, O., Fuchs, F. B., Posner, I., & Vedaldi, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_43 · Cited by 82
Causal Learning and Explanation of Deep Neural Networks via Autoencoded ActivationsPreprint
Harradon, M., Druce, J., & Ruttenberg, B. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00541 · Cited by 80
Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box ModelsPreprint
Adebayo, J., & Kagal, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04967 · Cited by 70
Importance of Self-Attention for Sentiment Analysis
Letarte, G., Paradis, F., Giguère, P., & Laviolette, F. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5429 · Cited by 70
Georgiev, P., Bhattacharya, S., Lane, N. D., & Mascolo, C. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3131895 · Cited by 67
Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning
Zhang, Q., Cao, R., Wu, Y. N., & Zhu, S. C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10924 · Cited by 66
Classification Using Class Cover Catch Digraphs
Priebe, C. E., Marchette, D. J., DeVinney, J. G., & Socolinsky, D. A. (2003) · Journal of Classification · doi:10.1007/s00357-003-0003-7 · Cited by 59
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 59
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54
MDLFace: Memorability augmented deep learning for video face recognition
Goswami, G., Bhardwaj, R., Singh, R., & Vatsa, M. (2014) · IEEE International Joint Conference on Biometrics · doi:10.1109/btas.2014.6996299 · Cited by 53
Uncertainty-Aware Attention for Reliable Interpretation and Prediction
Huh, J. (2019) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 52
Ground-Truth Adversarial Examples
Carlini, N., Katz, G., Barrett, C., & Dill, D. L. (2018) · Cited by 49
Generating Textual Adversarial Examples for Deep Learning Models: A Survey.Preprint
Zhang, W. E., Sheng, Q. Z., & Alhazmi, A. (2019) · arXiv (Cornell University) · Cited by 40
Spectral Algorithms for Computing Fair Support Vector MachinesPreprint
Olfat, M., & Aswani, A. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05895 · Cited by 34
TED: Teaching AI to Explain its DecisionsPreprint
Hind, M., Wei, D., Campbell, M. S., Codella, N., Dhurandhar, A., Mojsilović, A., Ramamurthy, K. N., & Varshney, K. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04896 · Cited by 30
c-Eval: A Unified Metric to Evaluate Feature-based Explanations via Perturbation
Vu, M. N., Nguyen, T. D., Phan, N., Gera, R., & Thai, M. T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671895 · Cited by 27
Pham, T., & Shen, Y. (2017) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1706.02795 · Cited by 25
General theories of explanation: buyer beware
Díez, J. A., Khalifa, K., & Leuridan, B. (2011) · Synthese · doi:10.1007/s11229-011-0020-8 · Cited by 24
A Formal Approach to Explainability
Wolf, L., Galanti, T., & Hazan, T. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314260 · Cited by 22
To what extent do human explanations of model behavior align with actual model behavior?
Prasad, G., Nie, Y., Bansal, M., Jia, R., Kiela, D., & Williams, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.1 · Cited by 22
Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval
Ray, A., Yao, Y., Kumar, R., Divakaran, A., & Burachas, G. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5275 · Cited by 19
Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?Preprint
Baan, J., ter Hoeve, M., van der Wees, M., Schuth, A., & de Rijke, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00570 · Cited by 17
On Learning Sparse Boolean Formulae for Explaining AI Decisions
Jha, S., Raman, V., Pinto, A., Sahai, T., & Francis, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-57288-8_7 · Cited by 15
Hidden Trigger Backdoor Attacks
Saha, A., Subramanya, A., & Pirsiavash, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6871 · Cited by 13
Relating Input Concepts to Convolutional Neural Network DecisionsPreprint
Xie, N., Sarker, M. K., Doran, D., Hitzler, P., & Raymer, M. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08006 · Cited by 9
Self-citationOptimal Transport Classifier: Defending Against Adversarial Attacks by Regularized Deep EmbeddingPreprint
Li, Y., Min, M. R., Yu, W., Hsieh, C., Lee, T. C. M., & Kruus, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07950 · Cited by 9
Neural Stethoscopes: Unifying Analytic, Auxiliary and Adversarial Network Probing.Preprint
Fuchs, F. B., Groth, O., Kosiorek, A. R., Bewley, A., Wulfmeier, M., Vedaldi, A., & Posner, I. (2018) · arXiv (Cornell University) · Cited by 6
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Podobnik, B., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 643
6G networks: Beyond Shannon towards semantic and goal-oriented communications
Calvanese Strinati, E., & Barbarossa, S. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.107930 · Cited by 525
Image-Based Surface Defect Detection Using Deep Learning: A Review
Bhatt, P. M., Malhan, R. K., Rajendran, P., Shah, B. C., Thakar, S., Yoon, Y. J., & Gupta, S. K. (2021) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4049535 · Cited by 362
A comprehensive review on deep learning-based methods for video anomaly detection
Nayak, R., Pati, U. C., & Das, S. K. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104078 · Cited by 324
Metadata
- Authors
- Ras, G., Xie, N., Van Gerven, M., & Doran, D.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2022-01-25
- Volume
- 73
- Pages
- 329-397
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3020975691
- Cited by
- 446
- References
- 245
- FWCI
- 26.66
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.