The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Visual interpretability for deep learning: a survey
Open accessAbstract
This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, interpretability is always Achilles’ heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of a low interpretability of their black-box representations. We believe that high model interpretability may help people break several bottlenecks of deep learning, e.g., learning from a few annotations, learning via human–computer communications at the semantic level, and semantically debugging network representations. We focus on convolutional neural networks (CNNs), and revisit the visualization of CNN representations, methods of diagnosing representations of pre-trained CNNs, approaches for disentangling pre-trained CNN representations, learning of CNNs with disentangled representations, and middle-to-end learning based on model interpretability. Finally, we discuss prospective trends in explainable artificial intelligence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Cited by
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Artificial intelligence in cancer imaging: Clinical challenges and applications
Bi, W. L., Hosny, A., Schabath, M. B., Giger, M. L., Birkbak, N. J., Mehrtash, A., Allison, T., Arnaout, O., Abbosh, C., Dunn, I. F., et al. (2019) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21552 · Cited by 1,846
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983
A Survey on Neural Network Interpretability
Zhang, Y., Tino, P., Leonardis, A., & Tang, K. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3100641 · Cited by 923
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
A Comprehensive Survey on Graph Anomaly Detection With Deep Learning
Ma, X., Wu, J., Xue, S., Yang, J., Zhou, C., Sheng, Q. Z., Xiong, H., & Akoglu, L. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3118815 · Cited by 835
Interpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833
Self-citationA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Big data analytics for intelligent manufacturing systems: A review
Wang, J., Xu, C., Zhang, J., & Zhong, R. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2021.03.005 · Cited by 603
Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study
Hosny, A., Parmar, C., Coroller, T. P., Grossmann, P., Zeleznik, R., Kumar, A., Bussink, J., Gillies, R. J., Mak, R. H., & Aerts, H. J. W. L. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002711 · Cited by 579
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
On Interpretability of Artificial Neural Networks: A Survey
Fan, F.-L., Xiong, J., Li, M., & Wang, G. (2021) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2021.3066428 · Cited by 561
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Explainability Methods for Graph Convolutional Neural Networks
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01103 · Cited by 553
Real-world data: A brief review of the methods, applications, challenges and opportunities
Liu, F., & Panagiotakos, D. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01768-6 · Cited by 496
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
Fine-Grained Image Analysis With Deep Learning: A Survey
Wei, X.-S., Song, Y.-Z., Aodha, O. M., Wu, J., Peng, Y., Tang, J., Yang, J., & Belongie, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3126648 · Cited by 434
A cloud detection algorithm for satellite imagery based on deep learning
Jeppesen, J. H., Jacobsen, R. H., Inceoglu, F., & Toftegaard, T. S. (2019) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.03.039 · Cited by 402
Deep learning in bioinformatics: Introduction, application, and perspective in the big data era
Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., & Gao, X. (2019) · Methods · doi:10.1016/j.ymeth.2019.04.008 · Cited by 401
Li, J., Chen, J., Tang, Y., Wang, C., Landman, B. A., & Zhou, S. K. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102762 · Cited by 389
Deep learning in mental health outcome research: a scoping review
Su, C., Xu, Z., Pathak, J., & Wang, F. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-0780-3 · Cited by 368
Dual-Sampling Attention Network for Diagnosis of COVID-19 From Community Acquired Pneumonia
Ouyang, X., Huo, J., Xia, L., Shan, F., Liu, J., Mo, Z., Yan, F., Ding, Z., Yang, Q., Song, B., et al. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2995508 · Cited by 377
Human‐centered artificial intelligence and machine learning
Riedl, M. O. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.117 · Cited by 371
Deep learning for smart fish farming: applications, opportunities and challenges
Yang, X., Zhang, S., Liu, J., Gao, Q., Dong, S., & Zhou, C. (2021) · Reviews in Aquaculture · doi:10.1111/raq.12464 · Cited by 364
Apostolopoulos, I. D., Aznaouridis, S. I., & Tzani, M. A. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-020-00529-4 · Cited by 335
Interpreting CNNs via Decision Trees
Zhang, Q., Yang, Y., Ma, H., & Wu, Y. N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00642 · Cited by 330
Self-citationSchwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328
Explainable Artificial Intelligence in CyberSecurity: A Survey
Capuano, N., Fenza, G., Loia, V., & Stanzione, C. (2022) · IEEE Access · doi:10.1109/access.2022.3204171 · Cited by 330
Explainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 319
Wang, B., Li, Y., Zhou, M., Han, Y., Zhang, M., Gao, Z., Liu, Z., Chen, P., Du, W., Zhang, X., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-36017-x · Cited by 259
A survey of visual analytics techniques for machine learning
Yuan, J., Chen, C., Yang, W., Liu, M., Xia, J., & Liu, S. (2021) · Computational Visual Media · doi:10.1007/s41095-020-0191-7 · Cited by 258
Lu, S., Chai, H., Sahoo, A., & Phung, B. T. (2020) · IEEE Transactions on Dielectrics and Electrical Insulation · doi:10.1109/tdei.2020.009070 · Cited by 258
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 254
Causal interpretability for machine learning-problems, methods and evaluation
Moraffah, R., Karami, M., Guo, R., Raglin, A., & Liu, H. (2020) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3400051.3400058 · Cited by 238
Crop yield prediction integrating genotype and weather variables using deep learning
Shook, J., Gangopadhyay, T., Wu, L., Ganapathysubramanian, B., Sarkar, S., & Singh, A. K. (2021) · PLOS ONE · doi:10.1371/journal.pone.0252402 · Cited by 222
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227
Zhao, W., Yang, J., Sun, Y., Li, C., Wu, W., Jin, L., Yang, Z., Ni, B., Gao, P., Wang, P., et al. (2018) · Cancer Research · doi:10.1158/0008-5472.can-18-0696 · Cited by 226
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199
How do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks
Wagner, J., Köhler, J. M., Gindele, T., Hetzel, L., Wiedemer, J. T., & Behnke, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00931 · Cited by 144
Interpretation of deep learning in genomics and epigenomics
Talukder, A., Barham, C., Li, X., & Hu, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa177 · Cited by 142
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126
Qian, B., Su, J., Wen, Z., Jha, D. N., Li, Y., Guan, Y., Puthal, D., James, P., Yang, R., Zomaya, A. Y., et al. (2021) · ACM Computing Surveys · doi:10.1145/3398020 · Cited by 132
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124
Interpretable Anomaly Detection with DIFFI: Depth-based feature importance of Isolation Forest
Carletti, M., Terzi, M., & Susto, G. A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105730 · Cited by 118
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis
Natekar, P., Kori, A., & Krishnamurthi, G. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00006 · Cited by 103
Kortylewski, A., Liu, Q., Wang, A., Sun, Y., & Yuille, A. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01401-3 · Cited by 98
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Neural-symbolic integration and the Semantic Web
Hitzler, P., Bianchi, F., Ebrahimi, M., & Sarker, M. K. (2020) · Semantic Web · doi:10.3233/sw-190368 · Cited by 70
Explainable Activity Recognition for Smart Home Systems
Das, D., Nishimura, Y., Vivek, R. P., Takeda, N., Fish, S. T., Plötz, T., & Chernova, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3561533 · Cited by 70
Designing Co-Creative AI for Public Spaces
Long, D., Jacob, M., & Magerko, B. (2019) · Proceedings of the 2019 on Creativity and Cognition · doi:10.1145/3325480.3325504 · Cited by 70
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67
Mohamed, E., Sirlantzis, K., & Howells, G. (2022) · Displays · doi:10.1016/j.displa.2022.102239 · Cited by 67
Leveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52
“The methods of interpretability can vary based on the complexity of a model and a wide range of survey papers summarize the different XAI models currently developed [2, 26, 34].”
Explainable AI for the Choquet Integral
Murray, B. J., Islam, M. A., Pinar, A. J., Anderson, D. T., Scott, G. J., Havens, T. C., & Keller, J. M. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2020.3005682 · Cited by 36
A computational model of language functions in flexible goal-directed behaviour
Granato, G., Borghi, A. M., & Baldassarre, G. (2020) · Scientific Reports · doi:10.1038/s41598-020-78252-y · Cited by 34
Benchmarking protein language models for protein crystallization
Mall, R., Kaushik, R., Martinez, Z. A., Thomson, M. W., & Castiglione, F. (2025) · Scientific Reports · doi:10.1038/s41598-025-86519-5 · Cited by 20
Scalable anomaly detection in manufacturing systems using an interpretable deep learning approach
Schlegl, T., Schlegl, S., West, N., & Deuse, J. (2021) · Procedia CIRP · doi:10.1016/j.procir.2021.11.261 · Cited by 15
Metadata
- Authors
- Zhang, Q.-S., & Zhu, S.-C.
- Venue
- Frontiers of Information Technology & Electronic Engineering
- Publisher
- Zhejiang University Press
- Published
- 2018-01
- Volume
- 19
- Issue
- 1
- Pages
- 27-39
- ISSN
- 2095-9184 (ISSN-L)
- 2095-9230
- OpenAlex
- W2963374347
- Cited by
- 950
- References
- 45
- FWCI
- 60.40
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.