Network Dissection: Quantifying Interpretability of Deep Visual Representations
Open accessAbstract
We propose a general framework called Network Dissection for quantifying the interpretability of latent representations of CNNs by evaluating the alignment between individual hidden units and a set of semantic concepts. Given any CNN model, the proposed method draws on a data set of concepts to score the semantics of hidden units at each intermediate convolutional layer. The units with semantics are labeled across a broad range of visual concepts including objects, parts, scenes, textures, materials, and colors. We use the proposed method to test the hypothesis that interpretability is an axis-independent property of the representation space, then we apply the method to compare the latent representations of various networks when trained to solve different classification problems. We further analyze the effect of training iterations, compare networks trained with different initializations, and measure the effect of dropout and batch normalization on the interpretability of deep visual representations. We demonstrate that the proposed method can shed light on characteristics of CNN models and training methods that go beyond measurements of their discriminative power.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Broden, objects, object parts and materials are segmented down to pixel level while textures and scenes are annotated at image level.
“Objects, object parts and materials are segmented down to pixel level while textures and scenes are annotated at image level.”
From Unified Perceptual Parsing for Scene Understanding (2018) · cites this paper as [26]
Broden was originally collected to discover the alignment between visual concepts and hidden units of CNNs for network interpretability.
“since it is originally collected to discover the alignment between visual concepts and hidden units of Convolutional Neural Networks (CNNs) for network interpretability [26,29]”
From Unified Perceptual Parsing for Scene Understanding (2018) · cites this paper as [26]
Broden samples from different classes are unbalanced.
“we find that samples from different classes are unbalanced.”
From Unified Perceptual Parsing for Scene Understanding (2018) · cites this paper as [26]
Network Dissection is a framework for quantifying the interpretability of individual units within a neural network.
“A somewhat different approach called Network Dissection [26] is a general framework designed to quantify the interpretability of individual units (e.g., neurons or channels) within a neural network.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Network Dissection authors confirmed that representations at different layers disentangle distinct categories of meaning and that different training techniques can significantly impact the representation learned by hidden units.
“With this technique, the authors confirmed that representations at different layers disentangle distinct categories of meaning and that different training techniques can significantly impact the representation learned by hidden units.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Network Dissection detects concepts beyond classes in classification tasks.
“Detection of concepts beyond classes in classification task”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Network Dissection considers only positive activations.
“Only positive activations are considered”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Network Dissection depends on datasets with pixel-level labels.
“Dependence on datasets with labels on pixel-level”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Network Dissection communicates inner workings in a nontechnical way.
“Communication of inner workings in a nontechnical way”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Self-citation(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
Scene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326 · see citing works
Self-citationUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · see citing works
Invariant visual representation by single neurons in the human brain
Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I. (2005) · Nature · doi:10.1038/nature03687 · Cited by 1,988 · see citing works
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850 · see citing works
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Generating Videos with Scene DynamicsPreprint
Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02612 · Cited by 976 · see citing works
Self-citationShuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
Zhang, R., Isola, P., & Efros, A. A. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.76 · Cited by 724 · see citing works
Learning Color Names for Real-World Applications
van de Weijer, J., Schmid, C., Verbeek, J. J., & Larlus, D. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2019809 · Cited by 710 · see citing works
Agrawal, P., Carreira, J., & Malik, J. (2015) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2015.13 · Cited by 639 · see citing works
Human action recognition by learning bases of action attributes and parts
Yao, B., Jiang, X., Khosla, A., Lin, A. L., Guibas, L. J., & Fei-Fei, L. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126386 · Cited by 631 · see citing works
Self-citationCNN Features off-the-shelf: an Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · arXiv · doi:10.48550/arxiv.1403.6382 · Cited by 607 · see citing works
Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.254 · Cited by 605 · see citing works
Understanding intermediate layers using linear classifier probes
Alain, G., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1610.01644 · Cited by 581 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 465 · see citing works
Self-citationSynthesizing the preferred inputs for neurons in neural networks via deep generator networksPreprint
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09304 · Cited by 435 · see citing works
Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Agrawal, P., Girshick, R., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_22 · Cited by 427 · see citing works
Bell, S., Bala, K., & Snavely, N. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601206 · Cited by 413 · see citing works
Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_48 · Cited by 385 · see citing works
Self-citationUnsupervised Visual Representation Learning by Context PredictionPreprint
Doersch, C., Gupta, A., & Efros, A. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05192 · Cited by 245 · see citing works
Learning Image Representations Tied to Ego-Motion
Jayaraman, D., & Grauman, K. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.166 · Cited by 229 · see citing works
Unsupervised Learning of Visual Representations using VideosPreprint
Wang, X., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00687 · Cited by 194 · see citing works
Places: An Image Database for Deep Scene UnderstandingPreprint
Zhou, B., Khosla, A., Lapedriza, À., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02055 · Cited by 173 · see citing works
Self-citationConvergent Learning: Do different neural networks learn the same representations?Preprint
Li, Y., Yosinski, J., Clune, J., Lipson, H., & Hopcroft, J. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07543 · Cited by 169 · see citing works
Learning to See by MovingPreprint
Agrawal, P., Carreira, J., & Malik, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.01596 · Cited by 138 · see citing works
Detect What You Can: Detecting and Representing Objects using Holistic Models and Body PartsPreprint
Chen, X., Mottaghi, R., Liu, X., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2031 · Cited by 91 · see citing works
Analyzing the Performance of Multilayer Neural Networks for Object RecognitionPreprint
Agrawal, P., Girshick, R., & Malik, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.1610 · Cited by 71 · see citing works
Object-Centric Representation Learning from Unlabeled VideosPreprint
Gao, R., Jayaraman, D., & Grauman, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00500 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [8][8] P. Diaconis. What is a random matrix? Notices of the AMS , 52(11):1348-1349, 2005. -
- [11][11] A. Gonzalez-Garcia, D. Modolo, and V. Ferrari. Do semantic parts emerge in convolutional neural networks? arXiv:1607.03738 , 2016. - · arXiv
- [15][15] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012. -
Cited by
Sorted by citations
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909 · see citing works
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works
Self-citationUnified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · see citing works
“1 Datasets In order to accomplish segmentation of a wide range of visual concepts from multiple levels, we utilize the Broadly and Densely Labeled Dataset (Broden) [26], a heterogeneous dataset that contains various visual concepts.”
BackgroundMethodSelf-citationLearning the signatures of the human grasp using a scalable tactile glove
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A., & Matusik, W. (2019) · Nature · doi:10.1038/s41586-019-1234-z · Cited by 1,162 · see citing works
Self-citationExplainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983 · see citing works
Pros and cons of GAN evaluation measures
Borji, A. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.10.009 · Cited by 963 · see citing works
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880 · see citing works
Interpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833 · see citing works
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
Salahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508 · see citing works
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · see citing works
Dimiduk, D. M., Holm, E. A., & Niezgoda, S. R. (2018) · Integrating Materials and Manufacturing Innovation · doi:10.1007/s40192-018-0117-8 · Cited by 347 · see citing works
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · see citing works
Zoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288 · see citing works
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227 · see citing works
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
A Survey of Data-driven and Knowledge-aware eXplainable AI
Li, X.-H., Cao, C. C., Shi, Y., Bai, W., Gao, H., Qiu, L., Wang, C., Gao, Y., Zhang, S., Xue, X., et al. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2983930 · Cited by 224 · see citing works
Explainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216 · see citing works
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211 · see citing works
Wang, T., Zhu, J.-Y., Torralba, A., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1811.10959 · Cited by 194 · see citing works
Self-citationFrom attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181 · see citing works
Interpretation and visualization techniques for deep learning models in medical imaging
Huff, D. T., Weisman, A. J., & Jeraj, R. (2021) · Physics in Medicine & Biology · doi:10.1088/1361-6560/abcd17 · Cited by 172 · see citing works
Looking beyond the hype: applied AI and machine learning in translational medicine
Toh, T. S., Dondelinger, F., & Wang, D. (2019) · EBioMedicine · doi:10.1016/j.ebiom.2019.08.027 · Cited by 159 · see citing works
Visual Imagery and Perception Share Neural Representations in the Alpha Frequency Band
Xie, S., Kaiser, D., & Cichy, R. M. (2020) · Current Biology · doi:10.1016/j.cub.2020.04.074 · Cited by 139 · see citing works
Interpretable Anomaly Detection with DIFFI: Depth-based feature importance of Isolation Forest
Carletti, M., Terzi, M., & Susto, G. A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105730 · Cited by 118 · see citing works
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Toward Trustworthy Artificial Intelligence (TAI) in the Context of Explainability and Robustness
chander, B., John, C., Warrier, L., & Gopalakrishnan, K. (2025) · ACM Computing Surveys · doi:10.1145/3675392 · Cited by 108 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis
Natekar, P., Kori, A., & Krishnamurthi, G. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00006 · Cited by 103 · see citing works
Condran, S., Bewong, M., Islam, M. Z., Maphosa, L., & Zheng, L. (2022) · IEEE Access · doi:10.1109/access.2022.3188649 · Cited by 99 · see citing works
Kortylewski, A., Liu, Q., Wang, A., Sun, Y., & Yuille, A. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01401-3 · Cited by 98 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Predicting cellular responses to perturbation across diverse contexts with State
Adduri, A. K., Gautam, D., Bevilacqua, B., Naghipourfar, M., Imran, A., Shah, R., Teyssier, N., Verma, R., Carpenter, C., Eraslan, B., et al. (2026) · Cell · doi:10.1016/j.cell.2026.07.052 · Cited by 100 · see citing works
Global Model Interpretation Via Recursive Partitioning
Yang, C., Rangarajan, A., & Ranka, S. (2018) · 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc/smartcity/dss.2018.00256 · Cited by 93 · see citing works
Smith, C. E., Yu, B., Srivastava, A., Halfaker, A., Terveen, L., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376783 · Cited by 77 · see citing works
A Disentangling Invertible Interpretation Network for Explaining Latent Representations
Esser, P., Rombach, R., & Ommer, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00924 · Cited by 72 · see citing works
Faithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73 · see citing works
HIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72 · see citing works
Explaining Neural Networks Semantically and Quantitatively
Chen, R., Chen, H., Huang, G., Ren, J., & Zhang, Q. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00928 · Cited by 56 · see citing works
Evaluating the Interpretability of Generative Models by Interactive Reconstruction
Ross, A., Chen, N., Hang, E. Z., Glassman, E. L., & Doshi-Velez, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445296 · Cited by 40 · see citing works
Relation-Based Counterfactual Explanations for Bayesian Network Classifiers
Albini, E., Rago, A., Baroni, P., & Toni, F. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/63 · Cited by 39 · see citing works
Emergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks
Mohsenzadeh, Y., Mullin, C., Lahner, B., & Oliva, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-61409-0 · Cited by 31 · see citing works
Self-citationDalal, S., Tilwani, D., Gaur, M., Jain, S., Shalin, V. L., & Sheth, A. P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483577 · Cited by 19 · see citing works
Deep active learning with Weighting filter for object detection
Huang, W., Sun, S., Lin, X., Zhang, D., & Ma, L. (2023) · Displays · doi:10.1016/j.displa.2022.102282 · Cited by 11 · see citing works
Sources of Opacity in Computer Systems: Towards a Comprehensive Taxonomy
Mann, S., Crook, B., Kästner, L., Schomäcker, A., & Speith, T. (2023) · 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00063 · Cited by 10 · see citing works
Neural networks memorise personal information from one sample
Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-48034-3 · Cited by 10 · see citing works
- Authors
- Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 3319-3327
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2610018085
- Cited by
- 1,389
- References
- 48
- Citations by year
- 2026: 63 · 2025: 130 · 2024: 156 · 2023: 154 · 2022: 156 · 2021: 224 · 2020: 194 · 2019: 166 · 2018: 120 · 2017: 25
- FWCI
- 62.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1524817)
- Nvidia
- Toyota Research Institute
- Office of Naval Research (N00014-16-1-3116, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- interpretability, convolutional neural networks, hidden units, semantic concepts, latent representations, training iterations, initialization, dropout, batch normalization, object recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
