Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Open accessAbstract
Deep learning has recently seen rapid development and received significant attention due to its state-of-the-art performance on previously-thought hard problems. However, because of the internal complexity and nonlinear structure of deep neural networks, the underlying decision making processes for why these models are achieving such performance are challenging and sometimes mystifying to interpret. As deep learning spreads across domains, it is of paramount importance that we equip users of deep learning with tools for understanding when a model works correctly, when it fails, and ultimately how to improve its performance. Standardized toolkits for building neural networks have helped democratize deep learning; visual analytics systems have now been developed to support model explanation, interpretation, debugging, and improvement. We present a survey of the role of visual analytics in deep learning research, which highlights its short yet impactful history and thoroughly summarizes the state-of-the-art using a human-centered interrogative framework, focusing on the Five W's and How (Why, Who, What, How, When, and Where). We conclude by highlighting research directions and open research problems. This survey helps researchers and practitioners in both visual analytics and deep learning to quickly learn key aspects of this young and rapidly growing body of research, whose impact spans a diverse range of domains.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Selecting an appropriate XAI method for a given task is challenging because of subjective end-user perceptions, individual needs, and sparse assessment guidelines.
“Because of that, selecting an appropriate XAI method for the task at hand remains a challenge, especially considering the subjective perceptions of end-users alongside their individual needs [18,19] and relatively sparse assessment guidelines.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,048 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · see citing works
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · see citing works
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,123 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
NIPS 2016 Tutorial: Generative Adversarial NetworksPreprint
Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.00160 · Cited by 1,735 · see citing works
Understanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · see citing works
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
Generative Visual Manipulation on the Natural Image Manifold
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_36 · Cited by 1,215 · see citing works
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164 · see citing works
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880 · see citing works
McMahan, H. B., Holt, G., Sculley, D., Young, M., Ebner, D., Grady, J., Nie, L., Phillips, T., Davydov, E. V., Golovin, D., et al. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488200 · Cited by 870 · see citing works
Wattenberg, M., Viégas, F., & Johnson, I. (2016) · Distill · doi:10.23915/distill.00002 · Cited by 854 · see citing works
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
On Detecting Adversarial PerturbationsPreprint
Metzen, J. H., Genewein, T., Fischer, V., & Bischoff, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04267 · Cited by 747 · see citing works
Towards Deep Neural Network Architectures Robust to Adversarial ExamplesPreprint
Gu, S., & Rigazio, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5068 · Cited by 658 · see citing works
Inverting Visual Representations with Convolutional Networks
Dosovitskiy, A., & Brox, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.522 · Cited by 631 · see citing works
word2vec Parameter Learning Explained
Rong, X. (2014) · arXiv · doi:10.48550/arxiv.1411.2738 · Cited by 626 · see citing works
The Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591 · see citing works
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · see citing works
Inceptionism: Going Deeper into Neural Networks
Mordvintsev, A., Olah, C., & Tyka, M. (2015) · Cited by 569 · see citing works
Towards Better Analysis of Deep Convolutional Neural Networks
Liu, M., Shi, J., Li, Z., Li, C., Zhu, J., & Liu, S. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598831 · Cited by 493 · see citing works
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 487 · see citing works
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · see citing works
Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?
Das, A., Agrawal, H., Zitnick, C. L., Parikh, D., & Batra, D. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1092 · Cited by 431 · see citing works
Visualizing high-dimensional data: advances in the past decade
Liu, S., Maljovec, D., Wang, B., Bremer, P.-T., & Pascucci, V. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2640960 · Cited by 427 · see citing works
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 399 · see citing works
Towards better analysis of machine learning models: A visual analytics perspective
Liu, S., Wang, X., Liu, M., & Zhu, J. (2017) · Visual Informatics · doi:10.1016/j.visinf.2017.01.006 · Cited by 397 · see citing works
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P., Kalro, A., & Chau, D. H. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 349 · see citing works
Self-citationVisualizing Dataflow Graphs of Deep Learning Models in TensorFlow
Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mané, D., Fritz, D., Krishnan, D., Viégas, F. B., & WATTENBERG, M. P. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744878 · Cited by 332 · see citing works
Learning how to explain neural networks: PatternNet and PatternAttributionPreprint
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K. R., Erhan, D., Kim, B., & Dähne, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05598 · Cited by 325 · see citing works
HOGgles: Visualizing Object Detection Features
Vondrick, C., Khosla, A., Malisiewicz, T., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.8 · Cited by 282 · see citing works
Visualizing the Hidden Activity of Artificial Neural Networks
Rauber, P. E., Fadel, S. G., Falcao, A. X., & Telea, A. C. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598838 · Cited by 280 · see citing works
Visual Analytics for Explainable Deep Learning
Choo, J., & Liu, S. (2018) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2018.042731661 · Cited by 268 · see citing works
Nguyen, A. M., Yosinski, J., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03616 · Cited by 248 · see citing works
Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG CompressionPreprint
Das, N., Shanbhogue, M., Chen, S.-T., Hohman, F., Chen, L., Kounavis, M. E., & Chau, D. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02900 · Cited by 241 · see citing works
Self-citationUnderstanding Hidden Memories of Recurrent Neural Networks
Ming, Y., Cao, S., Zhang, R., Li, Z., Chen, Y., Song, Y., & Qu, H. (2017) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585721 · Cited by 214 · see citing works
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200 · see citing works
Do Convolutional Neural Networks Learn Class Hierarchy?
Alsallakh, B., Jourabloo, A., Ye, M., Liu, X., & Ren, L. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744683 · Cited by 199 · see citing works
Embedding Projector: Interactive Visualization and Interpretation of EmbeddingsPreprint
Smilkov, D., Thorat, N., Nicholson, C., Reif, E., Viégas, F. B., & WATTENBERG, M. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05469 · Cited by 190 · see citing works
Graying the black box: Understanding DQNsPreprint
Zahavy, T., Ben-Zrihem, N., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02658 · Cited by 181 · see citing works
Visualizing and Understanding Neural Models in NLPPreprint
Li, J., Chen, X., Hovy, E. H., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01066 · Cited by 165 · see citing works
DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks
Pezzotti, N., Höllt, T., van Gemert, J. C., Lelieveldt, B. P. F., Eisemann, E., & Vilanova, A. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744358 · Cited by 165 · see citing works
Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent SpacePreprint
Nguyen, A. Q., Clune, J., Bengio, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00005 · Cited by 159 · see citing works
Analyzing the Training Processes of Deep Generative Models
Liu, M., Shi, J., Cao, K., et al. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744938 · Cited by 156 · see citing works
Das, N., Shanbhogue, M., Chen, S.-T., Hohman, F., Li, S., Chen, L., Kounavis, M. E., & Chau, D. H. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219910 · Cited by 154 · see citing works
Self-citationUsing Artificial Intelligence to Augment Human Intelligence
Carter, S., & Nielsen, M. (2017) · Distill · doi:10.23915/distill.00009 · Cited by 141 · see citing works
Visualizing Time-Dependent Data Using Dynamic t-SNE
Rauber, P., Falcão, A. X., & Telea, A. C. (2016) · Queen Mary Research Online (Queen Mary University of London) · doi:10.2312/eurovisshort.20161164 · Cited by 137 · see citing works
Investigating statistical machine learning as a tool for software development
Patel, K., Fogarty, J. A., Landay, J. A., & Harrison, B. L. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357160 · Cited by 131 · see citing works
Wall, E., Blaha, L. M., Franklin, L. R., & Endert, A. (2017) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585669 · Cited by 129 · see citing works
Visual Exploration of Semantic Relationships in Neural Word Embeddings
Liu, S., Bremer, P., Thiagarajan, J. J., et al. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745141 · Cited by 123 · see citing works
Cognitive Psychology for Deep Neural Networks: A Shape Bias Case StudyPreprint
Ritter, S., Barrett, D. G. T., Santoro, A., & Botvinick, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.08606 · Cited by 114 · see citing works
An Interactive Node-Link Visualization of Convolutional Neural Networks
Harley, A. W. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27857-5_77 · Cited by 112 · see citing works
Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationPreprint
Johnson, M., Schuster, M., Le, Q. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04558 · Cited by 109 · see citing works
The State‐of‐the‐Art in Predictive Visual Analytics
Lu, Y., Garcia, R., Hansen, B., et al. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13210 · Cited by 98 · see citing works
Opening the Black Box - Data Driven Visualization of Neural Network
Tzeng, F.-Y., & Ma, K. (2006) · doi:10.1109/vis.2005.73 · Cited by 96 · see citing works
A Deep Learning Approach for Population Estimation from Satellite Imagery
Robinson, C., Hohman, F., & Dilkina, B. (2017) · doi:10.1145/3149858.3149863 · Cited by 96 · see citing works
Self-citation(2017) · arXiv (Cornell University) · Cited by 86 · see citing works
GANViz: A Visual Analytics Approach to Understand the Adversarial Game
Wang, J., Gou, L., Yang, H., et al. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2816223 · Cited by 86 · see citing works
An Analysis of Machine- and Human-Analytics in Classification
Tam, G. K. L., Kothari, V., & Chen, M. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598829 · Cited by 85 · see citing works
Direct-Manipulation Visualization of Deep NetworksPreprint
Smilkov, D., Carter, S., Sculley, D., Viégas, F. B., & WATTENBERG, M. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03788 · Cited by 84 · see citing works
ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word Embedding
Park, D., Kim, S., Lee, J., Choo, J., Diakopoulos, N. A., & Elmqvist, N. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744478 · Cited by 82 · see citing works
Understanding Intra-Class Knowledge Inside CNNPreprint
Wei, D., Zhou, B., Torrabla, A., & Freeman, W. T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02379 · Cited by 79 · see citing works
Visual exploration of machine learning results using data cube analysis
Kahng, M., Fang, D., & Chau, D. H. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939503 · Cited by 78 · see citing works
Self-citationVisualizations of Deep Neural Networks in Computer Vision: A Survey
Seifert, C., Aamir, A., Balagopalan, A., Jain, D., Sharma, A., Grottel, S., & Gumhold, S. (2017) · Studies in big data · doi:10.1007/978-3-319-54024-5_6 · Cited by 70 · see citing works
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
Nushi, B., Kamar, E. S., Horvitz, E., & Kossmann, D. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10633 · Cited by 65 · see citing works
Recent progress and trends in predictive visual analytics
Lu, J., Chen, W., Ma, Y., Ke, J., Li, Z., Zhang, F., & Maciejewski, R. (2016) · Frontiers of Computer Science · doi:10.1007/s11704-016-6028-y · Cited by 63 · see citing works
Task-Driven Comparison of Topic Models
Alexander, E., & Gleicher, M. L. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467618 · Cited by 61 · see citing works
VISUALIZING LEARNING AND COMPUTATION IN ARTIFICIAL NEURAL NETWORKS
Craven, M. W., & SHAVLIK, J. W. (1992) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213092000260 · Cited by 58 · see citing works
VisualBackProp: visualizing CNNs for autonomous driving.Preprint
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Jackel, L., Müller, U., & Zieba, K. (2016) · arXiv (Cornell University) · Cited by 56 · see citing works
Shield: Fast, Practical Defense and Vaccination for Deep Learning using JPEG CompressionPreprint
Das, N., Shanbhogue, M., Chen, S.-T., Hohman, F., Li, S., Chen, L., Kounavis, M. E., & Chau, D. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06816 · Cited by 53 · see citing works
Self-citationVisualizing and Comparing Convolutional Neural NetworksPreprint
Yu, W., Yang, K., Bai, Y., Yao, H., & Rui, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6631 · Cited by 49 · see citing works
A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
Grün, F., Rupprecht, C., Navab, N., & Tombari, F. (2016) · arXiv · doi:10.48550/arxiv.1606.07757 · Cited by 49 · see citing works
Towards Transparent AI Systems: Interpreting Visual Question Answering ModelsPreprint
Goyal, Y., Mohapatra, A., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.08974 · Cited by 48 · see citing works
VisualBackProp: efficient visualization of CNNsPreprint
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Jackel, L., Müller, U., & Zieba, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05418 · Cited by 47 · see citing works
Li, H., Tian, Y., Mueller, K. D., & Chen, X. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.02.005 · Cited by 47 · see citing works
Adversarial-Playground: A visualization suite showing how adversarial examples fool deep learning
Norton, A. P., & Qi, Y. (2017) · IEEE Symposium on Visualization for Cyber Security (VIZSEC) · doi:10.1109/vizsec.2017.8062202 · Cited by 43 · see citing works
CNNComparator: Comparative Analytics of Convolutional Neural NetworksPreprint
Zeng, H., Haleem, H., Plantaz, X., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05285 · Cited by 27 · see citing works
Intelligible Artificial Intelligence
Weld, D. S., & Bansal, G. (2018) · Cited by 26 · see citing works
NVIS: an interactive visualization tool for neural networks
Streeter, M. J. V., Ward, M. O., & Álvarez, S. A. (2001) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.424934 · Cited by 23 · see citing works
Human-centered machine learning through interactive visualization
Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Weiskopf, D., North, S. C., & Keim, D. A. (2016) · Middlesex University Research Repository (Middlesex University Of London) · Cited by 23 · see citing works
ML-o-Scope: A Diagnostic Visualization System for Deep Machine Learning Pipelines
Brückner, D. (2014) · doi:10.21236/ada605112 · Cited by 16 · see citing works
Hohman, F., Hodas, N. O., & Chau, D. H. (2017) · doi:10.1145/3027063.3053103 · Cited by 16 · see citing works
Self-citationOlah, C., & Carter, S. (2017) · Distill · doi:10.23915/distill.00005 · Cited by 16 · see citing works
Experiments in Handwriting with a Neural Network
Carter, S., Ha, D., Johnson, I., & Olah, C. (2016) · Distill · doi:10.23915/distill.00004 · Cited by 15 · see citing works
Interpreting Visual Question Answering Models.Preprint
Goyal, Y., Mohapatra, A., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · Cited by 14 · see citing works
"I know it when I see it". Visualization and Intuitive InterpretabilityPreprint
Offert, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08042 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- [26]Goodfellow I, Bengio Y, and Courville A, Deep Learning MIT Press, 2016, http://www.deeplearningbook.org.
- [33]Cashman D, Patterson G, Mosca A, and Chang R, “ RNNbow: Visualizing learning via backpropagation gradients in recurrent neural networks,” in Workshop on Visual Analytics for Deep Learning, 2017.
- [34]Chae J, Gao S, Ramanthan A, Steed C, and Tourassi GD, “ Visualization for classification in deep neural networks,” in Workshop on Visual Analytics for Deep Learning, 2017.
- [35]Chung S, Park C, Suh S, Kang K, Choo J, and Kwon BC, “ ReVACNN: Steering convolutional neural network via real-time visual analytics,” in NIPS Workshop on Future of Interactive Learning Machines, 2016.
- [50]Rong X and Adar E, “ Visual tools for debugging neural language models,” in ICML Workshop on Vis for Deep Learning, 2016.
- [58]Zhong W, Xie C, Zhong Y, Wang Y, Xu W, Cheng S, and Mueller K, “ Evolutionary visual analysis of deep neural networks,” in ICML Workshop on Vis for Deep Learning, 2017.
- [67]Li L, Tompkin J, Michalatos P, and Pfister H, “ Hierarchical visual feature analysis for city street view datasets,” in Workshop on Visual Analytics for Deep Learning, 2017.
- [68]Erhan D, Bengio Y, Courville A, and Vincent P, “ Visualizing higher-layer features of a deep network,” University of Montreal, vol. 1341, 2009.
- [69]Selvaraju RR, Das A, Vedantam R, Cogswell M, Parikh D, and Batra D, “ Grad-CAM: Why did you say that? visual explanations from deep networks via gradient-based localization,” arXiv:1610.02391, 2016. · arXiv
- [70]Nguyen A, Dosovitskiy A, Yosinski J, Brox T, and Clune J, “ Synthesizing the preferred inputs for neurons in neural networks via deep generator networks,” in NIPS, 2016.
- [106]Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y, “ Generative adversarial nets,” in NIPS, 2014.
- [111]Tsai C-Y and Cox DD, “ Characterizing visual representations within convolutional neural networks: Toward a quantitative approach,” ICML Workshop on Vis for Deep Learning, 2016.
Cited by
Sorted by citations
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · see citing works
Solving the Black Box Problem: A Normative Framework for Explainable Artificial Intelligence
Zednik, C. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00382-7 · Cited by 419 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · see citing works
Deep convolutional neural networks for mammography: advances, challenges and applications
Abdelhafiz, D., Yang, C., Ammar, R., & Nabavi, S. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2823-4 · Cited by 274 · see citing works
Integrating Machine Learning with Human Knowledge
Deng, C., Ji, X., Rainey, C., Zhang, J., & Lu, W. (2020) · iScience · doi:10.1016/j.isci.2020.101656 · Cited by 243 · see citing works
A survey of visual analytics for Explainable Artificial Intelligence methods
Alicioglu, G., & Sun, B. (2022) · Computers & Graphics · doi:10.1016/j.cag.2021.09.002 · Cited by 221 · see citing works
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
How do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · see citing works
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126 · see citing works
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117 · see citing works
Understanding and Visualizing Data Iteration in Machine Learning
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376177 · Cited by 110 · see citing works
Self-citationAnchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450639 · Cited by 107 · see citing works
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78 · see citing works
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67 · see citing works
VisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30 · see citing works
How Transferable are Reasoning Patterns in VQA?
Kervadec, C., Jaunet, T., Antipov, G., Baccouche, M., Vuillemot, R., & Wolf, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00419 · Cited by 25 · see citing works
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511112 · Cited by 19 · see citing works
“To address this latter issue, researchers have actively explored a method or tool to provide explanations on the predictions of anML-based system [25, 40, 43, 50].”
MethodFAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · see citing works
Self-citation
- Authors
- Hohman, F., Kahng, M., Pienta, R., & Chau, D. H.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-08-01
- Volume
- 25
- Issue
- 8
- Pages
- 2674-2693
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- PMC
- PMC6703958
- OpenAlex
- W2784769278
- Cited by
- 582
- References
- 117
- Citations by year
- 2026: 31 · 2025: 39 · 2024: 58 · 2023: 84 · 2022: 84 · 2021: 107 · 2020: 102 · 2019: 54 · 2018: 23
- FWCI
- 35.76
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (IIS-1563816, CNS-1704701, TWC-1526254, DGE-1650044)
- NIBIB (U54EB020404)
- NASA
- Intel Corporation
- Symantec
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Keywords
- visual analytics, deep learning, model interpretability, model explanation, human-centered design, open problems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
