A task-and-technique centered survey on visual analytics for deep learning model engineering
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Although deep neural networks have achieved state-of-the-art performance in several artificial intelligence applications in the past decade, they are still hard to understand. In particular, the features learned by deep networks when determining whether a given input belongs to a specific class are only implicitly described concerning a considerable number of internal model parameters. This makes it harder to construct interpretable hypotheses of what the network is learning and how it is learning—both of which are essential when designing and improving a deep model to tackle a particular learning task. This challenge can be addressed by the use of visualization tools that allow machine learning experts to explore which components of a network are learning useful features for a pattern recognition task, and also to identify characteristics of the network that can be changed to improve its performance. We present a review of modern approaches aiming to use visual analytics and information visualization techniques to understand, interpret, and fine-tune deep learning models. For this, we propose a taxonomy of such approaches based on whether they provide tools for visualizing a network’s architecture, to facilitate the interpretation and analysis of the training process, or to allow for feature understanding. Next, we detail how these approaches tackle the tasks above for three common deep architectures: deep feedforward networks, convolutional neural networks, and recurrent neural networks. Additionally, we discuss the challenges faced by each network architecture and outline promising topics for future research in visualization techniques for deep learning models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258
Research works printed but not linked: 63
- Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. CoRR 2014;abs/1409.1556. arXiv:1409.1556. · arXiv
- Liu, 2017, Towards better analysis of deep convolutional neural networks, IEEE Trans Vis Comput Gr, 23, 91, 10.1109/TVCG.2016.2598831 · DOI · Paper page
- Marcus G. Deep learning: A critical appraisal. CoRR 2018;abs/1801.00631. arXiv:1801.00631. · arXiv
- van der Maaten, 2009, Dimensionality reduction: a comparative review, Mach Learn Res, 10, 66
- Cunningham, 2015, Linear dimensionality reduction: Survey, insights, and generalizations, J Mach Learn Res, 16, 2859
- Sorzano C, Vargas J, Montano AP. A survey of dimensionality reduction techniques. CoRR 2014;abs/1403.2877. arXiv:1403.2877. · arXiv
- Blum, 2013, A comparative review of dimension reduction methods in approximate Bayesian computation, Stat Sci, 28, 189, 10.1214/12-STS406 · DOI · Paper page
- Rauber, 2018, Projections as visual aids for classification system design, Inf Vis, 17, 282, 10.1177/1473871617713337 · DOI
- Krause, 2014, INFUSE: interactive feature selection for predictive modeling of high dimensional data, IEEE Trans Vis Comput Gr, 20, 1614, 10.1109/TVCG.2014.2346482 · DOI · Paper page
- Yuan, 2013, Dimension projection matrix/tree: interactive subspace visual exploration and analysis of high dimensional data, IEEE Trans Vis Comput Gr, 19, 2625, 10.1109/TVCG.2013.150 · DOI
- Turkay, 2011, Brushing dimensions: a dual visual analysis model for high-dimensional data, IEEE Trans Vis Comput Gr, 17, 2591, 10.1109/TVCG.2011.178 · DOI
- Gleicher, 2013, Explainers: expert explorations with crafted projections, IEEE Trans Vis Comput Gr, 19, 2042, 10.1109/TVCG.2013.157 · DOI · Paper page
- Gleicher, 2016, A framework for considering comprehensibility in modeling, Big Data, 4, 75, 10.1089/big.2016.0007 · DOI · Paper page
- Zeng H. Towards better understanding of deep learning with visualization. 2016. [M.Sc. thesis], Department of Computer Science and Engineering, Hong-Kong University of Science and Technology.
- Liu, 2017, Towards better analysis of machine learning models: a visual analytics perspective, Vis Inf, 1, 48
- Lu, 2017, The state-of-the-art in predictive visual analytics, Comput Gr Forum, 36, 539, 10.1111/cgf.13210 · DOI · Paper page
- Lipton Z. The mythos of model interpretability. CoRR 2016;abs/1606.03490. arXiv:1606.03490. · arXiv
- LeCun, 1989, Backpropagation applied to handwritten zip code recognition, Neural Comput, 1, 541, 10.1162/neco.1989.1.4.541 · DOI · Paper page
- Elman, 1990, Finding structure in time, Cognit Sci, 14, 179, 10.1207/s15516709cog1402_1 · DOI · Paper page
- Park, 2004, Receiver operating characteristic (ROC) curve: practical review for radiologists, Korean J Radiol, 5, 11, 10.3348/kjr.2004.5.1.11 · DOI · Paper page
- Fawcett, 2006, An introduction to ROC analysis, Pattern Recognit Lett, 27, 861, 10.1016/j.patrec.2005.10.010 · DOI · Paper page
- Brehmer, 2013, A multi-level typology of abstract visualization tasks, IEEE Trans Vis Comput Gr, 19, 2376, 10.1109/TVCG.2013.124 · DOI · Paper page
- Rauber, 2017, Visualizing the hidden activity of artificial neural networks, IEEE Trans Vis Comput Gr, 23, 101, 10.1109/TVCG.2016.2598838 · DOI · Paper page
- Wong, 2004, Visual analytics, IEEE Comput Gr Appl, 24, 20, 10.1109/MCG.2004.39 · DOI · Paper page
- Keim, 2008, Visual analytics: scope and challenges, 76, 10.1007/978-3-540-71080-6_6 · DOI · Paper page
- Lu, 2017, Recent progress and trends in predictive visual analytics, Front Comput Sci, 11, 192, 10.1007/s11704-016-6028-y · DOI
- IEEE VAST 2017 Symposium. 2017. http://ieeevis.org/year/2017/info/papers.
- Streeter, 2001, NVIS: an interactive visualization tool for neural networks, vol. 4302, 1
- Goodfellow, 2014, Generative adversarial nets, vol. 27, 2672
- Hinton, 2006, A fast learning algorithm for deep belief nets, Neural Comput, 18, 10.1162/neco.2006.18.7.1527 · DOI · Paper page
- Mnih, 2015, Human-level control through deep reinforcement learning, Nature, 518, 7540, 10.1038/nature14236 · DOI · Paper page
- Liu, 2018, Analyzing the training processes of deep generative models, IEEE Trans Vis Comput Gr, 24, 77, 10.1109/TVCG.2017.2744938 · DOI · Paper page
- Ren, 2017, Squares: supporting interactive performance analysis for multiclass classifiers, IEEE Trans Vis Comput Gr, 23, 61, 10.1109/TVCG.2016.2598828 · DOI · Paper page
- Mühlbacher, 2013, A partition-based framework for building and validating regression models, IEEE Trans Vis Comput Gr, 19, 1962, 10.1109/TVCG.2013.125 · DOI
- Sedlmair, 2014, Visual parameter space analysis: a conceptual framework, IEEE Trans Vis Comput Gr, 20, 2161, 10.1109/TVCG.2014.2346321 · DOI · Paper page
- Kahng, 2018, Activis: visual exploration of industry-scale deep neural network models, IEEE Trans Vis Comput Gr, 24, 88, 10.1109/TVCG.2017.2744718 · DOI · Paper page
- Wongsuphasawat, 2018, Visualizing dataflow graphs of deep learning models in tensorflow, IEEE Trans Vis Comput Gr, 24, 1, 10.1109/TVCG.2017.2744878 · DOI · Paper page
- Smilkov D, Carter S, Sculley D, Vigas FB, Wattenberg M. Direct-manipulation visualization of deep networks. CoRR 2017;abs/1708.03788. arXiv:1708.03788. · arXiv
- Pezzotti, 2018, DeepEyes: progressive visual analytics for designing deep neural networks, IEEE Trans Vis Comput Gr, 24, 98, 10.1109/TVCG.2017.2744358 · DOI · Paper page
- Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: ICCV. 2017, p. 618–626. · DOI · Paper page
- Alsallakh, 2018, Do convolutional neural networks learn class hierarchy?, IEEE Trans Vis Comput Gr, 24, 152, 10.1109/TVCG.2017.2744683 · DOI · Paper page
- Nguyen A, Yosinski J, Clune J. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. CoRR 2016;abs/1602.03616. arXiv:1602.03616. · arXiv
- Simonyan K, Vedaldi A, Zisserman A. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR 2013;abs/1312.6034. arXiv:1312.6034. · arXiv
- Mahendran, 2016, Visualizing deep convolutional neural networks using natural pre-images, Int J Comput Vis, 120, 233, 10.1007/s11263-016-0911-8 · DOI · Paper page
- Zintgraf LM, Cohen TS, Welling M. A new method to visualize deep neural networks. CoRR 2016;abs/1603.02518. arXiv:1603.02518. · arXiv
- Li H, Mueller K, Chen X. Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation. CoRR 2017;abs/1712.08268. arXiv:1712.08268. · arXiv
- Bojarski M, Choromanska A, Choromanski K, Firner B, Jackel L, Müller U, et al. VisualBackProp: Efficient visualization of CNNs. CoRR 2016;abs/1611.05418. arXiv:1611.05418. · arXiv
- Strobelt, 2018, LSTMVis: a tool for visual analysis of hidden state dynamics in recurrent neural networks, IEEE Trans Vis Comput Gr, 24, 667, 10.1109/TVCG.2017.2744158 · DOI · Paper page
- Li J, Chen X, Hovy E, Jurafsky D. Visualizing and understanding neural models in NLP. CoRR 2015;abs/1506.01066. arXiv:1506.01066. · DOI · Paper page
- Ding, 2017, Visualizing and understanding neural machine translation, vol. 1, 1150
- van der Zwan, 2016, CUBu: universal real-time bundling for large graphs, IEEE Trans Vis Comput Gr, 22, 2250, 10.1109/TVCG.2016.2515611 · DOI
- Lhuillier, 2017, State of the art in edge and trail bundling techniques, Comput Gr Forum, 36, 619, 10.1111/cgf.13213 · DOI
- Zhang L, Wang S, Liu B. Deep learning for sentiment analysis: A survey. CoRR 2018;abs/1801.07883. arXiv:1801.07883. · DOI · Paper page
- Zhang, 2015, Deep neural networks in machine translation: an overview, IEEE Intell Syst, 30, 16, 10.1109/MIS.2015.69 · DOI · Paper page
- Maaten, 2008, Visualizing data using t-SNE, J Mach Learn Res, 9, 2579
- Sacha, 2017, Visual interaction with dimensionality reduction: a structured literature analysis, IEEE Trans Vis Comput Gr, 23, 241, 10.1109/TVCG.2016.2598495 · DOI · Paper page
- Dosovitskiy, 2016, Generating images with perceptual similarity metrics based on deep networks, vol. 29, 658
- Brochu E, Cora VM, De Freitas, N. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR 2010;abs/1012.2599. arXiv:1012.2599. · arXiv
- Snoek, 2012, Practical Bayesian optimization of machine learning algorithms, vol. 25
- Wattenberg M. How to use t-SNE effectively. 2017. https://distill.pub/2016/misread-tsne. · DOI · Paper page
- Martins, 2014, Visual analysis of dimensionality reduction quality for parameterized projections, Comput Gr, 41, 26, 10.1016/j.cag.2014.01.006 · DOI
- McInnes L, Healy J. Umap: Uniform manifold approximation and projection for dimension reduction. CoRR 2018;abs/1802.03426. arXiv:1802.03426. · DOI · Paper page
- Pezzotti, 2016, Hierarchical stochastic neighbor embedding, Comput Gr Forum, 35, 21, 10.1111/cgf.12878 · DOI
Cited by
CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
Wang, Z. J., Turko, R., Shaikh, O., Park, H., Das, N., Hohman, F., Kahng, M., & Polo Chau, D. H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030418 · Cited by 327
A survey of visual analytics for Explainable Artificial Intelligence methods
Alicioglu, G., & Sun, B. (2022) · Computers & Graphics · doi:10.1016/j.cag.2021.09.002 · Cited by 221
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81
What Are People Doing About XAI User Experience? A Survey on AI Explainability Research and Practice
Ferreira, J. J., & Monteiro, M. S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-49760-6_4 · Cited by 68
Fahmy, H., Pastore, F., Briand, L., & Stifter, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569935 · Cited by 17
“Most of the approaches that aim to explain DNN outputs [22] concern the generation of heatmaps that capture the importance of pixels in image predictions.”
Background
Metadata
- Authors
- Garcia, R., Telea, A. C., Castro da Silva, B., Tørresen, J., & Dihl Comba, J. L.
- Venue
- Computers & Graphics
- Publisher
- Elsevier BV
- Published
- 2018-12
- Volume
- 77
- Pages
- 30-49
- ISSN
- 0097-8493 (ISSN-L)
- OpenAlex
- W2895658904
- Cited by
- 54
- References
- 23
- FWCI
- 4.24
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.