Visualizing the Hidden Activity of Artificial Neural Networks
Open accessAbstract
In machine learning, pattern classification assigns high-dimensional vectors (observations) to classes based on generalization from examples. Artificial neural networks currently achieve state-of-the-art results in this task. Although such networks are typically used as black-boxes, they are also widely believed to learn (high-dimensional) higher-level representations of the original observations. In this paper, we propose using dimensionality reduction for two tasks: visualizing the relationships between learned representations of observations, and visualizing the relationships between artificial neurons. Through experiments conducted in three traditional image classification benchmark datasets, we show how visualization can provide highly valuable feedback for network designers. For instance, our discoveries in one of these datasets (SVHN) include the presence of interpretable clusters of learned representations, and the partitioning of artificial neurons into groups with apparently related discriminative roles.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
t-SNE projections of hidden neuron activations and learned data representations provide valuable feedback about relationships between neurons and classes.
“It could be shown that these projections provide valuable feedback about the relationships between neurons and classes.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [55]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
(2014) · Cited by 33,419 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
The NumPy Array: A Structure for Efficient Numerical Computation
van der Walt, S., Colbert, S. C., & Varoquaux, G. P. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.37 · Cited by 10,445 · see citing works
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,048 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
Accelerating t-SNE using tree-based algorithms
van der Maaten, L. (2014) · Journal of machine learning research · doi:10.5555/2627435.2697068 · Cited by 2,207 · see citing works
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Cited by 2,180 · see citing works
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116 · cited 1× · see citing works
“Manual choice of hyperparameters requires significant expertise and effort, and comprehensive guides have been written on the subject [32, 5].”
Background(2013) · Cited by 2,056 · see citing works
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827 · see citing works
Understanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Deeply-Supervised NetsPreprint
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 1,661 · see citing works
Prechelt, L. (1998) · Lecture Notes in Computer Science · doi:10.1007/3-540-49430-8_3 · Cited by 1,649 · see citing works
Modern multidimensional scaling : theory and applications
Borg, I., & Groenen, P. J. F. (1997) · Springer eBooks · doi:10.1007/0-387-28981-x · Cited by 1,615 · see citing works
Modern Multidimensional Scaling: Theory and Applications
O'Connell, A. A., Borg, I., & Groenen, P. J. F. (1999) · Journal of the American Statistical Association · doi:10.2307/2669710 · Cited by 1,079 · see citing works
Theano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005 · see citing works
Deeplysupervised netsPreprint
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015) · Cited by 850 · see citing works
Gower, J. C., & Dijksterhuis, G. (2004) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198510581.001.0001 · Cited by 599 · see citing works
Visualizing high-dimensional data: advances in the past decade
Liu, S., Maljovec, D., Wang, B., Bremer, P.-T., & Pascucci, V. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2640960 · Cited by 427 · see citing works
Effectiveness of Animation in Trend Visualization
Robertson, G., Fernandez, R., Fisher, D., Lee, B., & Stasko, J. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2008.125 · Cited by 362 · see citing works
Paulovich, F. V., Nonato, L. G., Minghim, R., & Levkowitz, H. (2008) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70443 · Cited by 317 · see citing works
Local Affine Multidimensional Projection
Joia, P., Paulovich, F. V., Coimbra, D. B., Cuminato, J. A., & Nonato, L. G. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.220 · Cited by 278 · see citing works
Understanding how Deep Belief Networks perform acoustic modelling
Mohamed, A., Hinton, G. E., & Penn, G. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288863 · Cited by 272 · see citing works
Learning Features From Music Audio With Deep Belief Networks.
Hamel, P., & Eck, D. (2010) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1414969 · Cited by 264 · see citing works
Visualizing distortions and recovering topology in continuous projection techniques
Aupetit, M. J. M. (2006) · Neurocomputing · doi:10.1016/j.neucom.2006.11.018 · Cited by 147 · see citing works
Visual analysis of dimensionality reduction quality for parameterized projections
Martins, R. M., Coimbra, D. B., Minghim, R., et al. (2014) · Computers & Graphics · doi:10.1016/j.cag.2014.01.006 · Cited by 136 · see citing works
Mühlbacher, T., Piringer, H., Gratzl, S., Sedlmair, M., & Streit, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346578 · Cited by 131 · see citing works
Understanding Deep Features with Computer-Generated Imagery
Aubry, M., & Russell, B. C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.329 · Cited by 111 · see citing works
Yuan, X., Ren, D., Wang, Z., & Guo, C. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.150 · Cited by 104 · see citing works
Nonlinear dimensionality reduction of data manifolds with essential loops
Lee, J. A., & Verleysen, M. (2005) · Neurocomputing · doi:10.1016/j.neucom.2004.11.042 · Cited by 90 · see citing works
Brushing Dimensions - A Dual Visual Analysis Model for High-Dimensional Data
Turkay, Ç., Filzmoser, P., & Hauser, H. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.178 · Cited by 89 · see citing works
Bundled Visualization of DynamicGraph and Trail Data
Hurter, C., Ersoy, O., Fabrikant, S. I., et al. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.246 · Cited by 82 · see citing works
Temporal MDS Plots for Analysis of Multivariate Data
Jäckle, D., Fischer, F., Schreck, T., et al. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467553 · Cited by 79 · see citing works
CUBu: Universal Real-Time Bundling for Large Graphs
van der Zwan, M., Codreanu, V., & Telea, A. C. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2515611 · Cited by 61 · see citing works
Self-citationDistortion‐Guided Structure‐Driven Interactive Exploration of High‐Dimensional Data
Liu, S., Wang, B., Bremer, P., & Pascucci, V. (2014) · Computer Graphics Forum · doi:10.1111/cgf.12366 · Cited by 35 · see citing works
Interactive Image Feature Selection Aided by Dimensionality Reduction
Rauber, P., da Silva, R. R. O., Feringa, S., et al. (2015) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.2312/eurova.20151098 · Cited by 23 · see citing works
Self-citationGarcía-Fernández, F. J., Verleysen, M., Lee, J. A., & Díaz, I. (2013) · Consultation of the Doctoral Thesis Database (TESEO) (Ministerio de Educación, Cultura y Deporte) · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294 · see citing works
Rulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243 · see citing works
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips
Nguyen, T.-L., Kavuri, S., & Lee, M. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.06.010 · Cited by 84 · see citing works
An Interactive Approach to Bias Mitigation in Machine Learning
Wang, H., Mukhopadhyay, S., Xiao, Y., & Fang, S. (2021) · 2021 IEEE 20th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC) · doi:10.1109/iccicc53683.2021.9811333 · Cited by 12 · see citing works
- Authors
- Rauber, P. E., Fadel, S. G., Falcao, A. X., & Telea, A. C.
- Affiliations
- University of Groningen (NL)
- Universidade de São Paulo (BR)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2017-01
- Volume
- 23
- Issue
- 1
- Pages
- 101-110
- ISSN
- 1077-2626 (ISSN-L)
- OpenAlex
- W2512304460
- Cited by
- 280
- References
- 50
- Citations by year
- 2026: 10 · 2025: 16 · 2024: 23 · 2023: 23 · 2022: 33 · 2021: 44 · 2020: 50 · 2019: 33 · 2018: 29 · 2017: 18 · 2016: 1
- FWCI
- 21.96
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- CAPES
- FAPESP (2012/24121-9, 2015/19851-6)
- CNPq (302970/2014-2, 479070/2013-0)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial neural networks, neural network visualization, dimensionality reduction, learned representations, artificial neurons, interpretability, image classification, SVHN
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
