The DOI matches its DataCite record, with a small difference in title or year.
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Open accessAbstract
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less. Code: https://github.com/google/svcca/
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several other methods, such as t-distributed stochastic neighbor embedding (t-SNE) and correlation-based singular vector canonical correlation analysis (SVCCA) [48], play a significant role in directing towards the subspace of input for error-free predictions.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [48]
Citations
Cites
Still being checked.
Cited by
Sorted by citations
Items 301–342 of 342
Facial Landmark Correlation AnalysisPreprint
Yan, Y., Duffner, S., Phutane, P., Berthelier, A., Blanc, C., García, C., & Château, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10576
Knowledge Discovery from Layered Neural Networks Based on Non-negative Task Matrix Decomposition
Watanabe, C., Hiramatsu, K., & Kashino, K. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edp7136 · Cited by 2
Visualizing Classification Structure of Large-Scale ClassifiersPreprint
Alsallakh, B., Yan, Z., Ghaffarzadegan, S., Dai, Z., & Ren, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06068
OrthoReg: Robust Network Pruning Using Orthonormality RegularizationPreprint
Lubana, E. S., Trivedi, P., Hougen, C. D., Dick, R. P., & Hero, A. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05014
Hierarchical Modeling with Neurodynamical Agglomerative Analysis
Marino, M., Schröter, G., Heidemann, G., & Hertzberg, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61609-0_15
Analyzing Representations inside Convolutional Neural NetworksPreprint
Saini, U. S., & Papalexakis, E. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12516
Investigating Deep Neural Structures and their Interpretability in the Domain of Voice Conversion
Broughton, S. J., Jalal, A., & Moore, R. K. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1730
Measuring the Stability of Learned FeaturesPreprint
Sankaran, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10388
Coherence of Working Memory Study Between Deep Neural Network and NeurophysiologyPreprint
Ming, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10994
An Investigation of the Weight Space to Monitor the Training Progress of Neural NetworksPreprint
Schürholt, K., & Borth, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10424 · Cited by 1
CACTUS: Detecting and Resolving Conflicts in Objective FunctionsPreprint
Das, S., & Endert, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07805
Relieving the Plateau: Active Semi-Supervised Learning for a Better Landscape.Preprint
Kong, S. T., Jeon, S., Lee, J., Lee, H., & Jung, K.-H. (2021) · arXiv (Cornell University)
Layer Pruning on Demand with Intermediate CTCPreprint
Lee, J., Kang, J., & Watanabe, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09216
Exploring the Properties and Evolution of Neural Network Eigenspaces during Training
Richter, M. L., Malihi, L., Windler, A.-K. P., & Krumnack, U. (2022) · Iranian Conference on Machine Vision and Image Processing · doi:10.1109/mvip53647.2022.9738741 · Cited by 1
Maximizing Ensemble Diversity in Deep Q-LearningPreprint
Sheikh, H. U., Phielipp, M., & Bölöni, L. L. (2020) · arXiv (Cornell University)
Basis Scaling and Double Pruning for Efficient Transfer Learning.Preprint
Wong, K. C. L., Kashyap, S., & Moradi, M. (2021) · arXiv (Cornell University)
When Neural Networks Using Different Sensors Create Similar Features
Moreau, H., Vassilev, A., & Chen, L. (2022) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-99203-3_5
Convolutional Neural Network Dynamics: A Graph Perspective.Preprint
Vahedian, F., Li, R., Trivedi, P., Jin, D., & Koutra, D. (2021) · arXiv (Cornell University) · Cited by 1
Understanding the Dynamics of DNNs Using Graph ModularityPreprint
Lu, Y., Wen, Y., Zhang, Y., Chen, Z., Chen, J., Xuan, Q., Wang, Z., & Yang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12485
Selective Federated Transfer Learning using Representation SimilarityPreprint
Semwal, T., Wang, H., & Reddy, C. K. T. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/kbhq5
Comparison of Supervised and Self-Supervised Deep Representations Trained on Histological Images
Rymarczyk, D., Borowa, A., Bracha, A., Chronowski, M., Ozimek, W., & Zieliński, B. (2022) · Studies in health technology and informatics · doi:10.3233/shti220263
Yuan, P., & Nguyen, H. V. (2022) · Elsevier eBooks · doi:10.1016/b978-0-32-399851-2.00011-9 · Cited by 1
VampNet: Unsupervised Vampirizing of Convolutional Networks
Nguyen, T.-L. R., Château, T., & Magniez, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20713-6_19
Virtual Tasks but Real Gains: Improving Multi-Task Learning
Pranavan, T., Sim, T., & Li, J. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956353
Equivariance Score Based on Canonical Correlation Analysis
Ohta, N., & Nobuhara, H. (2022) · Joint International Conference on Soft Computing and Intelligent Systems and International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · doi:10.1109/scisisis55246.2022.10001954
Understanding how visual information is represented in humans and machines
Dwivedi, K. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.72036
Bootstrap Confidence Regions for Learned Feature Embeddings
Sankaran, K. (2023) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2023.2197478
Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio LearnersPreprint
Yadav, S., Theodoridis, S., Hansen, L. K., & Tan, Z.-H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.00561 · Cited by 1
Understanding the class separation process of convolutional neural networks
Yang, Y., Lian, X., Liu, W., Shen, L., & Lin, J. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/6/20230665
Hamerlik, E., Deb, M., & Takáč, M. (2023) · World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa59116.2023.10308930
On the transferability of the Deep Galerkin Method for solving partial differential equations
Aristotelous, A. C., & Papanicolaou, N. C. (2023) · AIP conference proceedings · doi:10.1063/5.0177426
Manipulating and Measuring Variation In Deep Neural Network (DNN) Representations of ObjectsPreprint
Chow, J. K.-J., & Palmeri, T. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4635716
Mitigating Catastrophic Forgetting in Continual Learning-Based Image Classification
Borde, S. (2023) · IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI) · doi:10.1109/soli60636.2023.10425549 · Cited by 1
A Non-Ergodic Framework for Understanding Emergent Capabilities in Large Language ModelsPreprint
Marin, J. (2025) · Qeios · doi:10.32388/00jqn6
Study of Foundation Models Knowledge Representations: Geometry Perspective
Magai, G., & Soroka, A. A. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-80463-2_10
Measuring Error Alignment for Decision-Making Systems
Xu, B., Bikakis, A., Onah, D. F. O., Vlachidis, A., & Dickens, L. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34988
Kwon, J. H., Ko, Y., Song, E., & Hong, D. (2026) · International Journal of Precision Engineering and Manufacturing · doi:10.1007/s12541-025-01396-2
Stecher, P. M., Sikimić, V., & Kähle, R. (2026) · Minds and Machines · doi:10.1007/s11023-026-09770-3
A number simplex in the human medial temporal lobePreprint
Zhu, H., Chericoni, A., Ismail, T., Mickiewicz, E. A., Franch, M. C., Nigam, T., Yan, X., Belanger, J. L., Chavez, A. G., Nair, J., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.25.734462
Nguyen, T., Raghu, M., & Kornblith, S. (2020) · arXiv · doi:10.48550/arxiv.2010.15327 · Cited by 112
Self-citationRevisiting Model Stitching to Compare Neural Representations
Bansal, Y., Nakkiran, P., & Barak, B. (2021) · arXiv · doi:10.48550/arxiv.2106.07682 · Cited by 26
Voita, E., Sennrich, R., & Titov, I. (2019) · arXiv · doi:10.48550/arxiv.1909.01380 · Cited by 17
Metadata
- Authors
- Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J.
- Affiliations
- Google (United States) (US)
- Uber AI (United States) (US)
- ORCID
- Gilmer, Justin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 30
- Pages
- 6076-6085
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1706.05806
- OpenAlex
- W2767204723
- Cited by
- 335
- References
- 0
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 33 · 2023: 55 · 2022: 40 · 2021: 77 · 2020: 65 · 2019: 41 · 2018: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- representation comparison, affine invariance, overparameterization, learning dynamics, training dynamics, interpretability, training protocols, overfitting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review