PreprintCited by 342Explainable Artificial Intelligence (XAI)Neural Networks and ApplicationsStochastic Gradient Optimization Techniques

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Open access
Raghu, Maithra, Gilmer, Justin, Yosinski, Jason, Sohl-Dickstein, Jascha·2017·arXiv

Abstract

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less. Code: https://github.com/google/svcca/

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.05806

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • t-SNE and SVCCA play a significant role in directing towards the subspace of input for error-free predictions.

    “Several other methods, such as t-distributed stochastic neighbor embedding (t-SNE) and correlation-based singular vector canonical correlation analysis (SVCCA) [48], play a significant role in directing towards the subspace of input for error-free predictions.”

Cites

Still being checked.

Cited by

Sorted by citations

Authors
Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J.
Affiliations
  • Google (United States) (US)
  • Uber AI (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
30
Pages
6076-6085
ISSN
2331-8422 (ISSN-L)
OpenAlex
W2767204723
Cited by
335
References
0
Citations by year
2026: 4 · 2025: 13 · 2024: 33 · 2023: 55 · 2022: 40 · 2021: 77 · 2020: 65 · 2019: 41 · 2018: 7
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
representation comparison, affine invariance, overparameterization, learning dynamics, training dynamics, interpretability, training protocols, overfitting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review