CoChat
Sign Up
PreprintCited by 305Neural Networks and ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

On the importance of single directions for generalization

Open access
Morcos, Ari S., Barrett, David G. T., Rabinowitz, Neil C., Botvinick, Matthew·2018·arXiv
DOI 10.48550/arxiv.1803.06959arXiv 1803.06959
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network's reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyperparameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.06959

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a significant body of results finding neurons which appear to be interpretable in RNNs, CNNs, and GANs, activating in response to some understandable property.

    “Interpretable Neurons - There is a significant body of results finding neurons which appear to be interpretable ( in RNNs ; in CNNs ; in GANs ), activating in response to some understandable property.”

    From Toy Models of Superposition (2022) · cites this paper as [16]

    Section 4 (Related Work; page unavailable) says that a variety of deep-learning studies have highlighted individual units selective for features or classes, supporting the general point about apparently interpretable neurons. It does not identify findings in GANs or establish the claim across all three named model types. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Alessandro Achille and Stefano Soatto. On the Emergence of Invariance and Disentangling in Deep Representations. pp. 1-17, 2017. URL http://arxiv.org/abs/1706.01350. · arXiv
  2. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. (2003):1-11, 2016. URL http://arxiv.org/abs/1610.01644. · arXiv
  3. Devansh Arpit, Stanisław Jastrze ¸bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. 2017. ISSN 1938-7228. URL http://arxiv.org/abs/1706.05394. · arXiv
  4. David G.T. Barrett, Sophie Denève, and Christian K. Machens. Optimal compensation for neuron loss. eLife, 5(DECEMBER2016), 2016. ISSN 2050084X. doi: 10.7554/eLife.12454. · DOI
  5. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. 2017. doi: 10.1109/CVPR.2017. 354. URL http://arxiv.org/abs/1704.05796. · DOI · Paper page
  6. Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of machine learning research: JMLR, 2(Mar):499-526, 2002. ISSN 1532-4435. URL http://www.jmlr.org/ papers/volume2/bousquet02a/bousquet02a.pdf.
  7. Kenneth H Britten, Michael N Shadlen, William T Newsome, and J Anthony Movshon. The anal- ysis of visual motion: a comparison of neuronal and psychophysical performance. Journal of Neuroscience, 12(12):4745-4765, 1992.
  8. Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks to internal architecture and weight perturbations. March 2017. URL http://arxiv. org/abs/1703.08245.
  9. Adam Coates, Andrej Karpathy, and Andrew Y Ng. Emergence of Object-Selective Features in Unsupervised Feature Learning. Nips'2012, pp. 1-9, 2012. ISSN 10495258. doi: 10.1002/elps.200900118. URL http://www.stanford.edu/{ ˜}acoates/papers/ coateskarpathyng{_}nips2012.pdf. · DOI
  10. Russell L De Valois, E William Yund, and Norva Hepler. The orientation and direction selectivity of cells in macaque visual cortex. Vision research, 22(5):531-544, 1982.
  11. Gintare Karolina Dziugaite and Daniel M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. 2017. URL http://arxiv.org/abs/1703.11008. · arXiv
  12. Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing Higher-Layer Features of a Deep Network Technical Report. pp. 1-13, 2009.
  13. David J Freedman and John a Assad. Experience-dependent representation of visual categories in parietal cortex. Nature, 443(7107):85-8, sep 2006. ISSN 1476-4687. doi: 10.1038/nature05078. URL http://www.ncbi.nlm.nih.gov/pubmed/16936716. · DOI · Paper page
  14. Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Comput, 9(1), 1997.
  15. Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Arxiv, 2015. URL http://arxiv.org/abs/1502. 03167.
  16. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikahail Smelyanskiy, and Ping Tak Pe- ter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR, pp. 1-16, 2017.
  17. Quoc V Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S Corrado, Jeff Dean, and Andrew Y Ng. Building high-level features using large scale unsupervised learning. International Conference in Machine Learning, 2011. ISSN 10535888. doi: 10.1109/MSP.2011. 940881. · DOI
  18. Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78-84, November 2013. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature12742. URL http://dx.doi.org/ 10.1038/nature12742. · DOI
  19. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR, (2015):1-17, 2017.
  20. Jorrit S. Montijn, Guido T. Meijer, Carien S. Lansink, and Cyriel M A Pennartz. Population-Level Neural Codes Are Robust to Single-Neuron Variability from a Multidimensional Coding Perspec- tive. Cell Reports, 16(9):2486-2498, 2016. ISSN 22111247. doi: 10.1016/j.celrep.2016.07.065. URL http://dx.doi.org/10.1016/j.celrep.2016.07.065. · DOI
  21. Ari S Morcos and Christopher D Harvey. History-dependent variability in population dynamics during evidence accumulation in cortex. Nature neuroscience, 19(12):nn.4403, October 2016. ISSN 1097-6256. doi: 10.1038/nn.4403. URL http://www.nature.com/articles/ nn.4403. · DOI
  22. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring Gen- eralization in Deep Learning. 2017. URL https://arxiv.org/abs/1706.08947. · arXiv
  23. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to Generate Reviews and Discovering Sentiment. 2017. URL http://arxiv.org/abs/1704.01444. · arXiv
  24. Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the Ex- pressive Power of Deep Neural Networks. 2016. URL http://arxiv.org/abs/1606. 05336.
  25. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Understanding and Improvement. pp. 1-16, 2017. URL http://arxiv.org/abs/1706.05806. · arXiv
  26. David Raposo, Matthew T Kaufman, and Anne K Churchland. A category-free neural population supports evolving demands during decision-making. Nature neuroscience, 17(12):1784-1792, November 2014. ISSN 1097-6256. doi: 10.1038/nn.3865. URL http://www.nature. com/doifinder/10.1038/nn.3865. · DOI
  27. Ravid Shwartz-Ziv and Naftali Tishby. Opening the Black Box of Deep Neural Networks via Infor- mation. arXiv, pp. 1-19, 2017. URL http://arxiv.org/abs/1703.00810. · arXiv
  28. Samuel L. Smith and Quoc V. Le. Understanding Generalization and Stochastic Gradient Descent. pp. 1-11, 2017. URL http://arxiv.org/abs/1710.06451. · arXiv
  29. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout : A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research (JMLR), 15:1929-1958, 2014. ISSN 15337928.
  30. Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. pp. 1-14, 2017. URL http://arxiv.org/abs/1705.08292. · arXiv
  31. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. · arXiv
  32. Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Com- puter VisionECCV 2014, 8689:818-833, 2014. ISSN 978-3-319-10589-5. doi: 10.1007/ 978-3-319-10590-1 53. URL http://link.springer.com/chapter/10.1007/ 978-3-319-10590-1{_}53. · DOI
  33. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017. URL http://arxiv.org/abs/ 1611.03530.
  34. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge in Deep Scene CNNs. 2014. URL http://arxiv.org/abs/1412.6856. · arXiv
  35. Joel Zylberberg. Untuned but not irrelevant: The role of untuned neurons in sensory information coding. bioRxiv, pp. 134379, September 2017. doi: 10.1101/134379. URL https://www. biorxiv.org/content/early/2017/09/21/134379. · DOI

Cited by

Sorted by citations

Metadata

Authors
Morcos, A. S., Barrett, D. G. T., Rabinowitz, N. C., & Botvinick, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1803.06959
arXiv
1803.06959
OpenAlex
W2785648721
Cited by
305
References
3
Citations by year
2026: 7 · 2025: 18 · 2024: 18 · 2023: 30 · 2022: 29 · 2021: 60 · 2020: 62 · 2019: 49 · 2018: 31
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
generalization, deep neural network, label noise, ensembles, dropout, batch normalization, task significance, training dynamics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review