CoChat
Sign Up
Cited by 284Neural Networks and ApplicationsCategorization, perception, and languageCognitive Science and Education Research

A mathematical theory of semantic development in deep neural networks

Open access
Saxe, Andrew M., McClelland, James L., Ganguli, Surya·2019·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1820226116PMC PMC6561300
Open this paper in CoChat — free accountOpen PDF

Abstract

Significance: Over the course of development, humans learn myriad facts about items in the world, and naturally group these items into useful categories and structures. This semantic knowledge is essential for diverse behaviors and inferences in adulthood. How is this richly structured semantic knowledge acquired, organized, deployed, and represented by neuronal networks in the brain? We address this question by studying how the nonlinear learning dynamics of deep linear networks acquires information about complex environmental structures. Our results show that this deep learning dynamics can self-organize emergent hidden representations in a manner that recapitulates many empirical phenomena in human semantic development. Such deep networks thus provide a mathematically tractable window into the development of internal neural representations through experience.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1820226116

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Certain networks (nonlinear and deep linear) split embedding vectors in a manner very much like what was observed in the toy model.

    “They find empirically that certain networks (nonlinear and deep linear) 'split' embedding vectors in a manner very much like what we observed.”

    From Toy Models of Superposition (2022) · cites this paper as [31]

    The comparison around Fig. 2 says a simulated nonlinear network shows progressive differentiation of its hidden representations, and that a deep linear network shows a resembling pattern. However, the paper says it mathematically derives the deep linear result, rather than finding that result empirically. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The learning dynamics observed in the toy model are directly related to previous findings on simple models.

    “The learning dynamics we observe here seem directly related to previous findings on simple models.”

    From Toy Models of Superposition (2022) · cites this paper as [31]

  • It seems likely that the cited theoretical analysis could be generalized to the toy model case.

    “Despite this difference, it seems likely that their analysis could be generalized to our case.”

    From Toy Models of Superposition (2022) · cites this paper as [31]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. 10.4159/harvard.9780674181816 · DOI
  2. Carey S (1985) Conceptual Change In Childhood (MIT Press, Cambridge, MA).
  3. 10.7551/mitpress/1602.001.0001 · DOI
  4. 10.7551/mitpress/6161.001.0001 · DOI
  5. 10.1016/S0885-2014(93)80003-C · DOI
  6. 10.1037/10034-000 · DOI
  7. 10.1016/0010-0285(76)90016-5 · DOI · Paper page
  8. 10.1016/0010-0285(75)90024-9 · DOI · Paper page
  9. 10.1037/0278-7393.11.1-4.629 · DOI
  10. 10.1016/S0022-5371(73)80056-8 · DOI
  11. 10.1037/0278-7393.11.1.70 · DOI
  12. 10.3758/BF03337190 · DOI
  13. 10.1016/S0022-5371(75)80055-7 · DOI · Paper page
  14. 10.1037/0033-295X.97.2.185 · DOI
  15. Rosch E (1978) Principles of Categorization in Cognition and Categorization, eds Rosch E, Lloyd B (Lawrence Erlbaum, Hillsdale, NJ), pp 27–48.
  16. 10.1037/0033-295X.92.3.289 · DOI
  17. Keil F (1991) The emergence of theoretical beliefs as constraints on concepts. The Epigenesis of Mind: Essays on Biology and Cognition, eds Carey S, Gelman R (Psychology Press, New York).
  18. 10.1017/S0140525X10000919 · DOI
  19. 10.1037/0012-1649.26.5.796 · DOI
  20. 10.1017/S0140525X98001253 · DOI
  21. 10.1016/j.tics.2013.06.007 · DOI
  22. 10.1162/jocn_a_00458 · DOI
  23. 10.1523/JNEUROSCI.5547-11.2012 · DOI
  24. 10.3389/fpsyg.2013.00128 · DOI
  25. 10.1016/j.neuron.2008.10.043 · DOI
  26. 10.1073/pnas.0802631105 · DOI
  27. Hinton G (1986) Learning distributed representations of concepts. Parallel Distributed Processing: Implications for Psychology and Neurobiology, ed Morris R (Oxford Univ Press, Oxford).
  28. Rumelhart D, Todd P (1993) Learning and Connectionist Representations in Attention and Performance XIV, eds Meyer D, Kornblum S (MIT Press, Cambridge, MA).
  29. McClelland J (1995) A connectionist perspective on knowledge and development. Developing Cognitive Competence: New Approaches to Process Modeling, eds Simon T, Halford G (Erlbaum, Hillsdale, NJ).
  30. 10.1111/j.2044-835X.1992.tb00575.x · DOI
  31. 10.1006/jecp.1997.2385 · DOI
  32. 10.1037/0033-295X.112.2.347 · DOI
  33. Poole B, Lahiri S, Raghu M, Sohl-Dickstein J, Ganguli S (2016) Exponential expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, eds Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R (MIT Press, Cambridge, MA), Vol 29, pp 3360–3368.
  34. 10.1207/s15516709cog1401_5 · DOI
  35. 10.2307/1129660 · DOI
  36. 10.1093/cercor/bht014 · DOI
  37. 10.1037/0096-1523.2.4.491 · DOI
  38. Anglin J (1977) Word, Object, and Conceptual Development (Norton, New York), p 302.
  39. 10.1016/0010-0285(91)90011-C · DOI
  40. 10.1037/0033-2909.111.2.291 · DOI
  41. 10.1016/j.acha.2005.02.001 · DOI
  42. 10.1007/s00357-007-0007-9 · DOI
  43. Gray R (2005) Toeplitz and circulant matrices: A review. Foundations Trends Communications Information Theory 2:155–239. · DOI
  44. 10.1080/09515080050002726 · DOI
  45. 10.1126/science.1063736 · DOI · Paper page
  46. 10.1002/hbm.21296 · DOI
  47. 10.1162/jocn_a_00189 · DOI

Cited by

Sorted by citations

Authors
Saxe, A. M., McClelland, J. L., & Ganguli, S.
Affiliations
  • University of Oxford (GB)
  • Stanford University (US)
  • Google (United States) (US)
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2019-05-17
Volume
116
Issue
23
Pages
11537-11546
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1820226116
PMC
PMC6561300
OpenAlex
W2898365215
Cited by
284
References
1
Citations by year
2026: 46 · 2025: 41 · 2024: 38 · 2023: 36 · 2022: 39 · 2021: 42 · 2020: 31 · 2019: 10 · 2018: 1
FWCI
12.27
Type
journal article
Language
en
Open access
Licence
www.pnas.org/site/aboutpnas/licenses.xhtml (vor, 184 d)
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Oxford University Research Archive (ORA) (University of Oxford) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • McKnight Foundation
  • Wellcome Trust
  • National Defense Science and Engineering Graduate
  • Air Force Office of Scientific Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Cognitive Science and Education Research
Keywords
semantic development, semantic cognition, deep linear networks, exact solutions, semantic illusions, conceptual distinction, semantic similarity, cross-species conservation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review