CoChat

Learning and development in neural networks: the importance of starting small

Elman, Jeffrey L. · 1993 · Cognition

The DOI matches its Crossref record; 39 of the 46 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.

Abstract

It is a striking fact that in humans the greatest learning occurs precisely at that point in time--childhood--when the most dramatic maturational changes also occur. This report describes possible synergistic interactions between maturational change and the ability to learn a complex domain (language), as investigated in connectionist networks. The networks are trained to process complex sentences involving relative clauses, number agreement, and several types of verb argument structure. Training fails in the case of networks which are fully formed and 'adultlike' in their capacity. Training succeeds only when networks begin with limited working memory and gradually 'mature' to the adult state. This result suggests that rather than being a limitation, developmental restrictions on resources may constitute a necessary prerequisite for mastering certain complex domains. Specifically, successful learning may depend on starting small.

Overview

CoChat Summary

TL;DR

Elman shows that simple recurrent networks fail to learn a complex artificial grammar when trained from the outset on the full “adult” language, but succeed when initial training is restricted by staged input or by limited memory that gradually matures. This suggests that developmental resource limitations can be a prerequisite, rather than merely a handicap, for mastering certain complex domains such as language.

Key findings

  • Networks trained from the outset on the full adult corpus failed to master the task, even on training data; performance was not uniformly bad but failed on some proximal verb-agreement predictions.
  • With incremental input across five phases—10,000 simple sentences (5 epochs); 7,500 simple/2,500 complex; 5,000/5,000; 2,500/7,500; and 10,000 complex—final performance yielded an error of 0.177 against empirically derived likelihood estimates, with mean cosine 0.852, and generalized to novel sentences.
  • With incremental memory, the network was trained on the full adult language while recurrent feedback was initially eliminated every third or fourth word, then increased to 4–5, 5–6, 6–7 words, and finally no interference; the first phase required 12 epochs rather than five, but final performance was as good as in the staged-input simulation.
  • As a counterexample, XOR is not learned if the fourth pattern is withheld until late; the network typically learns logical OR instead, and Harris (1991) reported similar results for parity.

Methods

  • A simple recurrent network (Elman, 1990, 1991; Jordan, 1986; Servan-Schreiber, Cleeremans, & McClelland, 1988) was trained to predict the next word in an artificial language with number agreement, verb argument structure, and relative clauses; words were encoded as random one-hot vectors that did not convey grammatical category, meaning, or number.
  • Two simulations were run: one staged the input corpora from simple to complex sentences, and one held the adult language constant while gradually increasing the network’s memory window by periodically depriving it of recurrent feedback.
  • Performance was measured against unconditional probability distributions of the training data, using error and cosine measures, with principal component analysis used to inspect hidden-unit state-space representations.

Limitations

  • The paper states that the discussed properties are associated with a specific approach to learning (gradient descent) and a specific class of connectionist networks involving distributed representations over units with continuously valued nonlinear activation functions; the principles do not necessarily extend to other architectures or learning algorithms.
  • The paper notes a disanalogy: in the first simulation the environment was carefully constructed to present only simple sentences early, whereas children hear exemplars of all aspects of the adult language from the beginning.
  • The human relevance is indirect and based on artificial neural network models; the paper also notes that the contribution of innateness and indirect negative evidence to language learnability remains unknown and controversial.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/0010-0277(93)90058-4

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 15 Not checked · 65 Own results and interpretations

23 more claims

Create a free account to see all 82 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 44
  • The quote only partly states this claim. 3
  • Probably not a substantial claim. 18

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 39 references

Research works printed but not linked: 2

  1. Baker, 1979, Syntactic theory and the projection problem, Linguistic Inquiry, 10, 533
  2. Nosofsky, R.M. (in press). Exemplers, prototypes, and similarity rules. In A. Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning theory to connectionist theory: Essays in honor of William K. Estes (Vol. 1). Hillsdale, NJ: Erlbaum.

Cited by

See all 2,017

Metadata

Authors
Elman, J. L.
Affiliations
University of California San Diego (US)
Corresponding author
Elman, Jeffrey L.
Venue
Cognition
Publisher
Elsevier BV
Published
1993-07
Volume
48
Issue
1
Pages
71-99
ISSN
0010-0277 (ISSN-L)
OpenAlex
W2151834591
Cited by
1,988
References
46
Citations by year
2026: 29 · 2025: 60 · 2024: 64 · 2023: 84 · 2022: 114 · 2021: 111 · 2020: 193 · 2019: 110 · 2018: 58 · 2017: 57 · 2016: 46 · 2015: 39 · 2014: 50 · 2013: 60 · 2012: 46
FWCI
67.51
Type
journal article
Language
en
Open access
closed
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Cognition (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
neural network development, developmental constraints, language acquisition, relative clauses, number agreement, verb argument structure, working memory, network maturation, child language acquisition
MeSH terms
Child, Child, Preschool, Humans, Infant, Language Development, Phonetics, Psycholinguistics, Mental Recall, Semantics, Vocabulary, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning and development in neural networks: the importance of starting small. https://app.cochat.ai/papers/10.1016/0010-0277(93)90058-4
The paper
Elman, J. L. (1993). Learning and development in neural networks: the importance of starting small. Cognition. https://doi.org/10.1016/0010-0277(93)90058-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.