Learning and development in neural networks: the importance of starting small
The DOI matches its Crossref record; 39 of the 46 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.
Abstract
It is a striking fact that in humans the greatest learning occurs precisely at that point in time--childhood--when the most dramatic maturational changes also occur. This report describes possible synergistic interactions between maturational change and the ability to learn a complex domain (language), as investigated in connectionist networks. The networks are trained to process complex sentences involving relative clauses, number agreement, and several types of verb argument structure. Training fails in the case of networks which are fully formed and 'adultlike' in their capacity. Training succeeds only when networks begin with limited working memory and gradually 'mature' to the adult state. This result suggests that rather than being a limitation, developmental restrictions on resources may constitute a necessary prerequisite for mastering certain complex domains. Specifically, successful learning may depend on starting small.
Overview
CoChat Summary
TL;DR
Elman shows that simple recurrent networks fail to learn a complex artificial grammar when trained from the outset on the full “adult” language, but succeed when initial training is restricted by staged input or by limited memory that gradually matures. This suggests that developmental resource limitations can be a prerequisite, rather than merely a handicap, for mastering certain complex domains such as language.
Key findings
- Networks trained from the outset on the full adult corpus failed to master the task, even on training data; performance was not uniformly bad but failed on some proximal verb-agreement predictions.
- With incremental input across five phases—10,000 simple sentences (5 epochs); 7,500 simple/2,500 complex; 5,000/5,000; 2,500/7,500; and 10,000 complex—final performance yielded an error of 0.177 against empirically derived likelihood estimates, with mean cosine 0.852, and generalized to novel sentences.
- With incremental memory, the network was trained on the full adult language while recurrent feedback was initially eliminated every third or fourth word, then increased to 4–5, 5–6, 6–7 words, and finally no interference; the first phase required 12 epochs rather than five, but final performance was as good as in the staged-input simulation.
- As a counterexample, XOR is not learned if the fourth pattern is withheld until late; the network typically learns logical OR instead, and Harris (1991) reported similar results for parity.
Methods
- A simple recurrent network (Elman, 1990, 1991; Jordan, 1986; Servan-Schreiber, Cleeremans, & McClelland, 1988) was trained to predict the next word in an artificial language with number agreement, verb argument structure, and relative clauses; words were encoded as random one-hot vectors that did not convey grammatical category, meaning, or number.
- Two simulations were run: one staged the input corpora from simple to complex sentences, and one held the adult language constant while gradually increasing the network’s memory window by periodically depriving it of recurrent feedback.
- Performance was measured against unconditional probability distributions of the training data, using error and cosine measures, with principal component analysis used to inspect hidden-unit state-space representations.
Limitations
- The paper states that the discussed properties are associated with a specific approach to learning (gradient descent) and a specific class of connectionist networks involving distributed representations over units with continuously valued nonlinear activation functions; the principles do not necessarily extend to other architectures or learning algorithms.
- The paper notes a disanalogy: in the first simulation the environment was carefully constructed to present only simple sentences early, whereas children hear exemplars of all aspects of the adult language from the beginning.
- The human relevance is indirect and based on artificial neural network models; the paper also notes that the contribution of innateness and indirect negative evidence to language learnability remains unknown and controversial.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 15 Not checked · 65 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Under these conditions, it turned out that the first phase had to be extended to much longer than in the previous simulation in order to achieve a comparable level of performance (12 epochs rather than five; for purposes of comparison, performance was measured only on the simple sentences even though the network was trained on complex sentences as well).”
- Also stated in: § Incremental memory · plain
“the first phase had to be extended to much longer than in the previous simulation in order to achieve a comparable level of performance (12 epochs rather than five; for purposes of comparison, performance was measured only on the simple sentences even though the network was trained on complex sentences as well).”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Incremental memory · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If one passes the same sentences through a network prior to training, the internal representations have no discernible structure.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the network fails to learn the task when the entire data set is presented all at once, but succeeds when the data are presented incrementally.”
- Also stated in: § Incremental input · plain
“when the training data were selected such that simple sentences were presented first, the network succeeded not only in mastering these, but then going on to master the complex sentences as well.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Incremental input · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end, performance on both the training data, and also on a wide range of novel data, was as good as in the prior simulation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The incremental learning strategy employed in the simulations described here is an example of how a system can learn a complex domain by having better initial data.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 82 claims and every source verdict.
- It may state two separate claims as one. 44
- The quote only partly states this claim. 3
- Probably not a substantial claim. 18
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Property 1: statistics as the basis for learning; the problem of sample size
“(And of course, because the outcome is the same as symmetry, we cannot know with this test item whether the network is making the classification on the basis of symmetry or contents of fifth bit position; further probes would be needed.)”
§ Property 1: statistics as the basis for learning; the problem of sample size
“Or the network might have extracted the generalization that all members of class 0 have a 1 in the fifth bit position, while class 1 patterns have a 0 in that position. In this case the text item belongs in class 1.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abstract
“Training succeeds only when networks begin with limited working memory and gradually 'mature' to the adult state.”
§ Incremental input
“However, when the training data were selected such that simple sentences were presented first, the network succeeded not only in mastering these, but then going on to master the complex sentences as well.”
Citations
Cites
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Self-citationChomsky, N. (1957) · doi:10.1515/9783112316009 · Cited by 7,885
Language identification in the limit
Gold, E. M. (1967) · Information and Control · doi:10.1016/s0019-9958(67)91165-5 · Cited by 3,572
Context theory of classification learning
Medin, D. L., & Schaffer, M. M. (1978) · Psychological Review · doi:10.1037/0033-295x.85.3.207 · Cited by 3,061 · cited 1×
“Exemplar-based models make this assumption quite explicitly (Estes, 1986; Medin & Schaffer, 1978; Nosofsky, in press) and claim that experiences are stored as individually retrievable exemplars.”
BackgroundThe Cascade-Correlation learning architecture
Fahlman, S. E., & Lebiere, C. J. (2018) · doi:10.1184/r1/6610403 · Cited by 2,372
Cognition and the Development of Language
Bloom, L., & Hayes, J. R. (1974) · Language · doi:10.2307/412449 · Cited by 2,081
Handbook of mathematical psychology: I.
Luce, R. D., Bush, R. R., & Galanter, E. (1963) · Wiley eBooks · Cited by 1,999
Maturational constraints on language learning
Newport, E. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1401_2 · Cited by 1,738
Formal Principles of Language Acquisition
Baker, C. L., Wexler, K., & Culicover, P. W. (1982) · Language · doi:10.2307/414105 · Cited by 1,513
Research works printed but not linked: 2
- Baker, 1979, Syntactic theory and the projection problem, Linguistic Inquiry, 10, 533
- Nosofsky, R.M. (in press). Exemplers, prototypes, and similarity rules. In A. Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning theory to connectionist theory: Essays in honor of William K. Estes (Vol. 1). Hillsdale, NJ: Erlbaum.
Cited by
The magical number 4 in short-term memory: A reconsideration of mental storage capacity
Cowan, N. (2001) · Behavioral and Brain Sciences · doi:10.1017/s0140525x01003922 · Cited by 6,779
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
“The idea dates back at least to Elman [11], who showed that both the evolution of tasks and the network architectures can be beneficial in the language processing scenario.”
BackgroundBeing There: Putting Brain, Body and World Together Again.
van Gelder, T., & Clark, A. (1998) · The Philosophical Review · doi:10.2307/2998391 · Cited by 3,128
Metadata
- Authors
- Elman, J. L.
- Affiliations
- University of California San Diego (US)
- Corresponding author
- Elman, Jeffrey L.
- ORCID
- Elman, Jeffrey L.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 1993-07
- Volume
- 48
- Issue
- 1
- Pages
- 71-99
- ISSN
- 0010-0277 (ISSN-L)
- OpenAlex
- W2151834591
- Cited by
- 1,988
- References
- 46
- Citations by year
- 2026: 29 · 2025: 60 · 2024: 64 · 2023: 84 · 2022: 114 · 2021: 111 · 2020: 193 · 2019: 110 · 2018: 58 · 2017: 57 · 2016: 46 · 2015: 39 · 2014: 50 · 2013: 60 · 2012: 46
- FWCI
- 67.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Cognition (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- neural network development, developmental constraints, language acquisition, relative clauses, number agreement, verb argument structure, working memory, network maturation, child language acquisition
- MeSH terms
- Child, Child, Preschool, Humans, Infant, Language Development, Phonetics, Psycholinguistics, Mental Recall, Semantics, Vocabulary, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning and development in neural networks: the importance of starting small. https://app.cochat.ai/papers/10.1016/0010-0277(93)90058-4
- The paper
- Elman, J. L. (1993). Learning and development in neural networks: the importance of starting small. Cognition. https://doi.org/10.1016/0010-0277(93)90058-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.