The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
Open accessAbstract
Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it's seen as key to the human capacity for generalization in language.Recent work (Lake and Baroni, 2018) has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool.Lake and Baroni's main experiment required the models to quickly bootstrap the meaning of new words.We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as "around" and "right") in novel contexts.Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of "X around right" to "jump around right"), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of "around right" from those of "right" and "around").
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The tasks SCAN and COGS feature long sequences as in-context study examples.
“The tasks from the machine-learning literature that we experimented with, SCAN11,66 and COGS16, feature long sequences as (in-context) study examples.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [66]
Section 2 describes SCAN as a sequence-to-sequence task that maps commands to action sequences, and notes a test split with longer action sequences than the training set. This supports the reference to sequences in SCAN, but the paper does not characterize them as long in-context study examples; it describes conventional training and test sets instead. COGS is attached to a separate citation in the citing sentence. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Self-citationNeural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Flagged first, then by citations
Research works printed but not linked: 14
- Matthew Botvinick and David Plaut. 2009. Empirical and computational support for context-dependent representations of serial order: Reply to Bowers, Damian, and Davis (2009). Psychological Review, 116(4):998-1002.
- Jeffrey Bowers, Markus Damian, and Colin David. 2009. A fundamental limitation of the conjunctive codes learned in PDP models of cognition: Com- ment on Botvinick and Plaut (2006). Psychological Review, 116(4):986-997.
- Philémon Brakel and Stefan Frank. 2009. Strong sys- tematicity in sentence processing by simple recur- rent networks. In Proceedings of CogSci, pages 1599-1604, Amsterdam, the Netherlands.
- Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259. · arXiv
- Morten Christiansen and Nick Chater. 1994. General- ization and connectionist language learning. Mind & Language, 9(3):273-287.
- Stefan Frank. 2014. Getting real about systematic- ity. In Paco Calvo and John Symons, editors, The architecture of cognition: Rethinking Fodor and Pylyshyn's systematicity challenge, pages 147-164.
- Aurélie Herbelot and Marco Baroni. 2017. High-risk learning: acquiring new word vectors from tiny data. In Proceedings of EMNLP, pages 304-309, Copen- hagen, Denmark.
- Brenden Lake and Marco Baroni. 2018. Generaliza- tion without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of ICML, pages 2879-2888, Stock- holm, Sweden.
- Andrew Kyle Lampinen and James L McClelland. 2017. One-shot and few-shot learning of word em- beddings. CoRR.
- Gary F Marcus. 1998. Rethinking Eliminative Connec- tionism. Cognitive Psychology, 282(37):243-282.
- Gary F Marcus. 2003. The Algebraic Mind: Inte- grating Connectionism and Cognitive Science. MIT Press, Cambridge, MA.
- Richard Montague. 1970. Universal grammar. Theo- ria, 36(3):373-398.
- Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural net- works. In Advances in neural information process- ing systems, pages 3104-3112.
- Frank van der Velde, Gwendid van der Voort van der Kleij, and Marc de Kamps. 2004. Lack of com- binatorial productivity in language processing with simple recurrent networks. Connection Science, 16(1):21-46.
Cited by
Sorted by citations
Brains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318
Human-like systematic generalization through a meta-learning neural network
Lake, B. M., & Baroni, M. (2023) · Nature · doi:10.1038/s41586-023-06668-3 · Cited by 230
Self-citationOn the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102
Metadata
- Authors
- Loula, J., Baroni, M., & Lake, B.
- Affiliations
- École Polytechnique (FR)
- Meta (United States) (US)
- Venue
- Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
- Published
- 2018
- Pages
- 108-114
- OpenAlex
- W2887970879
- Cited by
- 136
- References
- 4
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 9 · 2023: 22 · 2022: 19 · 2021: 25 · 2020: 29 · 2019: 19 · 2018: 2
- FWCI
- 6.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- compositional generalization, recurrent neural networks, sequence-to-sequence models, navigation instructions, function words, compositional rules, language generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
